Python高级编程技巧

ipython的使用

地址：https://ipython.org/install.html

简单的安装方法：pip install ipython

一些方便的使用方法：

• 输入要查看的对象，然后输入一个问号可以查看API，输入两个问号可以查看代码
• 可以直接调用shell命令，在前面带上！即可
• 按Tab可以自动语法补全
• 最近的命令输出结果，可以从_、__、___三个变量获得
• %hist或者%history查看历史命令
• %timeit可以进行命令的执行时间测试

数据结构与算法

列表生成器

In [5]: a=[1,2,3]

In [6]: [x*x for x in a if x>1]
Out[6]: [4, 9]

集合生成器（和列表生成器相同）

In [7]: a=[1,2,3]

In [8]: s = {x*x for x in a if x>1}

In [9]: s
Out[9]: {4, 9}

In [10]: type(s)
Out[10]: set

字典生成器

In [11]: a=[1,2,3]

In [12]: {str(x):x+1 for x in a if x>1}
Out[12]: {'2': 3, '3': 4}

range和xrange的使用

In [13]: range?
Docstring:
range(stop) -> list of integers
range(start, stop[, step]) -> list of integers

In [14]: range(10)
Out[14]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

In [15]: range(3,10)
Out[15]: [3, 4, 5, 6, 7, 8, 9]

In [16]: xrange?
Docstring:
xrange(stop) -> xrange object
xrange(start, stop[, step]) -> xrange object

In [19]: list(xrange(3,10))
Out[19]: [3, 4, 5, 6, 7, 8, 9]

In [21]: list(xrange(10))
Out[21]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

range和xrange的用法相同，只是range会直接生成一个内存列表，xrange会生成一个生产器，xrange的效率更高，更节省内存

filter用于过滤数据

In [22]: filter?
Docstring:
filter(function or None, sequence) -> list, tuple, or string

Return those items of sequence for which function(item) is true.  If
function is None, return the items that are true.  If sequence is a tuple
or string, return the same type, else return a list.
Type:      builtin_function_or_method

In [23]: filter(lambda x:x%3==0, xrange(10))
Out[23]: [0, 3, 6, 9]

使用collections.namedtuple给列表或者元组命名

In [24]: from collections import namedtuple

In [30]: Point = namedtuple('Point', ['x', 'y'])

In [31]: p = Point(11, 22)

In [32]: p.__dict__
Out[32]: OrderedDict([('x', 11), ('y', 22)])

In [33]: p.x
Out[33]: 11

In [34]: p.y
Out[34]: 22

random的使用

In [35]: from random import randint

In [36]: randint?
Signature: randint(a, b)
Docstring:
Return random integer in range [a, b], including both end points.

File:      /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/random.py
Type:      instancemethod

In [37]: randint(1,10)
Out[37]: 2

统计序列元素的频度和TOP N

In [42]: from collections import Counter

In [43]: Counter?

In [44]: c = Counter('aaabbbbccccccddddddeeeeee')

In [45]: c
Out[45]: Counter({'a': 3, 'b': 4, 'c': 6, 'd': 6, 'e': 6})

In [46]: c.most_common(3)
Out[46]: [('c', 6), ('e', 6), ('d', 6)]

将字典按value排序

In [47]: from random import randint

In [48]: keys = 'abcdefg'

In [50]: d = {x:randint(90,100) for x in keys}

In [51]: d
Out[51]: {'a': 90, 'b': 100, 'c': 94, 'd': 97, 'e': 94, 'f': 95, 'g': 91}

In [53]: d.items()
Out[53]: [('a', 90), ('c', 94), ('b', 100), ('e', 94), ('d', 97), ('g', 91), ('f', 95)]

In [54]: sorted?
Docstring: sorted(iterable, cmp=None, key=None, reverse=False) --> new sorted list
Type:      builtin_function_or_method

In [55]: sorted(d.items(), key=lambda x : x[1])
Out[55]: [('a', 90), ('g', 91), ('c', 94), ('e', 94), ('f', 95), ('d', 97), ('b', 100)]

获得多个词典的key的交集

In [99]: from random import randint, sample

In [100]: dd1 = {x:randint(90,100) for x in sample('abcdefghij', 5)}

In [101]: dd2 = {x:randint(90,100) for x in sample('abcdefghij', 5)}

In [102]: dd3 = {x:randint(90,100) for x in sample('abcdefghij', 5)}

In [103]: dd1
Out[103]: {'b': 100, 'c': 97, 'd': 100, 'e': 97, 'f': 92}

In [104]: dd2
Out[104]: {'a': 100, 'c': 90, 'g': 93, 'h': 93, 'j': 90}

In [105]: dd3
Out[105]: {'c': 96, 'e': 93, 'f': 91, 'h': 97, 'j': 90}

In [106]: mp = map(dict.viewkeys, (dd1, dd2, dd3))

In [107]: mp
Out[107]:
[dict_keys(['c', 'b', 'e', 'd', 'f']),
 dict_keys(['a', 'h', 'c', 'j', 'g']),
 dict_keys(['h', 'c', 'j', 'e', 'f'])]

In [108]: reduce(lambda x,y: x&y, mp)
Out[108]: {'c'}

怎样让字典按照插入有序

In [122]: from collections import OrderedDict

In [123]: d = OrderedDict()

In [124]: d['x'] = 1

In [125]: d['y'] = 2

In [126]: d['a'] = 2

In [127]: d['b'] = 2

In [128]: d
Out[128]: OrderedDict([('x', 1), ('y', 2), ('a', 2), ('b', 2)])

怎样实现长度为N的队列功能

In [138]: from collections import deque

In [139]: deque?
Docstring:
deque([iterable[, maxlen]]) --> deque object

Build an ordered collection with optimized access from its endpoints.
File:      /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py
Type:      type

In [141]: d = deque([], 3)

In [142]: d.append(1)

In [143]: d.append(2)

In [144]: d.append(3)

In [145]: d.append(4)

In [146]: d
Out[146]: deque([2, 3, 4])

迭代器

怎样齐头并进并行的遍历多个集合

In [147]: names = [x for x in 'abcdefg']

In [148]: ages = [x for x in range(21, 28)]

In [149]: scores = [randint(90,100) for x in range(7)]

In [150]: names
Out[150]: ['a', 'b', 'c', 'd', 'e', 'f', 'g']

In [151]: ages
Out[151]: [21, 22, 23, 24, 25, 26, 27]

In [152]: scores
Out[152]: [93, 90, 95, 97, 91, 93, 92]

In [153]:

In [153]: zip?
Docstring:
zip(seq1 [, seq2 [...]]) -> [(seq1[0], seq2[0] ...), (...)]

Return a list of tuples, where each tuple contains the i-th element
from each of the argument sequences.  The returned list is truncated
in length to the length of the shortest argument sequence.
Type:      builtin_function_or_method

In [154]: for name,age,score in zip(names, ages, scores):
     ...:     print name,age,score
     ...:
a 21 93
b 22 90
c 23 95
d 24 97
e 25 91
f 26 93
g 27 92

怎样串行的遍历多个集合

In [158]: lista = (randint(60,70) for x in range(10))

In [159]: list(lista)
Out[159]: [65, 60, 62, 64, 63, 60, 68, 67, 61, 62]

In [160]: listb = [randint(90,100) for x in range(20)]

In [161]: listb
Out[161]:
[98,
 96,
 97,
 98,
 95,
 95,
 90,
 99,
 92,
 92,
 99,
 92,
 100,
 95,
 100,
 100,
 93,
 91,
 92,
 98]

In [163]: from itertools import chain

In [164]: chain?
Docstring:
chain(*iterables) --> chain object

Return a chain object whose .next() method returns elements from the
first iterable until it is exhausted, then elements from the next
iterable, until all of the iterables are exhausted.
File:      /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
Type:      type

In [165]: for x in chain(lista, listb):
     ...:     print x,
     ...:
98 96 97 98 95 95 90 99 92 92 99 92 100 95 100 100 93 91 92 98

字符串

使用多种分隔符拆分字符串

In [166]: s = 'a,b;c/d'

In [167]: import re

In [169]: re.sub?
Signature: re.sub(pattern, repl, string, count=0, flags=0)
Docstring:
Return the string obtained by replacing the leftmost
non-overlapping occurrences of the pattern in string by the
replacement repl.  repl can be either a string or a callable;
if a string, backslash escapes in it are processed.  If it is
a callable, it's passed the match object and must return
a replacement string to be used.
File:      /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py
Type:      function

In [171]: re.sub(r'[,;/]', '-', s)
Out[171]: 'a-b-c-d'

如果进行字符串的模糊搜索与部分替换

In [172]: s = 'things happend in 2017-08-09, it is a sunddy'

In [175]: re.sub(r'(\d{4})-(\d{2})-(\d{2})', r'\2-\1-\3', s)
Out[175]: 'things happend in 08-2017-09, it is a sunddy'

列表JOIN时如果有数字元素怎么办

In [176]: print '\t'.join([str(x) for x in ['a','b',33,4.0,'e']])
a   b   33  4.0 e

文件

如何使用临时文件

In [186]: from tempfile import TemporaryFile,NamedTemporaryFile

In [187]: t = TemporaryFile()

In [188]: t.write('aa')

In [189]: t.close()

In [191]: NamedTemporaryFile?
Signature: NamedTemporaryFile(mode='w+b', bufsize=-1, suffix='', prefix='tmp', dir=None, delete=True)
Docstring:
Create and return a temporary file.
Arguments:
'prefix', 'suffix', 'dir' -- as for mkstemp.
'mode' -- the mode argument to os.fdopen (default "w+b").
'bufsize' -- the buffer size argument to os.fdopen (default -1).
'delete' -- whether the file is deleted on close (default True).
The file is created as mkstemp() would do it.

Returns an object with a file-like interface; the name of the file
is accessible as its 'name' attribute.  The file will be automatically
deleted when it is closed unless the 'delete' argument is set to False.
File:      /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tempfile.py
Type:      function

In [192]: t = NamedTemporaryFile()

In [193]: t.write("a")

In [194]: t.name
Out[194]: '/var/folders/sc/rpg0yq054hb7vdr83ms1rp2w0000gn/T/tmpQIONuU'

并发编程

如何使用多线程

In [8]: cat multi_threading.py
from threading import Thread

def func(x):
    print x, x*x*x

ts = []
for x in range(10):
    t = Thread(target=func, args=(x,))
    t.start()
    ts.append(t)

for t in ts:
    t.join()

print 'main thread over'

In [9]: %run multi_threading.py
0 0
1 1
2 8
3 27
4 64
5 125
6 216
7 343
8 512
9 729
main thread over

上一中是直接用函数执行，第二种是先创建一个类继承自Thread类

In [18]: cat multi_threading_class.py
from threading import Thread

class MyThread(Thread):
    def __init__(self, x):
        Thread.__init__(self)
        self.x = x

    def run(self):
        print self.x, self.x*self.x*self.x

ts = []
for x in range(10):
    t = MyThread(x)
    t.start()
    ts.append(t)

for t in ts:
    t.join()

print 'main thread over'

In [19]: %run multi_threading_class.py
0 0
1 1
2 8
3 27
4 64
 5 125
6 216
7 343
8 512
9 729
main thread over

线程间通信-生产者消费者模式

In [8]: cat producer_consumer.py
#coding:utf8

from threading import Thread,currentThread
from Queue import Queue
from time import sleep
from random import randint,sample
from itertools import chain

class Producer(Thread):
    def __init__(self, queue):
        Thread.__init__(self)
        self.queue = queue

    def run(self):
       for x in range(5):
            sleep(randint(1,3))
            ele = sample('abcdefg',1)[0]
            print 'producer %s: %s'%(currentThread().name, ele)
            self.queue.put(ele)

class Consumer(Thread):
    def __init__(self, queue):
        Thread.__init__(self)
        self.setDaemon(True)
        self.queue = queue

    def run(self):
        while(True):
            e = self.queue.get()
            sleep(1)
            print 'consumer %s: %s' % (currentThread().name, e)

queue = Queue()
tps = []
for x in range(3):
    tp = Producer(queue)
    tp.start()
    tps.append(tp)

for x in range(2):
    tc = Consumer(queue)
    tc.start()

for t in tps:
    t.join()

print 'main thread over'

In [9]: %run producer_consumer.py
producer Thread-301: a
^Cconsumer Thread-304: a
producer Thread-302: c
 producer Thread-303: g
producer Thread-301: c
consumer Thread-304: g
 consumer Thread-305: c
producer Thread-303: b