1. Process
Program: for example, xxx.py, which is a static program
Process: after a program runs, the resources used by code + are called processes. It is the basic unit for the operating system to allocate resources.
Not only can multitasking be completed through threads, but also processes
2. Status of the process
In work, the number of tasks is often greater than the number of cores of the cpu, that is, some tasks must be executing, while others are waiting for the cpu to execute, resulting in different states.
-
Ready status: the running conditions have been met and are waiting for cpu execution
-
Execution status: the cpu is executing its functions
-
Wait state: wait for certain conditions to be met. For example, a program sleep is in the wait state
Process creation - multiprocessing
Multiprocessing module is a cross platform version of multiprocessing module. It provides a Process class to represent a Process object. This object can be understood as an independent Process that can perform other things
1. Create process
import multiprocessing import time def test(): while True: print("--test--") time.sleep(1) if __name__ == "__main__": p = multiprocessing.Process(target=test) p.start() while True: print("--main--") time.sleep(1)
explain:
-
When creating a child Process, you only need to pass in an execution function and function parameters, create a Process instance, and start it with the start() method
2. Process pid
import multiprocessing import os def test(): print("child process running, PID =% d"% (OS. Getpid())) print("child process running ends") if __name__ == "__main__": print("parent process is running, PID =% d"% (OS. Getpid())) p = multiprocessing.Process(target=test) p.start()
The id of the currently running process can be obtained through the getpid() method in os.
3. The syntax structure of process is as follows
Process([group [, target [, name [, args [, kwargs]]]]])
-
target: if the function reference is passed, it can be considered that this child process will execute the code here
-
args: parameter passed to the function specified by target, passed in tuple
-
kwargs: pass named arguments to the function specified by target
-
Name: set a name for the process. You can not set it
-
Group: Specifies the process group, which is not used in most cases
Common methods for instance objects created by Process:
-
start(): start the child process instance (create child process)
-
is_alive(): determine whether the child process of the process is still alive
-
join([timeout]): whether to wait for the execution of the child process to end, or how many seconds to wait
-
terminate(): terminate the child process immediately regardless of whether the task is completed or not
Common properties of instance objects created by Process:
-
name: alias of the current process. The default is Process-N, and N is an integer incremented from 1
-
pid: pid of the current process (process number)
4. Pass parameters to the function specified by the child process
import multiprocessing import os import time def test(name, **kwargs): for i in range(10): print("child process running, name =% s, PID =% d"% (name, OS. Getpid())) print(kwargs) time.sleep(0.2) if __name__ == "__main__": p = multiprocessing.Process(target=test, args=("zhangsan",), kwargs={"xxoo": 666}) p.start() time.sleep(1) p.terminate() p.join()
Operation results:
The child process is running, name=zhangsan, pid=37751 {'xxoo': 666} The child process is running, name=zhangsan, pid=37751 {'xxoo': 666} The child process is running, name=zhangsan, pid=37751 {'xxoo': 666} The child process is running, name=zhangsan, pid=37751 {'xxoo': 666} The child process is running, name=zhangsan, pid=37751 {'xxoo': 666}
5. Global variables are not shared between processes
import multiprocessing import os import time g_nums = [11, 33] def test1(): "" "code to be executed by child processes" "" print("in test1, pid=%d, g_nums=%s", (os.getpid(), g_nums)) for i in range(4): g_nums.append(i) time.sleep(1) print("in test1, pid=%d, g_nums=%s", (os.getpid(), g_nums)) def test2(): "" "code to be executed by child processes" "" print("in test2, pid=%d, g_nums=%s", (os.getpid(), g_nums)) if __name__ == "__main__": p1 = multiprocessing.Process(target=test1) p1.start() p1.join() p2 = multiprocessing.Process(target=test2) p2.start()
Operation results:
in test1, pid=%d, g_nums=%s (37947, [11, 33]) in test1, pid=%d, g_nums=%s (37947, [11, 33, 0]) in test1, pid=%d, g_nums=%s (37947, [11, 33, 0, 1]) in test1, pid=%d, g_nums=%s (37947, [11, 33, 0, 1, 2]) in test1, pid=%d, g_nums=%s (37947, [11, 33, 0, 1, 2, 3]) in test2, pid=%d, g_nums=%s (37948, [11, 33])
Process and thread comparison
function
-
Process, can complete multiple tasks, such as running multiple QQ on one computer at the same time
-
Thread can complete multiple tasks, such as multiple chat windows in a QQ
Different definitions
-
Process is an independent unit for resource allocation and scheduling
-
Thread is an entity of a process and the basic unit of CPU scheduling and dispatching. It is a smaller basic unit that can run independently than a process. Thread basically does not own system resources, but only has some essential resources in operation (such as program counters, a set of registers and stacks), but it can share all the resources owned by the process with other threads belonging to the same process
difference
-
A program has at least one process, and a process has at least one thread
-
The division scale of threads is smaller than that of processes (less resources than processes), which makes the concurrency of multithreaded programs high.
-
The process has independent memory units during execution, and multiple threads share memory, which greatly improves the running efficiency of the program
-
Threads cannot execute independently and must exist in the process
-
A process can be understood as a pipeline in a factory, and the thread in it is the worker on the pipeline
Advantages and disadvantages
Threads and processes have their own advantages and disadvantages in use: thread execution overhead is small, but it is not conducive to resource management and protection; The process is the opposite.
Opening and closing principle:
Extend the function of the original function without modifying the original function and its calling mode
Modifications to the code are closed
The source code of the decorated function cannot be modified
You cannot modify the way decorated functions are called
Imagine the scene of using a gun in the game in a functional way
1 def game(): 2 print('Pressure bullet') 3 print('cock a gun') 4 print('Fire bullets') 5 game() 6 game() 7 game() 8 9 At this time, you need to add a sight to the gun,For example, it needs to be added when sniping long-range targets,Sniping short-range targets doesn't need to be added 10 At this point, the code above becomes the current code 11 12 def sight(): 13 print('Professional sniper sight') 14 game() 15 sight() 16 sight() 17 sight() 18 At this time, the design does not comply with the opening and closing principle(Because the original code and call name have been modified)
Decorator (dynamic agent in python)
Essence: is a closure
Composition: function + argument higher-order function + return value higher-order function + nested function + syntax sugar = decorator
Significance of existence: add new functions to the function without destroying the original function and the original function call
Writing method of general decorator:
1 def warpper(fn): # fn is the objective function, equivalent to func 2 def inner(*args,**kwargs): # Is the parameter of the objective function 3 '''Operation before executing the objective function''' 4 ret = fn(*args,**kwargs) # Call the objective function. ret is the return value of the objective function 5 '''Operation after executing the objective function''' 6 return ret # Return the return value of the objective function to ensure the normal end of the function 7 return inner 8 9 #Grammar sugar 10 @warpper #Equivalent to func = warpper(func) 11 def func(): 12 pass 13 func() #This is the inner function that is executed
The scene above is modified with the decorator
1 Mode 1 2 def game(): 3 print('Pressure bullet') 4 print('cock a gun') 5 print('Fire bullets') 6 7 def sight(fn): # fn receives a function 8 def inner(): 9 print('Install professional sniper sight') 10 fn() #Call the function passed in 11 print('make off with money') 12 return inner #Return function address 13 14 game = sight(game) #Pass the game function to the sight function 15 game() 16 17 Perform steps 18 The first step is to define two functions game()Is a normal function,sight()Function for decorator 19 Step 2 Definition game = sight(game)Equal to game Function is passed as an argument to sight(fn)Decorator function fn Formal parameter 20 Step 3 execute sight(fn),fn At parameter position,Equivalent to lower function game()Pass the parameter equal to fn 21 Step 4 execute inner function,then return hold inner The memory address of the function is returned to as a return value sight(game) 22 Step 5 then execute game(),Equivalent to execution inner function 23 Step 6,implement inner function,Print'Sniper mirror',implement fn()Formal parameter,because fn Formal parameter equals game function,So execute game()function,Print'Pressure bullet','Loading','Fire bullets' 24 Step 7 print'make off with money' 25 Step 8 return the printed results to game() 26 27 Mode II 28 def sight(fn): # fn receives a function 29 def inner(): 30 print('Install professional sniper sight') 31 fn() #Call the function passed in 32 print('make off with money') 33 return inner #Return function address 34 35 @sight #Equivalent to game = sight(game) 36 def game(): 37 print('Pressure bullet') 38 print('cock a gun') 39 print('Fire bullets') 40 game() 41 42 Perform steps 43 Step 1 execution sight(fn)function 44 Step 2 execute@sight,Equivalent to game Function and sight Decorator Association 45 The third step is to game Function is passed as an argument to sight(fn)Decorator function fn Formal parameter 46 Step 4 execute inner function,then return hold inner The memory address of the function is returned to as a return value@sight 47 Step 5 execute game()Equivalent to execution inner()function,because@sight amount to game = sight(game) 48 Step 6 print'Sight 49 Step 7 implementation fn function,because fn be equal to game function,So it will be executed game()function,Print'Pressure bullet','Loading','Fire bullets'.fn()Function execution completed 50 Step 8 print'make off with money' 51 Step 9 then return all printed results to game() 52 53 result 54 Install professional sniper sight 55 Pressure bullet 56 cock a gun 57 Fire bullets 58 make off with money
A simple decorator implementation
1 def warpper(fn): 2 def inner(): 3 print('Every time you execute the decorated function, you have to go through here first') 4 fn() 5 return inner 6 @warpper 7 def func(): 8 print('Yes func function') 9 func() 10 11 result 12 Every time you execute the decorated function, you have to go through here first 13 Yes func function
Decorator with one or more parameters
1 def sight(fn): #fn equals calling the game function 2 def inner(*args,**kwargs): #A tuple ("bob",123) was received 3 print('Start the game') 4 fn(*args,**kwargs) #All parameters received are scattered and passed to user and PWD 5 print('make off with money') 6 return inner 7 @sight 8 def game(user,pwd): 9 print('Login game user name and password:',user,pwd) 10 print('Pressure bullet') 11 print('cock a gun') 12 print('Fire bullets') 13 game('bob','123') 14 result 15 Start the game 16 Login game user name and password: bob 123 17 Pressure bullet 18 cock a gun 19 Fire bullets 20 make off with money
Dynamically pass one or more parameters to the decorator
1 def sight(fn): #Call the game function 2 def inner(*args,**kwargs): #A tuple ("bob",123) was received 3 print('Start the game') 4 fn(*args,**kwargs) #All parameters received are scattered and passed to normal parameters 5 print('make off with money') 6 return inner 7 @sight 8 def game(user,pwd): 9 print('Login game user name and password:',user,pwd) 10 print('Pressure bullet') 11 print('cock a gun') 12 print('Fire bullets') 13 return 'The game is displayed' 14 ret = game('bob','123') #Two parameters are passed to the decorator sight 15 print(ret) 16 17 @sight 18 def car(qq): 19 print('land QQ number%s'%qq) 20 print('Start chariot game') 21 ret2 = car(110110) #Passed a parameter to the decorator sight 22 print(ret2) 23 result 24 Start the game 25 Login game user name and password: bob 123 26 Pressure bullet 27 cock a gun 28 Fire bullets 29 make off with money 30 None 31 Start the game 32 land QQ No. 110110 33 Start chariot game 34 make off with money 35 None 36 You will find that the return value of both functions is None,But I game The return value is defined return 'The game is displayed',But they didn't return
Return value of decorator
1 Why do I define the return value,But the return value is still None And,Because even when I was game Defined in function return 'The game is displayed' 2 But there's only one in the decorator return inner Define return value,But this return value is returned inner The memory address of the function,Not at all inner 3 Function internal return So the default is None,So you should define a inner Function internal return Return value,And there is no variable that receives the return value, 4 So set ret = fn(*args,**kwargs)and return ret 5 6 def sight(fn): #Call the game function 7 def inner(*args,**kwargs): #A tuple ("bob",123) was received 8 print('Start the game') 9 ret = fn(*args,**kwargs) #All parameters received are scattered and passed to normal parameters 10 print('make off with money') 11 return ret 12 return inner 13 @sight 14 def game(user,pwd): 15 print('Login game user name and password:',user,pwd) 16 print('Pressure bullet') 17 print('cock a gun') 18 print('Fire bullets') 19 return 'The game is displayed' 20 ret = game('bob','123') #Two parameters are passed to the decorator sight 21 print(ret) 22 result 23 Start the game 24 Login game user name and password: bob 123 25 Pressure bullet 26 cock a gun 27 Fire bullets 28 make off with money 29 The game is displayed 30 31 32 Case 2 33 def wrapper_out(flag): #Parameters of the decorator itself 34 def wrapper(fn): #objective function 35 def inner(*args,**kwargs): #Parameters that the objective function needs to accept 36 if flag == True: 37 print('Ask a third party about the price') 38 ret = fn(*args,**kwargs) 39 print('Buy equipment') 40 return ret 41 else: 42 ret = fn(*args,**kwargs) 43 return ret 44 return inner 45 return wrapper 46 #Grammar sugar, @ decorator 47 @wrapper_out(True) 48 def func(a,b): #Decorated by wrapper 49 print(a,b) 50 print('gang up') 51 return 'func Return value' 52 abc = func('I am parameter 1','I am parameter 2') 53 print(abc) 54 result 55 Ask a third party about the price 56 I'm parameter 1 I'm parameter 2 57 gang up 58 Buy equipment 59 func Return value
Multiple decorators use the same function
1 def wrapper1(fn): 2 def inner(*args,**kwargs): 3 print('wrapper1-1') 4 ret = fn(*args,**kwargs) 5 print('wrapper1-2') 6 return ret 7 return inner 8 9 def wrapper2(fn): 10 def inner(*args,**kwargs): 11 print('wrapper2-1') 12 ret = fn(*args,**kwargs) 13 print('wrapper2-2') 14 return ret 15 return inner 16 17 def wrapper3(fn): 18 def inner(*args,**kwargs): 19 print('wrapper3-1') 20 ret = fn(*args,**kwargs) 21 print('wrapper3-2') 22 return ret 23 return inner 24 @wrapper1 25 @wrapper2 26 @wrapper3 27 def func(): 28 print('I'm testing Xiaobai') 29 func() 30 result 31 wrapper1-1 32 wrapper2-1 33 wrapper3-1 34 I'm testing Xiaobai 35 wrapper3-2 36 wrapper2-2 37 wrapper1-2
python list type
Classification: python
Introduction to list types
A list type is a container that can store any number and type of data.
For example, in the following lists, there are stored values, strings, and embedded lists. Moreover, any other type can be stored.
>>> L = [1, 2, 3, 4] >>> L = ["a", "b", "c", "d"] >>> L = [1, 2, "c", "d"] >>> L = [[1, 2, 3], "a", "b", [4, "c"]]
The list in python is a sequence in which the elements are stored according to the index order, and can be used for index value taking, slicing and other operations.
List structure
A list is a variable object. You can modify the elements in the list in place without changing any metadata in the list.
>>> L = ["a", "b", "c"] >>> id(L), id(L[0]) (57028736, 55712192) >>> L[0] = "aa" >>> id(L), id(L[0]) (57028736, 56954784)
From the change of id, when modifying the first element of the list, the id of the list itself does not change, but the id of the first element of the list has changed.
It's easy to understand by looking at the memory diagram in the following list.
The above is an illustration of the L = ["a", "b", "c"] list. The variable name l stores the memory address of the list. The list contains metadata such as type declaration and list length, as well as the memory addresses of the three elements belonging to the list. It should be noted that list elements do not exist directly in the list range, but are saved in the list in the form of addresses.
Therefore, when modifying the elements in the list, create a new element "aa" (because the string is an immutable type). The list itself does not change, but changes the address pointing to the first element in the list to the address of the new data "aa".
Because modifying the list data will not change the properties of the list itself, this behavior is called "modify in place".
Therefore, the list has several main features:
-
Any type of data can be stored and nested in the list
-
The list stores the element reference, that is, the address of each element, so it is a list variable object
-
The list is a variable sequence. Therefore, each element has a positional order, which can be obtained by index and sub list by slicing
Construction list
There are two common ways to construct lists:
-
Use brackets []
-
Use the list() construction method
When building a list with (middle) brackets, the elements of the list can be written across lines, which is a feature of various bracket types in python syntax.
For example:
>>> [1,2,3] [1, 2, 3] >>> L = [ 1, 2, 3 ] >>> list('abcde') ['a', 'b', 'c', 'd', 'e'] >>> list(range(0, 4)) [0, 1, 2, 3]
The above range() is used to generate a series of values, just like the seq command under Linux. However, range() does not generate data directly. Instead, it returns an iteratable object, indicating that these data can be generated one by one. Therefore, list() is used here to generate all the data of range() and form a list.
There is a very important feature of constructing lists in brackets: List parsing, which is also called "list push to" in many places. For example:
>>> [x for x in 'abcdef'] ['a', 'b', 'c', 'd', 'e', 'f']
list() is to directly construct all the given data at one time and store the whole list object directly in memory. List derivation is faster than list() in constructing lists, and the performance gap is quite large.
List basic operations
The list supports + * symbol operations:
>>> L = [1,2,3,4] >>> L1 = ['a','b','c'] >>> L + L1 [1, 2, 3, 4, 'a', 'b', 'c'] >>> [1,2] + list("34") [1, 2, '3', '4'] >>> L * 2 [1, 2, 3, 4, 1, 2, 3, 4] >>> 2 * L [1, 2, 3, 4, 1, 2, 3, 4]
Binary assignment can be performed by + =:
>>> L1 = [1,2,3,4] >>> L2= [5,6,7,8] >>> L1 += L2 >>> L1 [1, 2, 3, 4, 5, 6, 7, 8]
The assignment method of L1 += L2 has better performance than that of L1 = L1 + L2 for variable sequences (such as the list here). The former directly modifies the original address of L1, and the latter creates a new list object and copies the original L1 list. But in fact, the performance gap is very small. As mentioned earlier, the list stores element references, so copying only copies some references, not actual data objects.
A list is a sequence, and each element of the sequence type is stored according to the index position, so the list elements can be obtained by index:
>>> L = [1,2,3,4,5] >>> L[0] 1 >>> L = [ ... [1,2,3,4], ... [11,22,33,44], ... [111,222,333,444] ... ] >>> L[0][2] 3 >>> L[1][2] 33 >>> L[2][2] 333
Of course, you can also assign values to a given element by index to modify the list:
>>> L = [1,2,3,4,5] >>> L[0] = 11
When modifying list elements by assignment, you can not only modify single element assignment, but also multi-element slice assignment.
>>> L[1:3] = [22,33,44,55] >>> L [11, 22, 33, 44, 55, 4, 5]
When assigning values to the slices of the list above, it is actually a process of obtaining these elements, deleting them, and inserting new data. Therefore, the above is to delete the element of [1:3], and then insert new list data at this position.
Therefore, if a slice is assigned to an empty list, it means that the element or the element of this range is deleted directly.
>>> L [11, 22, 33, 44] >>> L[1:3] = [] >>> L [11, 44]
However, if an empty list is assigned to a single index element, it does not mean that the element is deleted, but that the empty list is nested in the list as an element.
>>> L = [1,2,3,4] >>> L[0] = [] >>> L [[], 2, 3, 4]
The difference between the two list assignment should not be difficult to straighten out after understanding the list structure mentioned above.
List other operations
A list is a sequence, so it can be used for sequence operations, such as index, slice, functions available for various sequences (such as append(), extend(), remove(), del, copy(), pop(), reverse()), etc. See python sequence operation for details
In addition to these sequence general operations, the list has a special list method sort, which is used to sort the list.
List sorting sort() and sorted()
sort() is a list type method, which is only applicable to lists; sorted() is a built-in function that supports various container types. They can all be sorted, and the usage is similar, but sort() sorts in place and does not return the sorted list, while sorted() returns a new sorted list.
>>> help(list.sort) Help on method_descriptor: sort(...) L.sort(key=None, reverse=False) -> None -- stable sort *IN PLACE* >>> help(sorted) Help on built-in function sorted in module builtins: sorted(iterable, /, *, key=None, reverse=False) Return a new list containing all items from the iterable in ascending order. A custom key function can be supplied to customize the sort order, and the reverse flag can be set to request the result in descending order.
This article only briefly introduces the usage of sorting.
For example, list L:
>>> L = ['python', 'shell', 'Perl', 'Go', 'PHP']
Use sort() and sorted() to sort L. note that sort() sorts l directly in place and does not reflect the sorting result through the return value, so there is no need to assign a value to the variable. Sorted () returns a new sorted result, which needs to be assigned to a variable to save the sorted result.
>>> sorted(L) ['Go', 'PHP', 'Perl', 'python', 'shell'] >>> L ['python', 'shell', 'Perl', 'Go', 'PHP'] >>> L.sort() >>> L ['Go', 'PHP', 'Perl', 'python', 'shell']
It is not difficult to find that sort() and sorted() are sorted in ascending order by default (a < B <... < Z < a < B <... < z). Both of them can specify the parameter reverse=True to indicate the order reversal, that is, the descending order is obtained by default:
>>> L.sort(reverse=True) >>> L ['shell', 'python', 'Perl', 'PHP', 'Go']
In python 3.x, sort() and sorted() do not allow sorting lists containing different data types. That is, if there are both numeric values and strings in the list, the sorting operation will report an error.
Another parameter of sort() and sorted() is key, which is key=None by default. This parameter is used to specify a custom sorting function to implement the sorting rules you need.
For example, the above list is no longer sorted by the default character order, but by the length of the string. Therefore, customize this sorting function:
>>> def sortByLen(s): ... return len(s)
Then call sort() or sorted() by specifying the parameter key = sortByLen, during which you can also specify reverse = True:
>>> L = ['shell', 'python', 'Perl', 'PHP', 'Go'] >>> sorted(L,key=sortByLen) ['Go', 'PHP', 'Perl', 'shell', 'python'] >>> L.sort(key=sortByLen,reverse=True) >>> L ['python', 'shell', 'Perl', 'PHP', 'Go']
For another example, sort by the second character of each element of the list.
def f(e): return e[1] L = ['shell', 'python', 'Perl', 'PHP', 'Go'] sorted(L, key=f) L.sort(key=f)
For more sorting methods, see sorting HOWTO. For example, specify two sorting criteria. One is sorted in ascending order according to the length of the string, and the same length is sorted in descending order according to the second character. The usage is actually very simple, but it takes a little space, so this article won't explain it.
List iteration and parsing
The list is a sequence that can be tested with in and iterated with for.
For example:
>>> L = ["a","b","c","d"] >>> 'c' in L True >>> for i in L: ... print(i) ... a b c d
Besides list parsing, it refers to applying an expression to each element in the sequence (such as the list here), and taking the calculated result of the expression as a new sequence element (such as the list here).
For a more popular explanation, take the list sequence as an example. First, take the elements of the list, operate the elements taken each time, and put the results obtained after the operation into a new list.
Because the parsing operation adds one element to a new list, it is also called "list derivation", which means that the list is derived from the elements.
The simplest way is to take out the characters in the string sequence and put them in the list:
>>> [ i for i in "abcdef" ] ['a', 'b', 'c', 'd', 'e', 'f']
Here is list parsing, because the brackets [] are used outside it, indicating that the elements after the operation are put into the new list. You can replace braces with curly braces, which becomes collection parsing or even dictionary parsing. But note that there is no direct tuple parsing, because the bracket of tuple is special, it will be considered as the priority surrounding bracket of expression, rather than the tuple construction symbol.
Take out the elements and do some operations on each element:
>>> [ i * 2 for i in "abcdef" ] ['aa', 'bb', 'cc', 'dd', 'ee', 'ff'] >>> L = [1,2,3,4] >>> [ i * 2 for i in L ] [2, 4, 6, 8] >>> [ (i * 2, i * 3) for i in L ] [(2, 3), (4, 6), (6, 9), (8, 12)]
Parsing operations are closely related to for, and can be rewritten into a for loop. For example, the results of the following two statements are consistent:
[ i * 2 for i in "abcdef" ] L = [] for i in "abcdef": L.append(i * 2)
However, the performance of parsing operation is better than that of for loop, which is in line with the concept of simplicity and efficiency.
People who have learned other languages may have thought that the expression operation of each element in the parsing process is similar to the callback function. In fact, there is a special map() function in python, which takes the first parameter as the callback function and returns an iteratable object. In other words, the same results as parsing can be achieved. For example:
>>> def f(x):return x * 2 ... >>> list(map(f,[1,2,3,4])) [2, 4, 6, 8]
The map() function will be explained in detail in later articles.