Three higher order functions, closures and decorators

The so - called higher - order function is a kind of function that takes function as parameter In addition, the function that returns the function is also a high - order function

3.1 pass function as parameter to another function

3.1.1 introduction of higher-order functions

Functions in Python are also objects:

>>> type(abs)
<class 'builtin_function_or_method'>

Therefore, the function can also be assigned to other variables. Of course, the variable can also be used like calling a function

>>> f=abs
>>> type(f)
<class 'builtin_function_or_method'>
>>> f(-1)
1

Since a function can be assigned to a variable, of course, it can also be used by passing another function as the parameter:

>>> def add(f, x, y):
...     return f(x)+f(y)
... 
>>> add(abs, -1, -1)
2

3.1.2 Python built-in high-order functions

map() and filter() are two built-in functions, which copy the function of generator expression So they all return an iterator

3.1.2.1 map()

map(func, *iterables), the prototype declared by the function, we can see that the map() function can receive one function and multiple iteratable objects

• When only one iteratable object is passed in, the function of map() is to apply func to each element of the iteratable object one by one, and the result is returned as an iterator:

>>> m = map(lambda x: x*x, range(10))
>>> type(m)
<class 'map'>
>>> from collections.abc import Iterable
>>> from collections.abc import Iterator
>>> isinstance(m, Iterable) # map is an iteratable object
True
>>> isinstance(m, Iterator) # map is also an iterator
True

Since map() returns an iterator instance, it can be used in a for loop:

>>> for n in m:
...     print(n)
... 
0
1
4
9
16
25
36
49
64
81

• map() can also accept multiple iteratable objects as parameters:

>>> m=map(lambda x, y: str(x)+str(y), ['x']*10, range(10))
>>> list(m)
['x0', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7', 'x8', 'x9']

3.1.2.2 filter()

filter(predicate, iterable), as can be seen from the function declaration prototype: filter() applies assertions to each element of an iteratable object, and returns the elements whose assertions are true as iterators

>>> def is_even(x):	# Judge a number as even
...     return x % 2 == 0
... 
>>> list(filter(is_even, range(10)))
[0, 2, 4, 6, 8]

In fact, most of the functions done by map() and filter() can be done by list inference

3.1.3 other commonly used high-order functions

• functools.reduce(func, iterable[, initializer]): use a function with two parameters to accumulate the elements in the iteratable object, from left to right, and finally return a calculated value

>>> reduce(lambda x,y: x*y, [1,2,3,4]) 
24
>>> reduce(lambda x,y: x+y, [1,2,3,4])  
10

3.2 return function

In addition to accepting functions as parameters, higher-order functions can also return functions as result values.

3.2.1 nesting of functions

When another function definition is placed in a function definition, it constitutes a nested definition of a function:

g_var = "hello global variable!" # Global variable
print(abs) # Built in variables
def outer():
    print(g_var)
    e_var="hello enclosed variable!" # Local variables (but also called enclosed variables)
    def inner():
        print(e_var)
        l_val="hello local variable!" # Local variable
        print(l_val)

    inner()

outer()

3.2.1.1 variable scope

With the definition of nested functions, we have to talk about the scope of variables in Python. The scope of variables in Python is divided into four types:

• Local variable: local scope, that is, the variables defined inside the function are local variables and also contain the parameters of the function

• Enclosed variable: closed scope. When the definition of another function is built into a function definition, the local variables in the outer function are called closed scope variables

• Global variable: global scope. The variables defined in the module are global variables

• Builtins variable: variable defined in Python builtins Here is the definition in builtins:

import builtins
>>> dir(builtins)
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'ModuleNotFoundError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError', 'RecursionError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopAsyncIteration', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'WindowsError', 'ZeroDivisionError', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

3.2.1.2 principles of finding variables by interpreter

The principle for Python interpreter to find variables is: L - > e - > G - > B

Analyze the code of the embedded function in the above code:

def inner():
    print(e_var)
    l_val="hello local variable!" # Local variable
    print(l_val)

When the interpreter encounters print(e_val), it first looks in the local scope (L), and the result is not defined, so it looks in the outer function definition, that is, in the closed scope (E), because e is defined in the outer function_ So you can output Hello!. variable

def outer():
    print(g_var)
    e_var="hello enclosed variable!" # Local variables (but also called enclosed variables)
    def inner():
        print(e_var)
        l_val="hello local variable!" # Local variable
        print(l_val)

From the above example, we can draw the following conclusions:

Python always starts to find variables in the scope. If the scope is not defined, it will find variables in the upper scope!

Here is a problematic Code:

g_var = "hello global variable!"
print(abs)
def outer():
    print(g_var)
    e_var="hello enclosed variable!"
    def inner():
       e_var = e_var + "+ local varialb"
       print(e_var)

    inner()

outer()

The above code will throw an exception: UnboundLocalError: local variable 'e_var' referenced before assignment

The exception is thrown in this line of code: e_var = e_var + "+ local varialb", the reason is as follows:

This is an assignment statement According to the operator priority, first perform the string + operation to the right of the equal sign, and then access the variable e here first_ VaR, so the interpreter starts to find the definition of the variable in this scope. If it cannot be found, it will find it outside. However, it must be noted that python defines variables through assignment statements. Therefore, there is actually a definition of variables here, but it violates the principle of defining before using! So an exception was thrown

3.2.1.3 scope promotion

The so-called variable scope promotion refers to the means to modify the variables in the closed scope in the nested function and the global variables in the closed scope

• nonlocal keyword is used to promote the scope within nested functions:

g_var = "hello global variable!"
print(abs)
def outer():
    print(g_var)
    e_var="hello enclosed variable!"
    def inner():
       nonlocal e_var
       e_var = e_var + "+ local varialb"
       print(e_var)

    inner()

outer()

The above code tells the interpreter through the nonlocal keyword that this is the e defined in the closed scope_ VaR, so the program can run correctly

Global keyword is used to promote a local variable to a global variable in a function:

g_var = "hello global variable!"
print(abs)
def outer():
    print(g_var)
    e_var="hello enclosed variable!"
    def inner():
        global g_var
        g_var = g_var + "+ local varialb"
        print(g_var)

    inner()

outer()

Similarly, the keyword global here is used to tell the interpreter g_var is defined by the global scope

3.2.2 closure

Simply put, a function definition refers to variables defined outside the function, and the function can be executed outside its definition environment. In other words, the closure must be the definition of nested function, and the internal function references the variables defined by the external function

def outer():
    i = 0
    def inner(name):
        nonlocal i
        i += 1
        print("{} The first {} Secondary operation output".format(name, i))
    return inner

In the above code, the function inner is defined in the outer function, and the internal function can access the parameters or local variables of the external function. When outer returns the function inner, the relevant parameters and variables are saved in the returned function. This program structure called closure has strong ability

Add test code as follows:

...
if __name__ == '__main__':

    f1 = outer()
    f1('f1')
    f1('f1')
    f1('f1')
    f2 = outer()
    f2('f2')
    f1('f1')
    f2('f2')

Run output:

f1 Output of the first operation
f1 Second run output
f1 3rd run output
f2 Output of the first operation
f1 Output of the 4th operation
f2 Second run output

From this example, we can see that in the closure, the outer variables referenced by the inner nested function are as follows:

The variables of the outer function are only associated with specific closures The variables of the outer function referenced by each instance of the closure do not interfere with each other

The modification of a closure instance's external variables will be passed to the next call of the closure instance

3.3 trimmer

Closures have many applications, among which decorators are a common scenario The function of decorator is usually used to enrich the function of an existing function Especially when you can't get the source code of the original function definition

3.3.1 simple decorator

For example, we want to add a log record every time we call a function, which is completed through the decorator. The code is as follows:

from datetime import datetime
def log(func):
    def wrapper(*args, **kwargs):
        print('call {}(): at {}'.format(func.__name__, datetime.now()))
        return func(*args, **kwargs)
    return wrapper

Observe the above log because it receives a function as a parameter and returns a function. In the inner function, first record the time of function call, and then call the function This is a decorator The use of decorators should be completed through @ syntax

@log
def hello():
    print('hello')

Put @ log in front of a function, then this function is a decorated function. It can not only complete the original functions, but also have the functions added by the decorator

>>> hello()
call hello(): at 2021-05-03 20:44:01.184195
hello

On the other hand, the @ log function is placed in the execution position:

now = log(now)

Since log() is a decorator and returns a function, the original now() function still exists, but now the now variable with the same name points to the new function, so calling now() will execute the new function, that is, the wrapper() function returned in the log() function.

The parameter definition of wrapper() function is wrapper(*args, **kw). Therefore, wrapper() function can accept calls with arbitrary parameters. In the wrapper() function, the log is printed first, and then the original function is called.

3.3.2 decorator with parameters

If the decorator itself needs to pass in parameters, it needs to write a high-order function that returns the decorator, which will be a little more complicated For example, a log decorator with custom text:

def log(text):
    def decorator(func):
        def wrapper(*args, **kwargs):
            print('{} call {}(): at {}'.format(text,func.__name__, datetime.now()))
            return func(*args, **kwargs)
        return wrapper
    return decorator

The decorator with parameters is used as follows. First, decorate a function through @ log(text):

@log('execute')
def hello():
    print('hello')

Then execute the function, and the result is as follows:

>>> hello()
execute call hello(): at 2021-05-04 07:54:17.025915
hello

Compared with two-layer nested decorators, the effect of three-layer nesting is equivalent to executing the following statement:

hello=log('execute')(hello)

In other words, first execute log('execute '), return the decorator function, then call the returned function, the parameter is the hello function, and the return value is finally the wrapper function

3.3.3 improvement of custom decorator

Although our decorator has been able to enhance the function of a function, there is another flaw, that is, the function__ name__ Attribute can still expose its original face, which should leave hidden dangers for some of the dependent function signatures

>>> hello.__name__
'wrapper'

Obviously, the bottom signature of the function hello is still wrapper, not hello!

Python's standard module functools can help us complete the required functions. A complete decorator is as follows:

import functools
from datetime import datetime

def log(text):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            print('{} call {}(): at {}'.format(text,func.__name__, datetime.now()))
            return func(*args, **kwargs)
        return wrapper
    return decorator

@log('execute')
def hello():
    print('hello')

The tests are as follows:

>>> hello()
execute call hello(): at 2021-05-04 08:13:22.007876
hello
>>> hello.__name__
'hello'

Note: decorator @ functools Wrap (func) must be placed in front of the wrapper