1. Tools for functools to manage functions
The functools module provides tools to adjust or extend functions and other callable objects without requiring a complete override.
1.1 Modifier
The main tool provided by the functools module is the partial class, which can be used to "wrap" a callable object with default parameters.The resulting object itself is callable and can be thought of as the original function.It has exactly the same parameters as the original function and can be called with additional locations or named functions.You can use partial instead of lambda to provide default parameters to a function, some of which can be unspecified.
1.1.1 Partial Objects
The first example shows two simple partial objects for the function myfunc().The output of show_details() contains the func, args, and keywords attributes of this partial object.
import functools def myfunc(a, b=2): "Docstring for myfunc()." print(' called myfunc with:', (a, b)) def show_details(name, f, is_partial=False): "Show details of a callable object." print('{}:'.format(name)) print(' object:', f) if not is_partial: print(' __name__:', f.__name__) if is_partial: print(' func:', f.func) print(' args:', f.args) print(' keywords:', f.keywords) return show_details('myfunc', myfunc) myfunc('a', 3) print() # Set a different default value for 'b', but require # the caller to provide 'a'. p1 = functools.partial(myfunc, b=4) show_details('partial with named default', p1, True) p1('passing a') p1('override b', b=5) print() # Set default values for both 'a' and 'b'. p2 = functools.partial(myfunc, 'default a', b=99) show_details('partial with defaults', p2, True) p2() p2(b='override b') print() print('Insufficient arguments:') p1()
At the end of this example, the first partial created earlier is called, but no value is passed in for a, which results in an exception.
1.1.2 Getting function properties
By default, the partial object does not have u name_ or u doc_u attributes.Without these properties, the modified function will be more difficult to debug.Use update_wrapper() to copy or add attributes to the partial object from the original function.
import functools def myfunc(a, b=2): "Docstring for myfunc()." print(' called myfunc with:', (a, b)) def show_details(name, f): "Show details of a callable object." print('{}:'.format(name)) print(' object:', f) print(' __name__:', end=' ') try: print(f.__name__) except AttributeError: print('(no __name__)') print(' __doc__', repr(f.__doc__)) print() show_details('myfunc', myfunc) p1 = functools.partial(myfunc, b=4) show_details('raw wrapper', p1) print('Updating wrapper:') print(' assign:', functools.WRAPPER_ASSIGNMENTS) print(' update:', functools.WRAPPER_UPDATES) print() functools.update_wrapper(p1, myfunc) show_details('updated wrapper', p1)
The attributes added to the wrapper are defined in WRAPPER_ASSIGNMENTS, and WARPPER_UPDATES lists the values to be modified.
1.1.3 Other callable
partial applies to any callable object, not just stand-alone functions.
import functools class MyClass: "Demonstration class for functools" def __call__(self, e, f=6): "Docstring for MyClass.__call__" print(' called object with:', (self, e, f)) def show_details(name, f): "Show details of a callable object." print('{}:'.format(name)) print(' object:', f) print(' __name__:', end=' ') try: print(f.__name__) except AttributeError: print('(no __name__)') print(' __doc__', repr(f.__doc__)) return o = MyClass() show_details('instance', o) o('e goes here') print() p = functools.partial(o, e='default for e', f=8) functools.update_wrapper(p, o) show_details('instance wrapper', p) p()
This example creates a partial object from a class instance that contains the u call_() method.
1.1.4 Methods and Functions
partial() returns a callable that can be used directly, while partialmethod() returns a callable that can be used as a unbound method for objects.In the following example, this stand-alone function is added as a property of MyClass twice, using partialmethod () as method1(), and partialmethod () as method2() once.
import functools def standalone(self, a=1, b=2): "Standalone function" print(' called standalone with:', (self, a, b)) if self is not None: print(' self.attr =', self.attr) class MyClass: "Demonstration class for functools" def __init__(self): self.attr = 'instance attribute' method1 = functools.partialmethod(standalone) method2 = functools.partial(standalone) o = MyClass() print('standalone') standalone(None) print() print('method1 as partialmethod') o.method1() print() print('method2 as partial') try: o.method2() except TypeError as err: print('ERROR: {}'.format(err))
method1() can be called from an instance of MyClass that is passed in as the first parameter, the same as the method defined by the usual method.method2() is not defined as a binding method, so the self parameter must be passed explicitly; otherwise, this call will result in a TypeError.
1.1.5 Get function properties of modifiers
Updating the properties of the wrapped callable is particularly useful for modifiers, as the converted function will end up with the properties of the original "bare" function.
import functools def show_details(name, f): "Show details of a callable object." print('{}:'.format(name)) print(' object:', f) print(' __name__:', end=' ') try: print(f.__name__) except AttributeError: print('(no __name__)') print(' __doc__', repr(f.__doc__)) print() def simple_decorator(f): @functools.wraps(f) def decorated(a='decorated defaults', b=1): print(' decorated:', (a, b)) print(' ', end=' ') return f(a, b=b) return decorated def myfunc(a, b=2): "myfunc() is not complicated" print(' myfunc:', (a, b)) return # The raw function show_details('myfunc', myfunc) myfunc('unwrapped, default b') myfunc('unwrapped, passing b', 3) print() # Wrap explicitly wrapped_myfunc = simple_decorator(myfunc) show_details('wrapped_myfunc', wrapped_myfunc) wrapped_myfunc() wrapped_myfunc('args to wrapped', 4) print() # Wrap with decorator syntax @simple_decorator def decorated_myfunc(a, b): myfunc(a, b) return show_details('decorated_myfunc', decorated_myfunc) decorated_myfunc() decorated_myfunc('args to decorated', 4)
functools provide a modifier wraps(), which applies update_wrapper() to the modified function.
1.2 Comparison
In Python 2, a class can define a u cmp_u() method that returns -1, 0, or 1.Python 2.1 depending on whether the object is less than, for, or greater than the element being compared. A rich comparison method API (u lt_(), u le_(), u eq_(), u ne_(), u gt_() and u ge_()) is introduced, which can complete a comparison operation and return a Boolean value.Python 3 obsoletes u cmp_() and replaces it with these new methods, and functools provide tools to make it easier to write classes that meet the new requirements, that is, to meet the new comparison requirements in Python 3.
1.2.1 Rich Comparison
Rich Compare API s are designed to support classes involving complex comparisons to implement individual tests in the most efficient way.However, if the classes are relatively simple, there is no need to manually create the various price-to-money methods.The total_ordering() class modifier adds the remaining methods to a class that provides some methods.
import functools import inspect from pprint import pprint @functools.total_ordering class MyObject: def __init__(self, val): self.val = val def __eq__(self, other): print(' testing __eq__({}, {})'.format( self.val, other.val)) return self.val == other.val def __gt__(self, other): print(' testing __gt__({}, {})'.format( self.val, other.val)) return self.val > other.val print('Methods:\n') pprint(inspect.getmembers(MyObject, inspect.isfunction)) a = MyObject(1) b = MyObject(2) print('\nComparisons:') for expr in ['a < b', 'a <= b', 'a == b', 'a >= b', 'a > b']: print('\n{:<6}:'.format(expr)) result = eval(expr) print(' result of {}: {}'.format(expr, result))
This class must provide an implementation of u eq_() and another rich comparison method.This modifier increases the implementation of the remaining methods, which use the comparison provided.If a comparison cannot be completed, this method should return NotImplemented to attempt a comparison using the inverse comparison operator on another object, and if it is still not possible, it will fail completely.
1.2.2 pairing sequence
Since Python 3 obsoletes old-fashioned comparison functions, CMP parameters are no longer supported in functions such as sort().For older programs that use comparison functions, you can use cmp_to_key() to convert the comparison function to a function that returns a collation key, which determines the position of the element in the final sequence.
import functools class MyObject: def __init__(self, val): self.val = val def __str__(self): return 'MyObject({})'.format(self.val) def compare_obj(a, b): """Old-style comparison function. """ print('comparing {} and {}'.format(a, b)) if a.val < b.val: return -1 elif a.val > b.val: return 1 return 0 # Make a key function using cmp_to_key() get_key = functools.cmp_to_key(compare_obj) def get_key_wrapper(o): "Wrapper function for get_key to allow for print statements." new_key = get_key(o) print('key_wrapper({}) -> {!r}'.format(o, new_key)) return new_key objs = [MyObject(x) for x in range(5, 0, -1)] for o in sorted(objs, key=get_key_wrapper): print(o)
Normally, cmp_to_key() can be used directly, but this example introduces an additional wrapper function that allows you to print more information when calling key functions.
As shown in the output, sorted() first calls get_key_wrapper() on each element in the sequence to generate a key.The key returned by cmp_to_key() is an instance of a class defined in functools that uses the old-fashioned comparison function passed in to implement the rich comparison API.After all keys have been created, sort the sequences by comparing them.
1.3 Cache
The lru_cache() modifier wraps a function in a "least recently used" cache.The parameters of the function are used to create a hash key and then map to the result.Subsequent calls that have the same parameters will get values from this cache without calling the function again.This modifier also adds a method for the function to check the status of the cache (cache_info()) and empty the cache (cache_clear()).(
import functools @functools.lru_cache() def expensive(a, b): print('expensive({}, {})'.format(a, b)) return a * b MAX = 2 print('First set of calls:') for i in range(MAX): for j in range(MAX): expensive(i, j) print(expensive.cache_info()) print('\nSecond set of calls:') for i in range(MAX + 1): for j in range(MAX + 1): expensive(i, j) print(expensive.cache_info()) print('\nClearing cache:') expensive.cache_clear() print(expensive.cache_info()) print('\nThird set of calls:') for i in range(MAX): for j in range(MAX): expensive(i, j) print(expensive.cache_info())
This example executes multiple expensive() calls in a set of nested loops.The second call has the same parameter value and the result is in the cache.These values must be recalculated when the cache is emptied and the loop is run again.
To avoid an unrestricted expansion of the cache due to a long-running process, specify a maximum size.The default is 128 elements, but you can change this size with the maxsize parameter for each cache.(
import functools @functools.lru_cache(maxsize=2) def expensive(a, b): print('called expensive({}, {})'.format(a, b)) return a * b def make_call(a, b): print('({}, {})'.format(a, b), end=' ') pre_hits = expensive.cache_info().hits expensive(a, b) post_hits = expensive.cache_info().hits if post_hits > pre_hits: print('cache hit') print('Establish the cache') make_call(1, 2) make_call(2, 3) print('\nUse cached items') make_call(1, 2) make_call(2, 3) print('\nCompute a new value, triggering cache expiration') make_call(3, 4) print('\nCache still contains one old item') make_call(2, 3) print('\nOldest item needs to be recomputed') make_call(1, 2)
In this example, the cache size is set to two elements.With a third set of different parameters (3,4), the oldest element in the cache is cleared and replaced with this new result.
The keys in the cache managed by lru_cache() must be hashed, so for a function wrapped in a cache lookup, all its parameters must be hashed.
import functools @functools.lru_cache(maxsize=2) def expensive(a, b): print('called expensive({}, {})'.format(a, b)) return a * b def make_call(a, b): print('({}, {})'.format(a, b), end=' ') pre_hits = expensive.cache_info().hits expensive(a, b) post_hits = expensive.cache_info().hits if post_hits > pre_hits: print('cache hit') make_call(1, 2) try: make_call([1], 2) except TypeError as err: print('ERROR: {}'.format(err)) try: make_call(1, {'2': 'two'}) except TypeError as err: print('ERROR: {}'.format(err))
If an object that cannot be hashed is passed into this function, a TypeError is produced.
1.4 Reduced Dataset
The reduce() function takes a callable and a data sequence as input.It invokes this callable with the values in the sequence and adds up the resulting output to produce a single value as output.(
import functools def do_reduce(a, b): print('do_reduce({}, {})'.format(a, b)) return a + b data = range(1, 5) print(data) result = functools.reduce(do_reduce, data) print('result: {}'.format(result))
This example adds up the numbers in the sequence.
The optional initializer parameter is placed at the top of the sequence and handled like any other element.This parameter can be used to update previously calculated values with new inputs.(
import functools def do_reduce(a, b): print('do_reduce({}, {})'.format(a, b)) return a + b data = range(1, 5) print(data) result = functools.reduce(do_reduce, data, 99) print('result: {}'.format(result))
In this example, reduce() is initialized with the previous sum of 99.
Without the initializer parameter, the sequence of only one element is automatically reduced to this value.An empty list generates an error unless an initializer parameter is provided.
import functools def do_reduce(a, b): print('do_reduce({}, {})'.format(a, b)) return a + b print('Single item in sequence:', functools.reduce(do_reduce, [1])) print('Single item in sequence with initializer:', functools.reduce(do_reduce, [1], 99)) print('Empty sequence with initializer:', functools.reduce(do_reduce, [], 99)) try: print('Empty sequence:', functools.reduce(do_reduce, [])) except TypeError as err: print('ERROR: {}'.format(err))
Since the initializer parameter is equivalent to a default value but is also combined with a new value (if the input sequence is not empty), it is important to carefully consider whether this parameter is used appropriately.If it makes no sense to combine the default with the new value, it is better to capture the TypeError instead of passing in a initializer parameter.
1.5 Generic Functions
In Python-like dynamic type languages, slightly different operations are usually required based on the type of parameter, especially when dealing with differences between element lists and individual elements.While it is easy to directly check the type of a parameter, there are cases where behavioral differences may be isolated into a single function, for which functools provide a singledispatch() modifier to register a set of generic functions that can be automatically switched based on the first parameter type of the function.(
import functools @functools.singledispatch def myfunc(arg): print('default myfunc({!r})'.format(arg)) @myfunc.register(int) def myfunc_int(arg): print('myfunc_int({})'.format(arg)) @myfunc.register(list) def myfunc_list(arg): print('myfunc_list()') for item in arg: print(' {}'.format(item)) myfunc('string argument') myfunc(1) myfunc(2.3) myfunc(['a', 'b', 'c'])
The register() property of the new function corresponds to another modifier for registering alternative implementations.The first function wrapped in singledispatch() is the default implementation, which is used when no other type of specific function is specified, in this case float.
When no exact match of this type is found, the inheritance order is calculated and the closest match type is used.(
import functools class A: pass class B(A): pass class C(A): pass class D(B): pass class E(C, D): pass @functools.singledispatch def myfunc(arg): print('default myfunc({})'.format(arg.__class__.__name__)) @myfunc.register(A) def myfunc_A(arg): print('myfunc_A({})'.format(arg.__class__.__name__)) @myfunc.register(B) def myfunc_B(arg): print('myfunc_B({})'.format(arg.__class__.__name__)) @myfunc.register(C) def myfunc_C(arg): print('myfunc_C({})'.format(arg.__class__.__name__)) myfunc(A()) myfunc(B()) myfunc(C()) myfunc(D()) myfunc(E())
In this example, classes D and E do not exactly match any registered generic function, and the selected function depends on the class hierarchy.
