Python is a scripting language. Compared with C/C + +, it has some shortcomings in efficiency and performance. However, there are many times when the efficiency of Python is not as exaggerated as expected. Let's move on to the topic.
1. Code optimization principle
This article will introduce a lot of techniques to speed up the running of Python code. Before going into the details of code optimization, you need to understand some basic principles of code optimization.
The first basic principle is to weigh the cost of optimization. Optimization comes at a price, and it is almost impossible to solve all performance problems. The usual choice is time for space or space for time. In addition, development costs also need to be considered.
The second basic principle is not to optimize too early. Many people start writing code with the goal of performance optimization, "making the right program faster is much easier than making the fast program correct". Therefore, the premise of optimization is that the code can work normally. Premature optimization may ignore the grasp of the overall performance index. Don't reverse the primary and secondary before getting the global results.
The third principle is not to optimize those unimportant parts. If every part of the code is optimized, these modifications will make the code difficult to read and understand. If your code is running slowly, you should first find the place where the code is running slowly, usually an internal loop, and focus on optimizing where it is running slowly.
2. Avoid
2.1 avoid module and function attribute access
#Not recommended. Code time: 14.5 seconds import math def computeSqrt(size: int): result = [] for i in range(size): result.append(math.sqrt(i)) return result def main(): size = 10000 for _ in range(size): result = computeSqrt(size) main()
Every use (when an attribute accesses an operator) triggers specific methods, such as__ getattribute__ () and__ getattr__ (), these methods will perform dictionary operation, so they will bring additional time overhead. Attribute access can be eliminated through the from import statement.
#Optimize the writing method for the first time. Code time: 10.9 seconds from math import sqrt def computeSqrt(size: int): result = [] for i in range(size): result.append(sqrt(i)) #Avoid math Use of sqrt return result def main(): size = 10000 for _ in range(size): result = computeSqrt(size) main()
In Section 1, we mentioned that local variables can be searched faster than global variables. Therefore, for frequently accessed variables sqrt, we can speed up the operation by changing them to local variables.
#The second optimization writing method. Code time: 9.9 seconds import math def computeSqrt(size: int): result = [] sqrt = math.sqrt #Assign to local variable for i in range(size): result.append(sqrt(i)) #Avoid math Use of sqrt return result def main(): size = 10000 for _ in range(size): result = computeSqrt(size) main()
Except math In addition to sqrt, there is also in the computeSqrt function The existence of is to call the append method of list. By assigning this method to a local variable, the for loop in the computeSqrt function can be completely eliminated use.
#Recommended writing. Code time: 7.9 seconds import math def computeSqrt(size: int): result = [] append = result.append sqrt = math.sqrt #Assign to local variable for i in range(size): append(sqrt(i)) #Avoid} result Append and math Use of sqrt + return result def main(): size = 10000 for _ in range(size): result = computeSqrt(size) main()
2.2 avoid intra class attribute access
#Not recommended. Code time: 10.4 seconds import math from typing import List class DemoClass: def __init__(self, value: int): self._value = value def computeSqrt(self, size: int) -> List[float]: result = [] append = result.append sqrt = math.sqrt for _ in range(size): append(sqrt(self._value)) return result def main(): size = 10000 for _ in range(size): demo_instance = DemoClass(size) result = demo_instance.computeSqrt(size) main()
Avoid The principle of also applies to in class properties. Visit self_ Value is slower than accessing a local variable. By assigning an attribute in the class that needs to be accessed frequently to a local variable, the running speed of the code can be improved.
#Recommended writing. Code time: 8.0 seconds import math from typing import List class DemoClass: def __init__(self, value: int): self._value = value def computeSqrt(self, size: int) -> List[float]: result = [] append = result.append sqrt = math.sqrt value = self._value for _ in range(size): append(sqrt(value)) #Avoid self_ Use of value return result def main(): size = 10000 for _ in range(size): demo_instance = DemoClass(size) demo_instance.computeSqrt(size) main()
3. Avoid global variables
#Not recommended. Code time: 26.8 seconds import math size = 10000 for x in range(size): for y in range(size): z = math.sqrt(x) + math.sqrt(y)
At the beginning, many programmers will write some simple scripts in Python. When writing scripts, they are usually used to writing them directly as global variables, such as the above code. Because the definition of global variables and the implementation of local variables are different, they will run slower than the definition of global variables in the code. By putting script statements into functions, you can usually achieve a speed increase of 15% - 30%.
#Recommended writing. Code time: 20.6 seconds import math def main(): #Define into functions to reduce the use of all variables size = 10000 for x in range(size): for y in range(size): z = math.sqrt(x) + math.sqrt(y) main()
4. Avoid unnecessary abstraction
#Not recommended, code time: 0.55 seconds class DemoClass: def __init__(self, value: int): self.value = value @property def value(self) -> int: return self._value @value.setter def value(self, x: int): self._value = x def main(): size = 1000000 for i in range(size): demo_instance = DemoClass(size) value = demo_instance.value demo_instance.value = i main()
Any time you use additional processing layers (such as decorators, property access, descriptors) to wrap code, it slows down the code. In most cases, it is necessary to re-examine the definition of using attribute accessors. Using getter/setter functions to access attributes is usually a code style left over by C/C + + programmers. If it's not really necessary, use simple attributes.
#Recommended writing method, code time: 0.33 seconds class DemoClass: def __init__(self, value: int): self.value = value #Avoid unnecessary property accessors def main(): size = 1000000 for i in range(size): demo_instance = DemoClass(size) value = demo_instance.value demo_instance.value = i main()
5. Use the short-circuit characteristics of if conditions
#Not recommended, code time: 0.05 seconds
from typing import List
def concatString(string_list: List[str]) -> str:
abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
abbr_count = 0
result = ''
for str_i in string_list:
if str_i in abbreviations:
result += str_i
return result
def main():
for _ in range(10000):
string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = concatString(string_list)
main()
The short-circuit characteristic of the if # condition refers to the statement such as if a and b, which will be returned directly when a is False and b will not be calculated; For statements such as if a or b, when a is True, it will be returned directly and b will not be evaluated. Therefore, in order to save running time, for or statements, variables with high probability of True value should be written in front of or, and should be pushed back.
#Recommended writing method, code time: 0.03 seconds
from typing import List
def concatString(string_list: List[str]) -> str:
abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
abbr_count = 0
result = ''
for str_i in string_list:
if str_i[-1] == '.' and str_i in abbreviations: #Using the short-circuit characteristics of ＾ if ＾ condition
result += str_i
return result
def main():
for _ in range(10000):
string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = concatString(string_list)
main()
6. Avoid data replication
6.1 avoid meaningless data replication
#Not recommended, code time: 6.5 seconds def main(): size = 10000 for _ in range(size): value = range(size) value_list = [x for x in value] square_list = [x * x for x in value_list] main()
Value in the above code_ List is completely unnecessary, which creates unnecessary data structures or copies.
#Recommended writing method, code time: 4.8 seconds def main(): size = 10000 for _ in range(size): value = range(size) square_list = [x * x for x in value] #Avoid meaningless duplication main()
Another situation is that they are too paranoid about Python's data sharing mechanism, do not well understand or trust Python's memory model, and abuse Python copy Functions such as deepcopy(). Generally, the copy operation can be removed from these codes.
6.2 intermediate variables are not used when exchanging values
#Not recommended, code time: 0.07 seconds def main(): size = 1000000 for _ in range(size): a = 3 b = 5 temp = a a = b b = temp main()
The above code creates a temporary variable temp when exchanging values. Without the help of intermediate variables, the code is simpler and runs faster.
#Recommended writing method, code time: 0.06 seconds def main(): size = 1000000 for _ in range(size): a = 3 b = 5 a, b = b, a #Without the aid of intermediate variables main()
6.3 string splicing uses join instead of+
#Not recommended, code time: 2.6 seconds import string from typing import List def concatString(string_list: List[str]) -> str: result = '' for str_i in string_list: result += str_i return result def main(): string_list = list(string.ascii_letters * 100) for _ in range(10000): result = concatString(string_list) main()
When using a + b to splice strings, because the string in Python is an immutable object, it will apply for a piece of memory space and copy a and b into the newly applied memory space respectively. Therefore, if n# strings are to be spliced, n-1# intermediate results will be generated. Each intermediate result needs to apply for and copy memory, which seriously affects the operation efficiency. When using join() to splice strings, you will first calculate the total memory space to be applied for, then apply for the required memory at one time, and copy each string element into the memory.
#Recommended writing method, code time: 0.3 seconds import string from typing import List def concatString(string_list: List[str]) -> str: return ''.join(string_list) #Use ， join ， instead of ，+ def main(): string_list = list(string.ascii_letters * 100) for _ in range(10000): result = concatString(string_list) main()
7. Cycle optimization
7.1 replace while loop with for loop
#Not recommended. Code time: 6.7 seconds def computeSum(size: int) -> int: sum_ = 0 i = 0 while i < size: sum_ += i i += 1 return sum_ def main(): size = 10000 for _ in range(size): sum_ = computeSum(size) main()
Python's for loop is much faster than the while loop.
#Recommended writing. Code time: 4.3 seconds def computeSum(size: int) -> int: sum_ = 0 for i in range(size): #The for loop replaces the while loop sum_ += i return sum_ def main(): size = 10000 for _ in range(size): sum_ = computeSum(size) main()
7.2 use implicit for loop instead of explicit for loop
For the above example, you can further replace the explicit for loop with an implicit for loop
#Recommended writing. Code time: 1.7 seconds def computeSum(size: int) -> int: return sum(range(size)) #Implicit for loop replaces explicit for loop def main(): size = 10000 for _ in range(size): sum = computeSum(size) main()
7.3 reduce the calculation of inner for loop
#Not recommended. Code time: 12.8 seconds import math def main(): size = 10000 sqrt = math.sqrt for x in range(size): for y in range(size): z = sqrt(x) + sqrt(y) main()
In the above code, sqrt(x) is located in the inner for loop, which will be recalculated during each training, which increases the time overhead.
#Recommended writing. Code time: 7.0 seconds
import math
def main():
size = 10000
sqrt = math.sqrt
for x in range(size):
sqrt_x = sqrt(x) #Reduce the calculation of inner {for} loop
for y in range(size):
z = sqrt_x + sqrt(y)
main()
-Copyright notice: all articles on this blog except for special notices
In addition, the copyright belongs to the author. Reprint please indicate the source!