Python -- iterators and generators

An iterator is an object that remembers where to traverse. The iterator object is accessed from the first element of the collection until all the elements are accessed. Iterators can only move forward, not backward.
Two basic methods. iter() and {next()

iter() creates an iterator object, {next() returns the next element of the iterator. When the iterator is already the last element, next() will throw an exception StopIteration
The iterator can also be iterated. When iterated by for in, it will not report an exception and will automatically stop when the exception occurs.
The iterator changes as the source object changes

To understand iterators and generators, we must first understand three words conceptually, I iterator / generator / iteratable

Iterator # iterator is an iterator class. The iterator usually refers to the iterator object.

Generator generator is a generator class. Usually, the generator refers to the generator object.

Iteratable class: iteratable is an iteratable class.

Iterator class and generator class are subclasses of iteratable class, that is, they are iteratable, and their objects are children of iteratable class.

Iteratable classes, such as arrays, dictionaries, etc. common objects that can be put into a for loop for iteration are sub objects of the iteratable class.

In addition to putting in the for loop, what can be put in iter() must also be iteratable.

The generator class is a subclass of the iterator class. There is little difference between the two, and the generator can be regarded as a special iterator.

In fact, the iter() function is used to generate an iterable object into an iterator (iterator object).

The key to becoming an iterable object is to implement it__ iter__ () constructor, which is called automatically during iteration__ inter__ You can write nothing, but not nothing. If you write it, it must return an iterator object (itself or others). For example, there are in arrays and dictionary classes__ iter__ () function. Although the user-defined generator (with yield) cannot see this function, the parent class is also implemented and will be called automatically.

To become an iterator object, that is, to become an iterator object directly__ inter__ () and__ next__ () both are indispensable

, because the iterator should be able to execute directly through next(), it is commonly understood that, for example, the list itself is an iteratable object, but it is not an iterator. It is impossible to directly next([1,2,3]) through next(iter([1,2,3])

The system built-in function iter() uses this object at runtime__ inter__ () method. So both methods can be tested__ inter__ () or inter(test)

The system built-in function next() uses this object at runtime__ next__ () method. So both methods can be tested__ next__ () or next(test)

Use and of next()__ inter__ It doesn't matter, anything is realized__ next__ Method can be called by next ().

The for loop will also become an iterator of the iterated object (whether it is implemented with its own or in the parent class), After generation, use__ next__ () the method iterates sequentially until the end.

Extension: List / dictionary generated iterators are not standard iterator classes, but are inherited from each other, such as list_iterator

Popular understanding --:

An iteratable object refers to an object that can be iterated and has the potential to be iterated (realizing _inter _), Nothing else can be done,

Iterators are like data streams, like view objects. After they are created, they will be affected by the source data. They can only be executed sequentially by next or

To put in for. It actually saves algorithms (referring to standard iterators, not custom class objects), and features an inert calculation sequence, which can perform calculations at any time,

Because of this, it is equivalent to holding elements of infinite length. For example, you can customize the iterator and generator, and then write a while loop, which can be executed all the time.

Popular understanding 2 --: an iterator is like a special algorithm + reference generated according to an iteratable object (occupying little space). As the name suggests, an iterator calculates the next value of the reference object according to the algorithm each time it is executed. For standard built-in iterators (such as those generated by lists / dictionaries), each next is to take down a value. For custom iterators (which implement ___ and have their own logic) or yield generators, it is to execute the logic in next once, or execute some code.

Generator is a subclass of iterator. It is divided into simple generator and custom generator (yield) according to the generation method.

The simple generator is g= (x for x in range(10)). Note that the parentheses are not []. At this time, G is a generator object generator

The simple generator object is no different from the iterator object. It is executed by next or for. It is also an inert sequence, affected by the source data, and so on, but some custom calculations are added.

Custom generators are common and can be used in a variety of ways.

Custom generator: any function that uses the keyword "yield" will become a generator, and the called function will become a generator object.

See testGenerator for details. For example, if the yield keyword is used in def custom(), aaa = custom(), and aaa is a generator object.

But at this time, aaa is equivalent to no execution, which is different from ordinary functions.

There are several points to note in the Custom Builder:

The value after the yield keyword can be any value or variable or, Every time the execution stops at yield, the value after it will be returned to next() or send()

Wait for any call to execute the function as the return value. The next execution will continue from the position to the left of yield.

Generally speaking, the custom function can be divided into several or countless sections that can be executed. The first part is from the beginning to the first yield, and the second part is from the yield to the second yield,

And so on. Next () is to execute each part in turn. send() is to assign a value to yield first, and then next().

Since yield has not been implemented in the first part, if AAA is used at the beginning Send ('test ') will report an error. If you want to use it, you can only use AAA send(None).

End the question. If the loop in the generator is not written dead but has an end condition, the execution function will report an error StopIteration when the condition is met to end the loop,

Therefore, if there is an end condition, add exception capture outside next()/send().

In other words, if the iterator has reached the last element, then next() will report an error. Put it into for in for iteration and no error will be reported.

If you want to get the return value of the whole function, you must catch the StopIteration error, and the return value is contained in the value of StopIteration.

Length of iterator: iterators cannot take the length normally,

def get_length(generator): if hasattr(generator,"__len__"): return len(generator) else: return sum(1 for _ in generator)

~. for custom function logic or infinite loop, only one count variable can be customized.

Python's for loop is essentially realized by continuously calling the next() function, for example:

for x in [1, 2, 3, 4, 5]: pass

Fully equivalent to:

it = iter([1, 2, 3, 4, 5]) # loop: while True: try: x = next(it) except StopIteration: # exit the loop when StopIteration is encountered # break

The following code can be looked at for easy understanding.

#!/usr/bin/python3
 
import sys         # Introducing sys module
 
list=[1,2,3,4]
it = iter(list)    # Create iterator object
 
while True:
    try:
        print (next(it))
    except StopIteration:
        sys.exit()

class MyNumbers:
    # def __init__(self):
    #     self.a = 88

    def __iter__(self):
        self.a = 1
        #Anything can be returned, but it must be an iterator class object or its children
        return iter([1,2,3])
        # return self
        # return (i for i in range(100))

    def __next__(self):
        x = self.a
        self.a += 1
        return x

def testIterator():
    myclass = MyNumbers()
    print(isinstance(myclass, Iterable))  # True
    print(issubclass(MyNumbers, Iterable))  # True
    print(isinstance(myclass, Iterator))  # True
    print(issubclass(MyNumbers, Iterator))  # True
    # Create an instance here mycalss The type of is <class '__main__.MyNumbers'> ，But it's an iterator object, equivalent to iterator Child objects of
    #  In fact, you don't need to do it again after the instance iter()After that, the instance itself is an iterator, which can be used directly next(myclass)Yes.

    myiter = iter(myclass)  #Generate a standard iterator, and the returned value is__inter__Returned.

    print(type(myclass))
    print(type(myiter))
    # for x in myiter:
    #     print(x)
    print(myiter.__next__())
    print(myiter.__next__())
    print(next(myiter))

def testIterator3():
# Test iterator iterator,generator  generator,Iteratable object iterable Relationship between
# Test iterator iterator,generator  generator,Iteratable object iterable Relationship between#  iterable > iterator > generator   The generator must be an iterator and an iteratable object, but an iteratable object is not necessarily an iterator xx，For example, a list is an iteratable object, and its# Class implements__ inter__ Method, but it is not an iterator.
    #An array
    arr = [1,2,3,4]
    #Generate an iterator iterator
    ii = iter(arr)
    #Generate a generator generator
    gg = (x for x in [5,6,7,8])

    print(type(ii)) #<class 'list_iterator'>
    print(type(gg)) #<class 'generator'>

    print(isinstance(arr,Iterable)) #True，Lists are iteratable objects
    print(isinstance(ii,Iterable)) #True，Iterators are iteratable objects
    print(isinstance(gg,Iterable)) #True，Generators are iteratable objects

    print(isinstance(arr,Iterator)) #Flase，The list is not an iterator object, that is, the list is not an iterator
    print(isinstance(ii,Iterator)) #True，Iterators are iterator objects
    print(isinstance(gg,Iterator)) #True，Generators are iterator objects
    
    print(issubclass(Iterator,Iterable))    #True，An iterator class is a subclass of an iteratable class
    print(issubclass(Generator,Iterator))    #True，The generator class is a subclass of the iterator class

def testIterator2():
#Test nesting iter Iteration, will the value be affected by the initial value
    arr = [6,7,8,9]
    arrIter = iter(arr)
    arrIter2 = iter(arrIter)  #Iterators can generate iterators again
    arrIter3 = iter(arrIter2)

    arr[2] = 888
    print(arr)    #[6, 7, 888, 9]
    print(arrIter)      #<list_iterator object at 0x00000265B935FEB0>
    print(arrIter2)     #<list_iterator object at 0x00000265B935FEB0>
    print(arrIter3)     #<list_iterator object at 0x00000265B935FEB0>
    print(next(arrIter),next(arrIter),next(arrIter))   # 6 7 888  The value of the iterator is affected
    print(next(arrIter2))    #  9
    #The iterator iterates repeatedly, and the returned value is itself, that is, the same object referenced by both. A reference similar to a variable.

#!/usr/bin/python3
 
import sys
 
def fibonacci(n): # generator function  - Fibonacci
    a, b, counter = 0, 1, 0
    while True:
        if (counter > n): 
            return
        yield a
        a, b = b, a + b
        counter += 1
f = fibonacci(10) # f Is an iterator that is returned by the generator
 
while True:
    try:
        print (next(f), end=" ")
    except StopIteration:
        sys.exit()

def testGenerator():
#Learn the concurrent use of generator through the example of producing steamed stuffed bun and eating steamed stuffed bun
    def consumer(name):  # consumer
        print("%s Ready to eat steamed stuffed bun!" % name)
        while True:
            baozi = yield 'I am yield Later,Each run to yield I will be returned'
            print("Steamed stuffed bun[%s]Here we are, by the way[%s]Eat!" % (baozi, name))

    def producer():  # producer
        c = consumer('A')
        c2 = consumer('B')

        # here c and c2 All generators generator At this time, they are equivalent to no implementation.
        print(type(c))
        print(type(c2))

        # print(c.__next__())
        print(next(c2))
        time.sleep(3)
        print(next(c2))
        time.sleep(3)
        print(next(c2))

        # print(c.send(None))
        # print(c.send(123))
        # time.sleep(3)
        # print(c.send(456))
        # print(c.send(789))

        print("Start making steamed stuffed buns!")
        for i in ["Leek stuffing","Fennel stuffing","Egg filling","Pork stuffing"]:
            time.sleep(1.5)
            print("Made two steamed stuffed buns")
            c.send(i)       #------------------------
            c2.send(i)      # .send(i):Give it first yield Send value again next

    producer()

Another point is that the small test on occupying memory space draws lessons from the article of a blogger. https://www.cnblogs.com/yinsedeyinse/p/11848287.html

import os
import psutil

def show_memory_info(hint):
    pid = os.getpid()
    p = psutil.Process(pid)

    info = p.memory_full_info()
    memory = info.uss / 1024. /1024
    print('{} memory used:{}MB'.format(hint,memory))

def test_iterator():
    show_memory_info('initing iterator')
    list_1 = [i for i in range(100000000)]
    show_memory_info('after iterator initiated')
    print(sum(list_1))

def test_generator():
    show_memory_info('intiting generator')
    list_2 = (i for i in range(100000000))
    show_memory_info('after generator initiated')
    print(sum(list_2))
    show_memory_info('after sum called')

test_iterator()
test_generator()

initing iterator memory used:7.21875MB
after iterator initiated memory used:1848.28515625MB
4999999950000000
intiting generator memory used:1.7109375MB
after generator initiated memory used:1.7421875MB
4999999950000000
after sum called memory used:2.109375MB


output
initing iterator memory used:7.21875MB
after iterator initiated memory used:1848.28515625MB
4999999950000000
intiting generator memory used:1.7109375MB
after generator initiated memory used:1.7421875MB
4999999950000000
after sum called memory used:2.109375MB

'''