1, Network programming

1. Computer network

• Multiple computers with independent functions in different geographical locations and their external devices are connected through communication lines to realize resource sharing and information transmission under the management and coordination of the protocol.

Network programming: different computer programs used to realize network interconnection can exchange data

2. Network model

What rules are used to communicate between computer networks. It is divided into OSI model and TCP/IP reference model

2.1OSI:

• Application layer: what the computer screen sees and generates data.
• Presentation layer: data processing, encryption and decryption, compression and decompression, coding.
• Session layer: establish a data transmission path through the transport layer, initiate and accept sessions, and maintain sessions during transmission
• Transport layer: data connection and transmission. An addressing mechanism is used to identify a specific application (port) - divided into data segments
• Network layer: route selection between different network systems based on network layer address (ip address) - divide and recombine data to form new data packets
• Data link layer: encapsulate and unpack the mac address (network card address) of the data, and address it with the hardware address and physical address of the receiving system—— Data frame
• Physical layer: physical equipment: optical fiber, coaxial cable, twisted pair, network card, repeater, hub, etc. - bit data

2.2TCP/IP:

• Application layer - application layer, presentation layer, session layer merging - http: Hypertext Transfer Protocol, data transfer protocol between web servers, ftp: file transfer protocol. smtp: Simple Mail Transfer Protocol, pop: mail protocol, telent: remote login protocol. DNS: domain name system, conversion of domain name and IP address.
• Transport layer
• network layer
• Data link layer - data link layer, physical layer merging

Both parties should abide by the same agreement before information exchange

3. Transport layer

3.1TCP

• Establish a stable link through three handshakes. The application data is divided into the most appropriate data for transmission
• Reliable connection, not prone to disorder, loss and other phenomena
• Connection and verification take time and are inefficient

Three handshakes

1. The client sends SYN message to the server and enters SYN - SEND status
2. After receiving the message, the server responds to a SYN ACK message and enters the SYN-RECV state
3. The client receives the SYN message from the server, responds to an ACK message and enters the connection state.

3.2UDP

• If all protocols are sent at one time, data loss may occur.

4. Network layer

There are gateways and routers

• IP address is actually a virtual address, which will be converted into network card address through ARP protocol
• The computer can only recognize binary data, and IP is a binary data. The length of IPV4 is 32 bits, divided into 4 segments and marked in decimal system.
• IPV6 is four times longer than IPV4 and is identified in hexadecimal and divided into eight segments,

IP address: there are network number segment and host number segment. 127.0.0.1: local address, 0.0.0.0: any address. ping IP address: test whether it is the same as this IP address

Port is the ID card of software, which is used to distinguish different software. The maximum is 2 to the 16th power minus 1

#Server configuration
import socket
server = socket.socket()
server.bind(('127.0.0.1',8989))	#Bind the IP address of the server to run, and set the port arbitrarily
server.listen(10)	#Set the maximum number of listeners
result = server.accept()	#Create a connection channel and obtain a peer socket. A tuple is returned. The first is the peer connection socket, and the second is the client's IP plus port
conn,addr = result #Unpacking
conn.recv(1024)	#Sets the maximum number of bytes received
conn.send(b'return')	#Return to client data

#Data should be equal	
conn.close()
server.close()

#Client configuration
import socket
client = socket.socket()
client.connect(('127.0.0.1',8989))	#The port address opened by the connection server
client.send(b'hello')	#Send byte data
client.close()

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-h2okl5dq-1635859188331) (C: \ users \ admin \ appdata \ roaming \ typora \ typora user images \ image-20211020212829055. PNG)]

• The transmitted data must be byte data -- string to byte bytes(info,encoding = 'utf8')

2, I/O multiplexing

1. Data flow

• Definition: it is an ordered data sequence of bytes with start and end points, including input stream and output stream.
• The input stream can only read, not write, obtain data from the keyboard or file, the output stream can only write, not read, and write things to the display, printer, etc. The first person is a machine.
• I/O -- send and receive data for these streams in the process of information exchange. fd is the file descriptor, an integer, which is the operation of integers.

2.I/O interaction

• User space - user programs and applications and code
• Memory space - kernel space, operating system and drivers
• Hard disk space

3. Basic IO model

What rules are used for IO operations, including blocking IO, non blocking IO and multiplexing IO

3.1 blocking IO model

Blocking IO is to concentrate on one thing. When blocking, you don't do other things. Only after you finish this thing can you do other things.

• Advantages: the process is blocked and suspended, does not consume cpu resources, can respond to each operation in time, and has low difficulty
• Disadvantages: it is not suitable for application development with large amount of concurrency. No other operations can be performed when the process is blocked

#Block IO and establish two sockets at the same time
import socket

server = socket.socket()
server.bind('127.0.0.1',8989)
server.listen(10)

all_conn = []

for i in range(2):
    conn,addr = server.accept()
    all_conn.append()


for conn in all_conn:
    #Get all peer connection sockets
    data = conn.recv(1024)
    if data:
        print(data)
    else:
        
        all_conn.remove(conn)
    conn.close()
server.close()

3.2 non blocking IO model

• Advantages: it is efficient to do other tasks while waiting for threads.
• Disadvantages: high consumption, polling and delay, and the task may be completed at any time in the middle

#non-blocking IO 
# -*- coding: utf-8 -*-
# @Time: 2021/10/29 19:19
# @Author: Maple traceless
# @Email: thesky_27@163.com
# @File: server01.py
# @Software: PyCharm
import socket
from time import sleep
server = socket.socket()

#Set to non blocking socket, which needs i to be set before other operations
server.setblocking(False)

server.bind('127.0.0.1',8989)
server.listen(10)

# while True:
#     try:
#         conn,addr = server.accept()
#         conn.recv(1024)
#     except BlockingIOError:
#         print("do other things")
#         sleep(1)
all_conn = []
while True:
    try:
        conn,addr = server.accept()
        conn.setblocking(False) #Set the peer socket to a non blocking word
        all_conn.append(conn)
    except BlockingIOError:
        pass
    except Exception as E:
        print(f"An unknown exception occurred{E}")

    #receive data 
    for conn in all_conn:
        try:
            data = conn.recv(1024)
            if data:
                print(data)
                conn.send(data)
            else:
                conn.close()
                all_conn.remove(conn)
        except BlockingIOError:
            pass
        except Exception as E:
            print(f"An unknown exception occurred{E}")

• Multiple events occurring at the same time are called concurrency. Only one instruction is executed at the same time, but it is fast switching execution. The macro is executed together, and the micro is not executed at the same time.
• Multiple events occurring at the same time point are called parallelism. Connecting to the client and receiving data are performed concurrently.

3.3 IO multiplexing model

Hire someone to see if you're hooked

• Advantages: less resources and less CPU consumption
• Disadvantages: two system calls are required

Principle: the select/epoll function will continuously poll the responsible socket and notify the user when there is data.

epoll is the best worker. It is a lazy event callback that users call themselves. It is the best IO multiplexer in linux, but only linux has it.

#Multiplexing
import socket,selectors #IO multiplexer module,

epoll_select = selectors.EpollSelector()    #instantiation 
selectors.DefaultSelector() #Default selector, automatically selected according to different operating systems
server = socket.socket()
server.bind('127.0.0.1',8989)
server.listen(10)

def f_accept(server):
    conn,addr = server.accept() #Generate peer-to-peer connection socket
    epoll_select.register(conn, selectors.EVENT_READ, f_recv)


def f_recv(conn):
    data = conn.recv(1024)
    if data:
        print(data)
    else:
        conn.close()

#selectors.EVENT_ When the read event occurs, the callback function is called, and the third parameter
epoll_select.register(server,selectors.EVENT_READ,f_accept)

while True:
    events = epoll_select.select()   #Query, return a list, save a tuple,
    for key,mask in events:
        func = key.data     #Function body
        conn = key.fileobj  #Registered socket
        func(conn)      #Call f_accept method.

3, Process and thread

1.CPU time slice

• The time allocated by the cpu to each program, and each thread is allocated a time period, which becomes a time slice
• Single core CPU can only realize concurrency. To truly realize parallelism, multi-core CPU is required

2. Process

• Running program. A process produces an independent space, and processes are independent and do not affect each other
• Timestamp: 0:00:00 GMT, January 1, 1970. time.time() -- get timestamp
• time.localtime(time.time()) -- get the current time.

Single process: it is the usual execution sequence

Multi process: it is as follows

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())

print('main_start',new_time())
p1 = multiprocessing.Process(target=func,args=(1,2))
p1.start()
time.sleep(5)#Simulate time-consuming operations,
print('main_end',new_time())
#It took five seconds.

• target: indicates the tasks that the process needs to perform, and lets the child process perform the tasks it needs. args: tuple type is required and parameters are passed.
• There is no effect if the number of processes is more than the number of cores. The core of cpu is to share several time-consuming tasks

3. Thread

• A program can have multiple processes, each process can have multiple threads, but there is at least one process and thread.
• Threads share the same address space.

Multithreading -- similar to a multiprocess module.

import time
import threading

def new_time():
    return time.asctime(time.localtime(time.time()))

def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())

print('main_start',new_time())
p1 = threading.Thread(target=func,args=(1,2))
p1.start()
time.sleep(5)#Simulate time-consuming operations,
print('main_end',new_time())

• Multithreading is not suitable for computing intensive scenarios, but it is suitable for IO intensive scenarios
• Automatic switching will be implemented because of blocking.

GIL global interpretation lock -- let only one thread in any process execute. Equivalent to concurrency.

4. Server concurrency

4.1 multi process concurrent server

import socket
import multiprocessing
'''
The main process left connects to the client, generates peer socket and channel connection
 Create a child process to receive clients
'''
def f_recv(conn):
    while True:
        data = conn.recv(1024)
        if data:
            print(data)
            conn.send(data)
        else:
            conn.close()
            break



def f_accept():
    sever = socket.socket()
    sever.bind(('127.0.0.1',4546))
    sever.listen(10)

    while True:
        conn,addr =sever.accept()
        p1 = multiprocessing.Process(target=f_recv,args=(conn,))
        p1.start()
if __name__=='__main__':
    f_accept()

4, Process and thread supplement

1.join method

• After a process or thread adds a join method, it will wait for the subtask to end. If it does not end, it will block until the subtask ends.
• It is usually placed at the back.
• Wait for the end of the subtask, both process and thread

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())

print('main_start',new_time())
p1 = multiprocessing.Process(target=func,args=(1,2))
p1.start()
p1.join()   #Wait until the subtask is completed, and blocking will occur
time.sleep(5)#Simulate time-consuming operations,
print('main_end',new_time())

2. Get the current process

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print(multiprocessing.current_process())    #Gets the current process object
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2))
    p1.start()
    p1.join()   #Wait until the subtask is completed, and blocking will occur
    time.sleep(5)#Simulate time-consuming operations,
    print('main_end',new_time())

3. Get the name of the child process

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2))
    print(p1.name)
    p1.name='Process one'
    print(p1.name)
    p1.start()
    time.sleep(5)#Simulate time-consuming operations,
    print('main_end',new_time())

4. Termination of tasks

• You can force the child process to end after the main process ends
• However, the thread cannot be terminated and can only wait for the end.

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2))
    p1.start()
    time.sleep(5)#Simulate time-consuming operations,
    
    p1.terminate()  #Terminate p1 child process
    print('main_end',new_time())

5. PID of process

• In Linux 1, as soon as the process is created, the system will assign a PID. During operation, the PID will not change.
• You can view the resource usage of the process through PID, and you can also control the operation of the process
• The pid of the process will not change. The pid can be found only after start

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2))
    print('before start',p1.pid)
    p1.start()
    print('after start', p1.pid)
    time.sleep(5)#Simulate time-consuming operations,
    p1.terminate()  #Terminate p1 child process
    print('main_end',new_time())

6. ident of thread

• Similar to the pid of the process in the operating system, it is allocated by the python interpreter.
• Processes are allocated by the operating system.

7. Life cycle

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2))
    print(p1.is_alive())
    p1.start()
    print(p1.is_alive())
    p1.join()
    print(p1.is_alive())
    time.sleep(5)#Simulate time-consuming operations,
    print('main_end',new_time())

8. Guard mode

• After the daemon mode is turned on, the main process ends and the child process ends automatically.

import time
import multiprocessing

def new_time():
    return time.asctime(time.localtime(time.time()))


def func(x,y):
    print(x+y)
    print('func_start',new_time())
    time.sleep(5)
    print('func_end', new_time())
if __name__ == '__main__':
    print('main_start',new_time())
    p1 = multiprocessing.Process(target=func,args=(1,2),daemon=True)    #Enable guard mode
    p1.start()
    time.sleep(5)#Simulate time-consuming operations,
    print('main_end',new_time())

5, Object oriented programming (code)

• Customize the process object to complete the concurrent operation of multiple processes on the redis database.
• redis high-speed storage database, concurrent operation.

import redis
import multiprocessing
"""
Custom:
    adopt class Keyword defines a class
    Class to inherit
"""
class RedisProcess(multiprocessing.Process):
    def __init__(self,db,key,values):
        super().__init__()#Call the method of the parent class to create the child process and instantiate the creation process
        self.connect = redis.StrictRedis(db=db) #For local connection, you only need to specify a database
        self.key = key
        self.values = values
    def set(self):
        self.connect.set(self.key,self.values)
    def get(self):
        return self.connect.get(self.key)
    def run(self):
        self.set()
        self.get()