Python Network Programming (Xiaobai can understand it at a glance)

1, Basic concepts of network programming

1. mac address and ip address

For example, on the campus, the male god named Lucy can be separated from the campus by student number. If you want to find Lucy male god, you have to pass the ID card number. Here the student ID number is temporary, and the change is out of campus. The ID number is unique and invariable. It can be found everywhere.

In the computer, there is a similar concept. What remains unchanged is the mac address, which can uniquely identify your machine; The change is the ip address, which makes it easier to find your machine. (it can be compared with express delivery in life)

1.1 ip address

ipv4 protocol: it is stipulated that the ip address is composed of four decimal points, and each bit is eight binary (the range is 0 ~ 255), so the ip address range is 0.0.0 ~ 255.255.255.255

Public network address: the address where we need to apply for purchase ourselves

Intranet address: reserved fields, 192.168.0.0 ~ 192.168.255.255 and 172.16.0.0 ~ 172.31.255.255 and 10.0.0.0 ~ 10.255.255. All addresses in these ranges are intranet addresses.

The router is used to access the public network address and the switch is used to access the intranet address.

There is a special ip address, 127.0.0.1 local loopback address. It is usually used for testing.

View your own ip address: enter ipconfig in cmd to view. linux and mac use ifconfig.

For example, in the network segment 172.168.14.0, there can be at most 254 machines, and one is the gateway IP. Because 0 in the IP address is used to specify the network address number, 192.168.14.0 represents the network segment, not the IP.

Subnet mask: it is also an ip address used to judge whether two machines are in the same LAN. Use one ip address and subnet mask for bitwise sum (first convert each ip address into binary, and then bitwise sum), use another ip address and subnet mask for bitwise sum, and then compare whether the two bitwise and subsequent ip addresses are the same. If so, it means that they are in the same LAN, otherwise, they are not in the same LAN.

For example, 192.168.12.1 and 192.168.13.1 have the same two bits in front (192.168), so only two 255 are required, that is, the subnet mask is 255.255.0.0, which indicates that the two IPS are in the same LAN.

There are two machines, 192.168.12.1 And 192.168.13.1，How to judge whether the two addresses are in the same network segment
 The subnet mask is 255.255.255.0-->11111111.11111111.11111111.00000000
192.168.12.1-->11000000.10101000.00000110.00000001
 By bit and after is 11000000.10101000.00000110.00000000-->192.168.12.0

192.168.13.1-->11000000.10101000.00000111.00000001
 By bit and after is 11000000.10101000.00000111.00000000-->192.168.13.0

If the two addresses are different, it means that they are not in the same LAN. On the contrary, they are in the same LAN.
The above example shows that it is not in the same LAN.

1.2 mac address

On the network card, it is unique.

arp protocol: address resolution protocol. The mac address of a machine is obtained through its ip address, which uses the broadcast and unicast functions of the switch.

port

The range of ports: 0 ~ 65535, which is used to confirm the of specific applications on the machine.

2. LAN

2.1 communication in LAN

Multiple machines connected to the same switch form a local area network, and all message transmission is handled by the switch. The switch only recognizes mac addresses.

I know the ip address of a machine, and then I want to send a message to it. Because the switch doesn't know the ip address, I need to get the mac address of the machine first, and then both parties know the mac address of the other party. In this way, it is very convenient to use the switch for data transmission.

Switch: broadcast, unicast, multicast

Network segment: 192.168.12 XX

2.2 communication between LANs

You need a router. The router knows the ip address. The router provides gateway ip. All machines in the same LAN share a gateway. We can't access the ip address of other intranet except this LAN.

Gateway: the "gateway" from one network to another.

2, The simplest network communication - socket implementation code

server.py

import socket
# socket
sk = socket.socket()  # Create a server-side object
sk.bind(('127.0.0.1', 9000))  # Bind an address to the server
sk.listen()  # Start listening for connections from clients

conn, adder = sk.accept()  # Establish a connection. conn is the connection

# The server sends a message to the client
conn.send(b'hello')

# The server receives messages from the client
msg = conn.recv(1024)
print(msg)
conn.close()  # Close connection

sk.close()

client.py

import socket

sk = socket.socket()

# It must be consistent with that on the server side
sk.connect(('127.0.0.1', 9000))
msg = sk.recv(1024)
print(msg)

# The client sends a message to the server
sk.send(b'bye')

sk.close()

3, tcp, udp, osi

Seven layer protocol: application layer, presentation layer, session layer, transport layer, network layer, data link layer and physical layer

3.1 layer 5 protocol

3.1.1 theory

Application layer: python code, b'hello '

Transport layer: the port to prepare how to transmit and use, b 'sender's port + hello + receiver's port'

Network layer: ip, b 'sender's ip + sender's port + hello + receiver's port + receiver's port'

Data link layer: mac, b 'sender's mac address + sender's ip + sender's port + hello + receiver's port + receiver's port + receiver's mac address'

Physical layer: converted into electrical signal and transmitted through network cable

3.1.2 protocols and physical devices involved in each layer

The fifth layer is the application layer: python code.

The fourth transport layer: port, udp, tcp. Four layer router, four layer switch.

The third network layer: ipv4 and ipv6. Router, layer 3 switch. (home router with switch function)

The second data link layer: mac, arp protocol. Network card, layer 2 switch.

The first physical layer: network cable.

3.2 tcp protocol and udp protocol

3.2.1 tcp protocol

tcp (voice chat) - offline cache of HD movies, qq remote control, email

You need to establish a connection before you can communicate.

Features: occupied connection, reliable (message will not be lost), high real-time and slow

Establish connection: refers to three handshakes

Disconnect: refers to four waves

3.2.2 udp protocol

udp (text messaging) - play videos online, send messages via qq, and send messages via wechat

You can communicate without establishing a connection.

Features: no occupation of connection, unreliable (messages may be lost due to network instability), fast

4, tcp protocol code

The operating system will uniformly allocate all resources of the computer

socket(): server of tcp protocol

socket(type=socket.SOCK_DGRAM): server of udp protocol [used in udp protocol]

Bind: bind an id and port

Listen: listen, which represents the opening of socket service

accept: wait until there is a client to access and establish a connection with the client

Send: send the message directly through the connection without writing the address

sendto: you need to write the address of the opposite party [used in udp protocol]

recv: receive messages only

recvfrom: receive message and address [used in udp protocol]

connect: the method of client / tcp protocol to establish a connection with server

Close: close service (sk.close) / connection (conn.close)

4.1 be able to receive requests from multiple clients

server.py

import socket

sk = socket.socket()
sk.bind(('127.0.0.1', 9000))  # Request operating system resources
sk.listen()

while True:
    # print(f'sk:{sk}')
    # conn stores the connection information between a client and a server
    conn, adder = sk.accept()  # Can shake hands with multiple clients
    # print(f'conn:{conn}')
    conn.send(b'hello')
    msg = conn.recv(1024)
    print(msg)
    conn.close()  # Wave and disconnect

sk.close()  # Return the resources of the requested operating system

client.py

import socket

sk = socket.socket()

# It must be consistent with that on the server side
sk.connect(('127.0.0.1', 9000))
msg = sk.recv(1024)
print(msg)

# The client sends a message to the server
sk.send(b'bye')

sk.close()

4.2 in the connection, say a few more words with the client. And solve the problem that Chinese cannot be transmitted

server.py

import socket

sk = socket.socket()
sk.bind(('127.0.0.1', 9000))  # Request operating system resources
sk.listen()

while True:  # To shake hands with multiple clients
    # conn stores the connection information between a client and a server
    conn, adder = sk.accept()  # Can shake hands with multiple clients

    while True:
        send_msg = input('please input you want say:')

        conn.send(send_msg.encode('utf-8'))
        if send_msg.upper() == 'Q':
            break
        msg = conn.recv(1024).decode('utf-8')
        if msg.upper() == 'Q':
            break
        print(msg)
    conn.close()  # Wave and disconnect

sk.close()

client.py

import socket

sk = socket.socket()

sk.connect(('127.0.0.1', 9000))

while True:
    msg = sk.recv(1024).decode('utf-8')
    if msg.upper() == 'Q':
        break
    print(msg)

    send_msg = input('please input you want say:')
    sk.send(send_msg.encode('utf-8'))
    if send_msg.upper() == 'Q':
        break

sk.close()

4.3 where will the program block and when will it end

input() Wait until user input enter key

accept() Blocking. After a client establishes a connection with me

recv() Block until you receive a message from the other party

recvfrom() Block until you receive a message from the other party

connect() Block until server End to end client Services, start and current client When establishing a connection

5, udp protocol code

server.py

import socket

sk = socket.socket(type=socket.SOCK_DGRAM)  # Represents a udp protocol
sk.bind(('127.0.0.1', 9000))

# The server cannot send a message first because the server does not know the ip address of the client
while True:
    msg, addr = sk.recvfrom(1024)
    print(msg.decode('utf-8'))
    send_msg = input('please input you want say:')
    sk.sendto(send_msg.encode('utf-8'), addr)
# The server side does not need to judge to exit
# Because I don't communicate with this client, I have to communicate with other clients

client.py

import socket

sk = socket.socket(type=socket.SOCK_DGRAM)
server = ('127.0.0.1', 9000)

while True:
    send_msg = input('please input you want say:')
    if send_msg.upper() == 'Q':
        break
    sk.sendto(send_msg.encode('utf-8'), server)
    msg = sk.recv(1024).decode('utf-8')
    if msg.upper() == 'Q':
        break
    print(msg)

6, Sticking package problem

6.1 packet sticking in TCP protocol

Two or more messages sent separately are connected together, which is the phenomenon of sticking packets

server.py

import socket

sk = socket.socket()
sk.bind(('127.0.0.1', 9001))
sk.listen()

conn, addr = sk.accept()
conn.send(b'hello')
conn.send(b'nice')
conn.close()
sk.close()

client.py

import time
import socket

sk = socket.socket()
sk.connect(('127.0.0.1', 9001))

time.sleep(0.1)
msg1 = sk.recv(1024)
print(msg1)
msg2 = sk.recv(1024)
print(msg2)

sk.close()

Start the server first and then the client. The client prints the following results:

b'hellonice'
b''

The phenomenon of sticking package occurs. The results after removing the sleep time are as follows:

b'hellonice'
b'nice'

[note] even if the sleep time is not set, it is possible to stick the bag

6.2 two reasons for sticking

Packet sticking only occurs in tcp protocol, because in tcp protocol, there is no boundary between multiple messages (streaming transmission), and there are a lot of optimization algorithms.

• Sender: the two messages are very short, and the interval between sending is also very short. Due to the optimization mechanism, they are combined and sent together
• Receiving end: sticky packets caused by multiple messages piled together in the receiver's cache due to failure to receive them in time

6.3 essence of sticking

The transmission of tcp protocol is streaming, and there is no boundary between data

6.4 essence of solving the problem of sticking package

Set boundary

6.5 solving sticking

Custom protocol

Server: first send the data length of 4 bytes, and then send the data according to the length

Client: first receive 4 bytes, know the length of the data, and then receive the data according to the length

6.6 struct module

The module can convert a type, such as a number, into a fixed length of bytes.

The range of numbers is - 2 * * 31-1 ~ 2 * * 31, with a total of 2 * * 32 numbers. pack can convert any number in this range into 4 bytes in length.

import struct

num1 = 123456789
num2 = 12345
num3 = 123

ret1 = struct.pack('i', num1)
print(ret1, len(ret1))  # b'\x15\xcd[\x07' 4
ret2 = struct.pack('i', num2)
print(ret2, len(ret2))  # b'90\x00\x00' 4
ret3 = struct.pack('i', num3)
print(ret3, len(ret3))  # b'{\x00\x00\x00' 4

print(struct.unpack('i', ret1))  # (123456789,)
print(struct.unpack('i', ret2))  # (12345,)
print(struct.unpack('i', ret3))  # (123,)

6.7 code for solving sticking package

server.py

import socket
import struct

sk = socket.socket()
sk.bind(('127.0.0.1', 9001))
sk.listen()

conn, addr = sk.accept()
msg1 = input('>>>')
msg2 = input('>>>')
blen = struct.pack('i', len(msg1.encode()))
conn.send(blen)
conn.send(msg1.encode())
conn.send(msg2.encode())
conn.close()
sk.close()

client.py

import time
import socket
import struct

sk = socket.socket()
sk.connect(('127.0.0.1', 9001))

time.sleep(0.1)
length = sk.recv(4)
length = struct.unpack('i', length)[0]
msg1 = sk.recv(length)
msg2 = sk.recv(1024)

print(msg1.decode('utf-8'))
print(msg2.decode('utf-8'))

sk.close()

6.8 summary

1. recv(1024) does not mean that 1024 bytes must be received, but can only receive 1024 bytes at most
2. The problem of sticking packets must be avoided for two consecutive pieces of data
3. The custom protocol of tcp protocol solves the problem of sticking packets: first send the length of data, and then send data. [compose the sent data into json format: send the length of json first, then send json (the length of data to be sent next is stored in json), and then send data]

7, Multi person chat based on udp protocol

7.1 multi person chat based on udp protocol

Automatically identify users. ip and port cannot be used

server.py

import socket

friend_lst = {'jack': '31', 'tom': '34'}
sk = socket.socket(type=socket.SOCK_DGRAM)
sk.bind(('127.0.0.1', 9000))
while 1:
    msg, addr = sk.recvfrom(1024)
    msg = msg.decode('utf-8')
    name, message = msg.split('|')
    print(f'\033[1;{friend_lst.get(name,"30")}m {name}: {message} \033[0m')
    content = input('>>>')
    sk.sendto(content.encode('utf-8'), addr)

client.py

import socket

name = 'jack'
sk = socket.socket(type=socket.SOCK_DGRAM)

while 1:
    content = input('>>>')
    if content.upper() == 'Q':
        break
    content = f'{name}|{content}'
    sk.sendto(content.encode('utf-8'), ('127.0.0.1', 9000))
    msg = sk.recv(1024).decode('utf-8')
    if msg.upper() == 'Q':
        break
    print(msg)

8, File transfer based on tcp protocol

server.py

import socket
import json
# receive
sk = socket.socket()
sk.bind(('127.0.0.1', 9002))
sk.listen()
conn, addr = sk.accept()
msg = conn.recv(1024).decode('utf-8')
msg = json.loads(msg)

with open(msg['filename'], 'wb') as f:
    content = conn.recv(msg['filesize'])
    print('-->', len(content))
    f.write(content)

print(msg)
conn.close()
sk.close()

client.py

import os
import json
import socket

# send out
sk = socket.socket()
sk.connect(('127.0.0.1', 9002))

# File name, file size
abs_path = r'E:\A.PythonProject\action\day30\task\a.txt'
filename = os.path.basename(abs_path)
filesize = os.path.getsize(abs_path)
dic = {'filename': filename, 'filesize': filesize}
str_dic = json.dumps(dic)
sk.send(str_dic.encode('utf-8'))

with open(abs_path, 'rb') as f:
    content = f.read()
    sk.send(content)
sk.close()

8.1 optimization

The problem with the above code: the dictionary sent may send sticky packets with the contents of the file; When sending a large file, the received file size is smaller than the original one

server.py

import socket
import json
# receive
import struct

sk = socket.socket()
sk.bind(('127.0.0.1', 9002))
sk.listen()
conn, addr = sk.accept()
msg_len = conn.recv(4)
dic_len = struct.unpack('i', msg_len)[0]
msg = conn.recv(dic_len).decode('utf-8')
msg = json.loads(msg)

with open(msg['filename'], 'wb') as f:
    while msg['filesize'] > 0:
        content = conn.recv(1024)
        # msg['filesize'] -= 1024. The size passed is not necessarily 1024
        msg['filesize'] -= len(content)
        f.write(content)

conn.close()
sk.close()

client.py

import os
import json
import struct
import socket

# send out
sk = socket.socket()
sk.connect(('127.0.0.1', 9002))

# File name, file size
# Must be an absolute path
abs_path = r'E:\A.PythonProject\action\day30\task\a.txt'
filename = os.path.basename(abs_path)
filesize = os.path.getsize(abs_path)
dic = {'filename': filename, 'filesize': filesize}
str_dic = json.dumps(dic)
b_dic = str_dic.encode('utf-8')
mlen = struct.pack('i', len(b_dic))
sk.send(mlen)  # 4 bytes represent the length after the dictionary is converted to bytes
sk.send(b_dic)  # Specific dictionary data

with open(abs_path, 'rb') as f:
    while filesize > 0:
        content = f.read(1024)
        filesize -= 1024
        sk.send(content)
sk.close()

9, Verify the legitimacy of the client

server.py

import hashlib
import os
import socket
secret_key = b'nice@123!'
sk = socket.socket()
sk.bind(('127.0.0.1', 9001))
sk.listen()

conn, addr = sk.accept()
# Create a random string
rand = os.urandom(32)
# Send random string
conn.send(rand)
# According to the sent string + secret_key for summary
sha = hashlib.sha1(secret_key)
sha.update(rand)
res = sha.hexdigest()
# Waiting to receive summary results from the client
res_client = conn.recv(1024).decode('utf-8')
# Make a comparison. If it is consistent, it will show that it is a legal client and can continue to operate
if res_client == res:
    print('Is a legal client')
    conn.send(b'hello')
else:
    # If not, close the connection
    conn.close()

client.py

import hashlib
import socket
secret_key = b'nice@123!'
sk = socket.socket()
sk.connect(('127.0.0.1', 9001))
# Receive the random string sent by the server
rand = sk.recv(32)
# According to the sent string + secret_key for summary
sha = hashlib.sha1(secret_key)
sha.update(rand)
res = sha.hexdigest()
# The summary result is sent back to the server
sk.send(res.encode('utf-8'))
# Continue to communicate with the server
msg = sk.recv(1024)
print(msg)

hmac module

To replace hashlib module

import os
import hmac

h = hmac.new(b'nice@123!', os.urandom(32))
ret = h.digest()
print(ret)

Therefore, the code to verify the legitimacy of the client can be simplified.

10, Socket server module

The socket server module is based on the socket module. The socket module is the bottom module, with low encapsulation and unstable efficiency; Socket server module has high encapsulation and fixed efficiency.

10.1 function

The server side of tcp protocol handles concurrent client requests

server.py

import socketserver
import time


class Myserver(socketserver.BaseRequestHandler):
    # Override handle
    def handle(self):
        # print(self.request)
        # <socket.socket fd=472, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 9001), raddr=('127.0.0.1', 9040)>
        # Equivalent to conn, it has both its own ip and the ip of the client
        conn = self.request
        while 1:
            try:
                content = conn.recv(1024).decode('utf-8')
                conn.send(content.upper().encode('utf-8'))
                time.sleep(0.5)
            except ConnectionResetError:
                break


server = socketserver.ThreadingTCPServer(('127.0.0.1', 9001), Myserver)
server.serve_forever()

client.py

import socket

sk = socket.socket()
sk.connect(('127.0.0.1', 9001))
while 1:
    sk.send(b'hello')
    content = sk.recv(1024).decode('utf-8')
    print(content)

10.2 notes

• Only the code of the server side is changed, but the code of the client side is not changed
• When the client connects, it directly interacts with the code in the handle