Review lessons
Step by step talk back to the package
To get straight to the point, the third article has mentioned how to receive packets back. At present, network programming is all about Tcp, and the protocol request mode itself will determine the characteristics. Let's talk about the order of contracting and receiving packets first.
Tcp is first of all a duplex, which means that the client C sends messages to the S end, and the S end can also send messages to the client. In the transmission protocol data type, the transmission in the network card layer is binary file stream. Why is a stream rather than a package? You can review the previous, or you can choose to memorize by rote.
If the C-side sends a message to the S-side, the S-side must reply. This is not necessarily true. Because the protocol data type is also transmitted in the protocol request / transmission mode, which is directly related to the service, and can be roughly divided into response required and no response required. The most common thing that does not require a response is a heartbeat packet. Why is it called a bag instead of a heartbeat? It can only be said to be a general name.
After the client connects to the service, it transmits a fixed section of information to the server every about a period of time, which is the same as the human ECG to confirm that the current client is active. If it is not active, the server will disconnect the client object (Tcp waves). Here, it generally refers to how many times it is not received before it is judged to be inactive.
The benefits of disconnection can be remembered in this way. Each socket link must have a cost, and the same is true for maintaining the link state. Some things done by each socket will cause memory changes (software cache and several levels of hardware cache are not considered here). If some data is stored in the database, the memory data will be exchanged to the database or database cache, The interaction process also generates memory fragmentation and database IO overhead. If it is only active, a series of non user overhead will be generated according to what the socket does. Therefore, it is necessary to eliminate inactive requests.
For example, 30 minutes without experience / many minutes without moving (moving will also generate mobile data packets, which will be synchronized to the active users around him one by one), and then it will enter the hang up state. In fact, this is similar to marking fb as inactive (leave = no alive). If it is inactive, it only needs to be synchronized regularly.
If you are completely closed, you will automatically hang up to gain experience regardless of the business.
The above mainly starts slowly. The client sends 100 bytes three times and waits for hundreds of ms, but the server only returns a group of packets. This is the problem of Tcp's unique sticky packets.
However, it is found that this span is a little large. The examples are relatively basic. We will slowly extend it later through an example of Tcp heartbeat (divided into four or five chapters). Anyway, we will make it clear in the end.
Heartbeat learning and design
Write an example to do some intuitive reasoning and design. The example will also combine some previous knowledge points.
Heartbeat information: the json string # in actual combat is in the form of bytes. The buttons will be relatively small, just to make the server count and the client active.
Explain in detail the management dictionary of a client connection handle recorded on the server side. The linked client object will be a "fd" on the server side. Whether it is active or not will be a field "status": "live", "leave" and "close" represent active, inactive, suspended and closed respectively.
The server side carries out development and design, and supports a two-way order ("live" < -- > "leave" < -- > "close")
The status: "live" status condition is the default for the client, not in the socket Error that layer is blocked, so the link is connected. The link is divided into correct address, correct writing and firewall permission (this is not used in the test here)
Server development required:
1.socket server program
2. Regular receiving packet
3. Judge that the data after 4 bytes is in deserialization JSON loads. After the server has one more bind than the client, turn on the listening function. The server supports multiple clients.
4. Matching function. After receiving the client information, the server will match with a cursor (this function is completed in Chapter 5).
For example, if the cursor is set to 3 and the threshold is 5, the number of seconds a message is received is - 1, and the number of seconds a message is not received is + 1 When the threshold reaches 5 times, count once and reach 3 times, modify the current client link management dictionary to "leave". If the threshold is reached less than 3 times, it is still alive.
5. A user instance object list will be used to manage the clients that add simulation links. Add one every 10 seconds, the client is simulated false, and the port number is + 1.
Client development required:
1.socket client program
2. Jason contracting + data structure
3. Every 10 seconds.
About status:"leave" status: if the record is kept for 30 minutes (the code will be written for 30 seconds), it will go to the next status "close". If the heartbeat is stable for many times and does not trigger the threshold value of 5, it will fall back to "alive".
As for the arrival status:"close", the server will kick off the client. In the user's list management, the server will store the ip: port of the client.
Heartbeat client front
Write this example first. There are only "live" state conditions and no switching conditions. According to past examples, we need to define a data structure of the network layer first. This time, we don't need protobuff.
Tcp for protocol transmission uses struct with high universality, and the transmission data structure is Json
The Tcp data structure is 4 bytes in the header, and 4 bytes are the length of the inner package. The whole package length is the value of 4 bytes (fixed) + 4 bytes in the header (dynamic package length).
Json C2S {"ip": pass the local Ip, "status": the above-mentioned status uses the default, "pid": it is used for the client to manage its own} pid, which is mainly used to kill its own process.
Here we need to add a knowledge. What is not finished in Chapter 3 is how to write the 4-byte packet header length + the client transfer object
# ip_add address to the server
ip_addr = socket.gethostbyname(socket.gethostname())
# The client manages its own pid locally
pid = os.getpid()
# The list used for state switching is ordered. It is used as the state machine index through idx. The default is alive
status = ["alive", "leave", "close"]
idx = 0
data = {'ip': ip_addr, 'status': status[idx], 'pid': pid}
# The length of data. str here is only used for the following +. Int - > str does not affect the following results
pack_len = str(len(data))
print(pack_len) # The output is 3
# i is 4 bytes + pack_len struct.calcsize looks at the fmt area
client_len = struct.calcsize("!i" + pack_len + 's')
print(client_len) # The output is 4 + 3Pay attention to the comments. The dynamic calculation length comes out, but pay attention to struct The dynamic package compression is oriented to several objects in the package header. If only one object is passed into json, the output should be like this
This code is about the example of pressure package
buffer = json.dumps(data)
print(len(buffer))
packet_head = struct.pack("!i",len(buffer))
buffer_client = packet_head+buffer.encode("utf-8")
print(buffer_client) # b'\x00\x00\x008{"ip": "192.168.1.105", "status": "alive", "pid": 26464}'Heartbeat client
It only includes communicating with the server, sending TCP stuct and json, and sending json data as heartbeat packet. Note that the actual heartbeat will indeed be fixed data, but it will not be passed in.
The server just parses the string to determine which client sent the heartbeat. The state switching function will be written in the next article.
import datetime
import socket, sys, os
import time, json,struct
class HeartBeatClient:
interval = 10 #Heartbeat transmission interval
def __init__(self, addr: tuple):
try:
self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.client.connect(addr)
except socket.error as e:
print("Error creating socket: %s" % e)
sys.exit()
def encode_json(self, conn,data:dict):
"""Contract awarding"""
buffer = json.dumps(data)
print(f"The length of packets sent by the current client{len(buffer)}")
packet_head = struct.pack("!i", len(buffer))
conn.send(packet_head + buffer.encode("utf-8"))
def loop_client(self, conn: socket.socket):
"""
Circular client
:param conn:
:return:
"""
status = ["alive", "leave", "close"] #The status list is ordered
idx = 0 #Status index
while True:
ip_addr = socket.gethostbyname(socket.gethostname())
pid = os.getpid()
data = {"ip": ip_addr, "status": status[idx], "pid": pid}
buffer = json.dumps(data)
try:
self.encode_json(conn,buffer)
except socket.error:
print("client send failed")
break
else:
now =datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print({"time":now,"fd":os.getpid(),"status":status[idx]})
time.sleep(self.interval)
self.client_close(conn)
def client_close(self,conn):
if conn:
conn.close()
if __name__ == '__main__':
client = HeartBeatClient(("127.0.0.1", 14000))
client.loop_client(client.client)Strengthen the example of strcut transmission. It is recommended to knock the code by hand.
Heartbeat server
It mainly corresponds to the heartbeat server function written by the client. Note that the mode is to start the server first and then start the client file.
To simulate fd_port increases by 1, which means that many clients are connected. If you are not familiar with Threading here, you need to learn by yourself.
import socket
import json
from threading import Thread
import ast
class HeartBeatServer:
def __init__(self, addr: tuple, max: int):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind(addr)
# max links supported
self.sock.listen(max)
self.user = []
def encode_buffer(self, conn: socket.socket, data: str or bytes or dict):
"""
send data
:param conn: The handle that the client links to the server
:param data: data
:return:
"""
if isinstance(data, str):
data = data.encode("utf-8")
elif isinstance(data, dict):
data = str(data).encode("utf-8")
conn.send(str(data).encode("utf-8"))
def match_data(self,recv)->bool:
"""
Match deserialized data
:param recv:
:return:
"""
if isinstance(recv,dict):
return recv.get("ip") and recv.get("status")
def loop_thread(self, conn: socket.socket, fd_port: int):
"""
The server only judges whether the packet header of the client is legal.
:return:
"""
head_len = 4
self.user.append("192.168.1.105:88888")
idx = 1
while True:
try:
# Keep receiving packets
data = conn.recv(1024 * 4)
if len(data) > head_len:
recv = json.loads(data[4:])
recv = ast.literal_eval(recv)
if not self.match_data(recv):
# The pop() list is thrown out and the lost data is returned. The default is the last one
print(f"client fd addr:{self.user.pop()} Connected over!")
break
# To simulate fd_port increases by 1, which means that many clients are connected
self.user.append(f"{recv.get('ip')}:{fd_port + idx}")
print(f"Current server management user-->{self.user}")
else:
print(f"client fd addr:{self.user[-1]}Illegal data")
except socket.error:
print(f"client fd addr:{self.user.pop()} Connected over!")
break
else:
idx += 1
conn.close()
def start(self):
"""
Service startup
:return:
"""
while True:
conn, addr = self.sock.accept()
t = Thread(target=self.loop_thread, args=(conn, addr[1]))
t.start()
if __name__ == '__main__':
server = HeartBeatServer(("127.0.0.1", 14000), 100)
server.start()Here is an important concept
Why can't we use self here Sock is because of self A sock is a handle to the server itself
Conn is the handle that the server receives from the client object, so this conn is managed. When the client is disconnected, the server waves through Conn
If it's self The sock service is shut down.
ending
The fifth chapter is also written in a part of the draft
This article was first published in the tester home community by Chen Ziang, a senior game testing and development engineer. There are four articles on network programming in Python. Today, I share the fourth. Original link: https://testerhome.com/topics/29411
The above is today's sharing. Have you learned it~
Want to learn more about dry goods and cutting-edge technology?
Want to get to know the test industry and elite?
Welcome to the 2022 MTSC Conference (the 10th China Internet testing and Development Conference)
The industry's evaluation of MTSC conference: landing, pragmatic, in-depth and re sharing
Testing Summit China is a technical conference of software testing and development industry initiated by TesterHome. There is also an alias: Make Testing Super Cool!
MTSC conference, with the main purpose of software quality assurance system and testing and R & D technology exchange, began in 2015 and has been successfully held for nine sessions. A total of 1000 + enterprises and 10000 + test engineers, test managers and CTO s attended the meeting, which attracted extensive attention from the whole industry and was the top conference of China's Internet quality assurance industry.
In order to ensure the quality of topics, topics will be collected half a year in advance every year, and then after several months of review, the topics to be shared at the conference will be finally determined. The topic of MTSC 2022 is still being solicited. We sincerely invite senior test technology experts, quality management managers and test rookies to introduce / recommend the topic!
Issue submission method
Direct click https://www.wjx.top/vj/wZwCju... Enter the delivery entrance, fill in and submit according to the format. You are welcome to introduce yourself / recommend topics.
Issue deadline
In order to facilitate the review team to have more time to review and optimize communication with lecturers, and present better topics for the majority of participants, the deadline for submission of topics (no PPT, introduction of topic information such as outline) is advanced to March 30, 2022
Issue solicitation and selection process
Overall process: case submission > preliminary review and approval > ppt submission > confirmation of topics > conference speech
Overall time node:
Case submission > > March 30
Preliminary review and approval > > April 15
ppt submission > > May 15
Issue confirmation > > May 30
Conference speech > > July 22-23
Sharing instructor benefits
- Exercise speech ability and enhance personal brand
- Get the opportunity to communicate face-to-face with industry experts and draw on their strengths
- Enhance the company's brand and increase the attractiveness of your team
- Get free conference tickets and information:
Each lecturer will receive a free ticket to the conference, and the subsequent ppts and videos will be given to the lecturer at the first time - The travel expenses of field lecturers shall be borne by the Organizing Committee of MTSC conference
MTSC 2022 early bird tickets have been quietly sold. Click for details.