Chapter 15 network programming

Basic knowledge

TCP/IP protocol

abbreviationFull nameProtocol type
IPInternet ProtocolNetwork protocol
TCPTransmission Control ProtocalTransmission control protocol

IP address

Meaning: in the network, the unique identification of the communication entity (which can be host, printer, router port, etc.)

Format: 32bit integer, separated by dots into four 8-bit binary numbers, converted into decimal numbers between 0 and 255


Meaning: the entrance and exit for the application to communicate with the outside world. Different programs have different ports

Format: 16 bit integer, 0 ~ 65535, divided into three categories

Rangeport typepurpose
0~1023Recognized Well Known PortBind specific services
1024~49151Register Registered PortApplication service common port
49152~5535Dynamic and/or Private PortApplication service dynamic port, not actively used


Uniform Resource Locator: consists of protocol name, host, port and resource path

Format: protocol://host:port/path

give an example:

#Basic network support for Python

Built in network related modules

socket, asyncore, asynchatNetwork programming based on transport layer TCP and UDP protocol
cgiCommon Gateway Interface
emailE-mail and MIME message processing
mailboxOperate mailboxes in different formats
mailcapMailcap file processing
httplib, http.clientSupport HTTP protocol, HTTP client
urllibURL processing
xmlrpc, xmlrpc.server, xmlrpc.clientServer side and client side of XML-RPC protocol

Clients supporting various protocols


Parse url string

urlparse()urlstring, scheme='', allow_fragments=TrueParse the url string and return the ParseResult object
urlunparse()ParseResult object (tuple)Reorganize the ParseResult object into urlstring

The properties of the ParseResult object and its position in the tuple

Index number012345
meaningagreementHostname + portResource pathAdditional parametersQuery stringidentifier
  • The default value is an empty string
from urllib.parse import *
a = urlparse('http://www.crazyit.ort:80/index.php;ye?name=fk#fr')
#Complete result: ParseResult(scheme='http', netloc='www.crazyit.ort:80', path='/index.php', params='yeeku', query='name=fk', fragment='fr ') 
print(a[0])        #http specifies the location (an element in the result)
print(a.scheme)    #http specify properties
print(a.geturl())  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr

b = urlunparse(a)  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr
urlunparse(('http', 'www.crazyit.ort:80', '/index.php', 'yeeku', 'name=fk', 'fr'))  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr
More propertiesusernamepasswordhostnameport
meaninguser namepasswordhost nameport
  • The default value is None

Interpret query string

methodMore parameterseffect
encoding = 'utf-8',
The parsing result is returned in the form of a dictionary
parse_qsl(qs)Parsing results are returned in list form
encoding =None,
quote_via = quote_plus
Restore the parsing result to a string
from urllib.parse import *
str_ = 'name=fkit&name=%E7%96%AF%E7%8B%82java&age=12'
a = parse_qs(str_)  #{'name': ['fkit', 'crazy java'], 'age': ['12']}
b = parse_qsl(str_) #[('name','fkit'), ('name', 'crazy java'), ('age','12 ')]
urlencode(a) #name=fkit&name=%E7%96%AF%E7%8B%82java&age=12
urlencode(b) #name=fkit&name=%E7%96%AF%E7%8B%82java&age=12

urllib. Method under request

Three methods are mainly used: parse, Request and urlopen. In addition, the error sub module of urllib is used to handle various exceptions caused by the Request module; There is a robot parser for parsing robots Txt file

  1. Request
    Request(url, data=None, header={}, origin_req_host=None, unverifiable=False, method=None)

The request method can be: GET, POST, PUT, PATCH, DELETE

from urllib.request import *
params = 'test'.encode('utf-8')
req = Request(url='', data=params, method='GET')
a = urlopen(req).status
b = urlopen(req).read().decode('utf-8')
  1. urlopen(url, data=None)
    The URL can be a URL string or a Request object. The former returns http client. Httpresponse object, which can be used to send GET and POST requests; The latter returns the Request object and can send more requests (set when creating the Request object)

GET step: GET http.client.HTTPResponse object -- read data (bytes) -- transcoding

from urllib.request import urlopen
url = ''
a = urlopen(url).read().decode('utf-8') 


To access the protected page on the Web, you can log in first, get the cookie, and then log in with the cookie (including session id)

Cookies can be saved directly in the cookie jar object or locally

  1. Output cookie
from http.cookiejar import *
from urllib.request import *
#Create Cookie Jar object -- create cookie processor object -- build opener
cookiejar = CookieJar()
handler   = HTTPCookieProcessor(cookiejar)
opener    = build_opener(handler)
#When the page is accessed, the cookie is automatically saved to the cookie jar"")
#<CookieJar[<Cookie BAIDUID=8B5D06A58F6D475B0F310E86B1F914CE:FG=1 for>, <Cookie BIDUPSID=8B5D06A58F6D475B10D9D3BA585B05B0 for>, <Cookie H_PS_PSSID=32617_1423_32734_32705_7626_32116_32719_22159 for>, <Cookie PSTM=1600391588 for>, <Cookie BDSVRTM=0 for>, <Cookie BD_HOME=1 for>]>
  1. Save cookie s locally
from http.cookiejar import *
from urllib.request import *
cookiejar   = MozillaCookieJar("cookie.txt")  #There is a save implementation
handler     = HTTPCookieProcessor(cookiejar)
opener      = build_opener(handler)
response ="")
  1. Read cookie s locally
from http.cookiejar import *
from urllib.request import *
cookiejar = MozillaCookieJar()  #With load implementation
handler  = HTTPCookieProcessor(cookiejar)
opener   = build_opener(handler)
response ="")
response ='utf-8')

#Network programming socket based on TCP protocol

Basis of agreement

TCP protocol: end-to-end protocol. After sending a message, you should receive a confirmation message. If you can't receive it, you will resend the message, which is reliable

IP protocol: the communication parties are computers and socket s

TCP protocol: both sides of communication are socket1 and socket2

Creating TCP server with socket

socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)

familyNetwork typeAF_INETIPv4

Method of socket object

socket connection

bind(address)socket binding address (IP address, port tuple) of the server
listen([backlogI])Listen to the socket of the server
connect(address)The socket of the client connects to the remote server. Error report exception
connect_ex(address)The socket of the client connects to the remote server. Error identifier returned on error
accept()Receive connections from clients
shoudown(how)To close the connection, you need to specify the connection method
close()Close connection (clean socket)


It is only applicable to one-stop communication protocols, such as HTTP protocol. It only sends request data once, reads response data once, and then closes completely

how parametermeaning
SHUT_RDOnly the input channel is closed
SHUT_WROnly close the output channel (used at the end of output)
SHUT_RDWRClose all (socket will not be cleaned)

Receive data from socket

When calling the following method, if no valid data is read, the thread will be blocked. There are two solutions:

  1. Multithreading. Each client connection has a thread
  2. selectors module. Allow socket s to communicate in a non blocking manner
    |Method | parameter | return|
    |recv()|bufsize, [flags]|bytes object|
    |recvfrom()|bufsize, [flags] | tuple: bytes, address|
    |recvmsg()|bufsize, [ancbufsize], [flags] | tuple: data, ancdata, msg_flags, address|
    |recv_into()|bufsize, [ancbufsize], [flags] | are equivalent to recv respectively_ into(),recvfrom_into(),recvmsg_into(), and put the received data into buffers|
    |recvfrom_into()| | |
    |recvmsg_into()| | |

Send data to socket

send()bytes, [flags]Commonly used in TCP protocol (connection established)
sendto()bytes, addressCommonly used in UDP protocol (connection not established)
sendfile()file, offset=0, count=NoneSend all contents of the file
  • Create associated file
    makefile(mode='r', buffering=None, *, encoding=None,errors=None, newline=None)

socket communication (TCP)

Server: create - bind - Listen

import socket
server_socket = socket.socket()
server_socket.bind(('', 30000))  #localhost for test and real IP for actual combat

Client: create - connect

user_socket = socket.socket()
user_socket.connect(('', 30000))

After connection

After connection, it communicates through socket objects without distinguishing between server and client

Note: call recv()

Multithreading - chat room case (one to many)

When sending and reading data, the thread will be blocked and the program cannot continue. The server and client can open a thread for each socket, put the program sending and reading data in the thread, and send the read data to the socket in L after sending and reading data - the number of clients = the number of threads

Note: in the following codes, run the server code first, and then the run client code

Server code

import socket, threading

def read_(user_socket):                 #Each user_socket represents a client
    try:     return user_socket.recv()  #Data received successfully, return data
    except:  L.remove(user_socket)      #Receiving failed, indicating that the client has been closed and moved out of the list

def test(user_socket):
        while True:  #Enter the dead cycle
            content = read_(user_socket)
            if content is None: break  #Skip when receive fails
            for user_socket in L:  #Send the information to all users in the L list
    except e: 

server_socket = socket.socket()
server_socket.bind(('', 30000)) 

L = []       #Create socket list
while True:  #Enter the dead cycle
    user_socket, addr = server_socket.accept()
    L.append(user_socket)  #Put all users in the L list
    threading.Thread(target=test, args=(user_socket,)).start()

Client code

Main thread: read the user's keyboard input

Sub thread: read socket data

import socket, threading

user_socket = socket.socket()
user_socket.connect(('', 30000))

def read_(server_socket):
    while True:  #Enter the dead cycle

threading.Thread(target=read_(server_socket), args(user_socket,)).start()
while True:  #Enter the dead cycle
    line = input('')
    if input('') is None or input('') == 'exit': break

Record user information - chat room case (one-to-one)

User 1 - server - User 2

Build a dictionary with user name as key and session id as value. Duplicate user names are not allowed

The selectors module allows socket s to communicate in a non blocking manner

Compared with socket, selectors add conditions to avoid blocking caused by unable to read data

parameterConditions that trigger the event
EVENT_READWhen data is readable; When a client is connected
EVENT_WRITEWhen writing data

Server code

Create - bind - Listen - set to non blocking - replace the dead loop of socket with the dead loop of selectors

import socket, selectors

def close_skt(skt):
def read_(skt, mask):   #Listener function for 'data readable' event
        data = skt.recv()
        if data:
            for user_socket in L:

def accept_(server_socket, mask):   #Listening function for the event "a client is connected"
    user_socket, addr = server_socket.accept()
    sel.register(user_socket, selectors.EVENT_READ, read_)

server_socket = socket.socket()
server_socket.bind(('', 30000)) 

L = []
sel = selectors.DefaultSelector()
sel.register(server_socket, selectors.EVENT_READ, accept_)

while True:
    events =       #Continuous monitoring of events
    for key, mask in events:
        callback =
        callback(key.fileobj, mask)

Client code (omitted)

#Network programming based on UDP protocol (omitted)

Advantages: Fast - no need to establish a connection; Just send it, regardless of whether the other party receives it or not

Disadvantages: unreliable; The size is limited to 64KB

Applications: Games, video conferencing

#Email support

EmailMessage create message

  • msg[‘to’]. addresses[0].username is the name of the first recipient
    |Method | function|
    |walk() | returns each part of the content in the form of an iteratable object|
    |set_content(str, type, code) | set the content and specify the text, type and code|
    |add_attachemnt(maintype, subtype, filename, cid=img) | add an attachment. Set the main type, subtype (restricted by the main type), file name and resource ID of the attachment (required for body reference)|
from email.message import EmailMessage
import email.utils

msg = EmailMessage()                    #Create mail object
msg.set_content('hi', 'html', 'utf-8')  #Set mail content, format and code
#Add attachment
file = open('1.png', 'rb').read()
a = email.utils.make_msgid()
msg.add_attachment(file, maintype='image', subtype='jpeq', filename='test.png', cid=a)

Email SMTP Lib

Tencent enterprise mailbox test failed. After entering the authorization code, you will still be prompted to enter the authorization code

  1. Enter the address and port of the SMTP server to connect to the sending server
  2. Enter the account and password and log in to the mailbox
import smtplib
from email.message import EmailMessage

msg = EmailMessage()  #Create message

conn = smtplib.SMTP_SSL('', 465)  
conn.login('', '123')
conn.sendmail('', '', msg.as_string())

Receive mail poplib

Used to connect to pop server. There are two classes, poplib POP3 (basic), poplib.POP3_SSL (SSL based)


  1. Enter the address and port of the POP server to connect to the receiving server
  2. Enter the account and password and log in to the mailbox
import poplib

conn = poplib.POP3_SSL('', 995)

Methods for POP3 objects

methodReturn tupleexplain
stat()(number and size of messages)
list()(server response code, number, size, octets of each email)octets: octal number
retr(N)(server response code, mail content, octets)N is the mail number, starting from 1
  • The message content is a byte list and needs to be spliced: b '\ r\n.join()
    msg = BytesParser(policy=default).parsebytes(data)

Get different parts of the message: msg ['from'] sender

Type of message content (maintype)meaning
multipartContainer. Including text, attachments, etc
Sample code to be supplemented

Parse mail

import os.path
content = msg.walk()

for message in msg.walk():
    a = message.get_content_maintype()
    if a == 'multipart':    print('container')
    elif a == 'text':       print('text', message.get_content())
    else:  #Save attachment to local
        name = message.get_filename()
        fp = open(os.path.join('.', name), 'wb')

Keywords: Python network Network Protocol TCP/IP

Added by webdevelope on Wed, 22 Dec 2021 00:22:29 +0200