Basic knowledge

TCP/IP protocol

abbreviation	Full name	Protocol type
IP	Internet Protocol	Network protocol
TCP	Transmission Control Protocal	Transmission control protocol

IP address

Meaning: in the network, the unique identification of the communication entity (which can be host, printer, router port, etc.)

Format: 32bit integer, separated by dots into four 8-bit binary numbers, converted into decimal numbers between 0 and 255

port

Meaning: the entrance and exit for the application to communicate with the outside world. Different programs have different ports

Format: 16 bit integer, 0 ~ 65535, divided into three categories

Range	port type	purpose
0~1023	Recognized Well Known Port	Bind specific services
1024~49151	Register Registered Port	Application service common port
49152~5535	Dynamic and/or Private Port	Application service dynamic port, not actively used

URL

Uniform Resource Locator: consists of protocol name, host, port and resource path

Format: protocol://host:port/path

give an example: http://www.123.com/index.php

#Basic network support for Python

Built in network related modules

modular	effect
socket, asyncore, asynchat	Network programming based on transport layer TCP and UDP protocol
cgi	Common Gateway Interface
email	E-mail and MIME message processing
mailbox	Operate mailboxes in different formats
mailcap	Mailcap file processing
httplib, http.client	Support HTTP protocol, HTTP client
urllib	URL processing
xmlrpc, xmlrpc.server, xmlrpc.client	Server side and client side of XML-RPC protocol

Clients supporting various protocols

modular	ftblib	imaplib	nntplib	smtplib	poplib	telnetlib
agreement	FTP	IMAP4	NTTP	SMTP	POP3	TELNET

Parse url string

method	parameter	effect
urlparse()	urlstring, scheme='', allow_fragments=True	Parse the url string and return the ParseResult object
urlunparse()	ParseResult object (tuple)	Reorganize the ParseResult object into urlstring

The properties of the ParseResult object and its position in the tuple

attribute	scheme	netloc	path	params	query	fragment
Index number	0	1	2	3	4	5
meaning	agreement	Hostname + port	Resource path	Additional parameters	Query string	identifier

• The default value is an empty string

from urllib.parse import *
a = urlparse('http://www.crazyit.ort:80/index.php;ye?name=fk#fr')
print(a)  
#Complete result: ParseResult(scheme='http', netloc='www.crazyit.ort:80', path='/index.php', params='yeeku', query='name=fk', fragment='fr ') 
print(a[0])        #http specifies the location (an element in the result)
print(a.scheme)    #http specify properties
print(a.geturl())  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr

b = urlunparse(a)  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr
urlunparse(('http', 'www.crazyit.ort:80', '/index.php', 'yeeku', 'name=fk', 'fr'))  #http://www.crazyit.ort:80/index.php;yeeku?name=fk#fr

More properties	username	password	hostname	port
meaning	user name	password	host name	port

• The default value is None

Interpret query string

method	More parameters	effect
parse_qs(qs)	keep_blank_values=False, strict_parsing=False, encoding = 'utf-8', errors='replace'	The parsing result is returned in the form of a dictionary
parse_qsl(qs)		Parsing results are returned in list form
urlencode(query)	doseq=False, safe='', encoding =None, errors=None, quote_via = quote_plus	Restore the parsing result to a string

from urllib.parse import *
str_ = 'name=fkit&name=%E7%96%AF%E7%8B%82java&age=12'
a = parse_qs(str_)  #{'name': ['fkit', 'crazy java'], 'age': ['12']}
b = parse_qsl(str_) #[('name','fkit'), ('name', 'crazy java'), ('age','12 ')]
urlencode(a) #name=fkit&name=%E7%96%AF%E7%8B%82java&age=12
urlencode(b) #name=fkit&name=%E7%96%AF%E7%8B%82java&age=12

urllib. Method under request

Three methods are mainly used: parse, Request and urlopen. In addition, the error sub module of urllib is used to handle various exceptions caused by the Request module; There is a robot parser for parsing robots Txt file

1. Request
   Request(url, data=None, header={}, origin_req_host=None, unverifiable=False, method=None)

The request method can be: GET, POST, PUT, PATCH, DELETE

from urllib.request import *
params = 'test'.encode('utf-8')
req = Request(url='https://fkjava.org/', data=params, method='GET')
a = urlopen(req).status
b = urlopen(req).read().decode('utf-8')

1. urlopen(url, data=None)
   The URL can be a URL string or a Request object. The former returns http client. Httpresponse object, which can be used to send GET and POST requests; The latter returns the Request object and can send more requests (set when creating the Request object)

GET step: GET http.client.HTTPResponse object -- read data (bytes) -- transcoding

from urllib.request import urlopen
url = 'https://alexa.chinaz.com/lovetoywholesale.com'
a = urlopen(url).read().decode('utf-8') 

cookie

To access the protected page on the Web, you can log in first, get the cookie, and then log in with the cookie (including session id)

Cookies can be saved directly in the cookie jar object or locally

1. Output cookie

from http.cookiejar import *
from urllib.request import *
#Create Cookie Jar object -- create cookie processor object -- build opener
cookiejar = CookieJar()
handler   = HTTPCookieProcessor(cookiejar)
opener    = build_opener(handler)
#When the page is accessed, the cookie is automatically saved to the cookie jar
opener.open("http://www.baidu.com")
print(cookiejar)
#<CookieJar[<Cookie BAIDUID=8B5D06A58F6D475B0F310E86B1F914CE:FG=1 for .baidu.com/>, <Cookie BIDUPSID=8B5D06A58F6D475B10D9D3BA585B05B0 for .baidu.com/>, <Cookie H_PS_PSSID=32617_1423_32734_32705_7626_32116_32719_22159 for .baidu.com/>, <Cookie PSTM=1600391588 for .baidu.com/>, <Cookie BDSVRTM=0 for www.baidu.com/>, <Cookie BD_HOME=1 for www.baidu.com/>]>

1. Save cookie s locally

from http.cookiejar import *
from urllib.request import *
cookiejar   = MozillaCookieJar("cookie.txt")  #There is a save implementation
handler     = HTTPCookieProcessor(cookiejar)
opener      = build_opener(handler)
response = opener.open("http://www.baidu.com")
cookiejar.save()

1. Read cookie s locally

from http.cookiejar import *
from urllib.request import *
cookiejar = MozillaCookieJar()  #With load implementation
cookiejar.load('cookie.txt')
handler  = HTTPCookieProcessor(cookiejar)
opener   = build_opener(handler)
response = opener.open("http://www.baidu.com")
response = response.read().decode('utf-8')
print(response)

#Network programming socket based on TCP protocol

Basis of agreement

TCP protocol: end-to-end protocol. After sending a message, you should receive a confirmation message. If you can't receive it, you will resend the message, which is reliable

IP protocol: the communication parties are computers and socket s

TCP protocol: both sides of communication are socket1 and socket2

Creating TCP server with socket

socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)

parameter	meaning	Optional	agreement
family	Network type	AF_INET	IPv4
		AF_INET6	IPv6
type	Sock type	SOCK_STREAM	TCP
		SOCK_DGRAM	UDP
		SOCK_RAW	original

Method of socket object

socket connection

method	effect
bind(address)	socket binding address (IP address, port tuple) of the server
listen([backlogI])	Listen to the socket of the server
connect(address)	The socket of the client connects to the remote server. Error report exception
connect_ex(address)	The socket of the client connects to the remote server. Error identifier returned on error
accept()	Receive connections from clients
shoudown(how)	To close the connection, you need to specify the connection method
close()	Close connection (clean socket)

shutdown(how)

It is only applicable to one-stop communication protocols, such as HTTP protocol. It only sends request data once, reads response data once, and then closes completely

how parameter	meaning
SHUT_RD	Only the input channel is closed
SHUT_WR	Only close the output channel (used at the end of output)
SHUT_RDWR	Close all (socket will not be cleaned)

Receive data from socket

When calling the following method, if no valid data is read, the thread will be blocked. There are two solutions:

1. Multithreading. Each client connection has a thread
2. selectors module. Allow socket s to communicate in a non blocking manner
   |Method | parameter | return|
   |:----|:----|:----|
   |recv()|bufsize, [flags]|bytes object|
   |recvfrom()|bufsize, [flags] | tuple: bytes, address|
   |recvmsg()|bufsize, [ancbufsize], [flags] | tuple: data, ancdata, msg_flags, address|
   |recv_into()|bufsize, [ancbufsize], [flags] | are equivalent to recv respectively_ into()，recvfrom_into()，recvmsg_into(), and put the received data into buffers|
   |recvfrom_into()| | |
   |recvmsg_into()| | |

Send data to socket

method	parameter	explain
send()	bytes, [flags]	Commonly used in TCP protocol (connection established)
sendto()	bytes, address	Commonly used in UDP protocol (connection not established)
sendfile()	file, offset=0, count=None	Send all contents of the file

• Create associated file
  makefile(mode='r', buffering=None, *, encoding=None,errors=None, newline=None)

socket communication (TCP)

Server: create - bind - Listen

import socket
server_socket = socket.socket()
server_socket.bind(('192.168.1.88', 30000))  #localhost for test and real IP for actual combat
server_socket.listen()
server_socket.accept

Client: create - connect

user_socket = socket.socket()
user_socket.connect(('192.168.1.88', 30000))

After connection

After connection, it communicates through socket objects without distinguishing between server and client

Note: call recv()

Multithreading - chat room case (one to many)

When sending and reading data, the thread will be blocked and the program cannot continue. The server and client can open a thread for each socket, put the program sending and reading data in the thread, and send the read data to the socket in L after sending and reading data - the number of clients = the number of threads

Note: in the following codes, run the server code first, and then the run client code

Server code

import socket, threading

def read_(user_socket):                 #Each user_socket represents a client
    try:     return user_socket.recv()  #Data received successfully, return data
    except:  L.remove(user_socket)      #Receiving failed, indicating that the client has been closed and moved out of the list

def test(user_socket):
    try:
        while True:  #Enter the dead cycle
            content = read_(user_socket)
            print(content) 
            if content is None: break  #Skip when receive fails
            for user_socket in L:  #Send the information to all users in the L list
                user_socket.send(content.encode('utf-8'))
    except e: 
        print(e.strerror)

server_socket = socket.socket()
server_socket.bind(('192.168.1.88', 30000)) 
server_socket.listen()  

L = []       #Create socket list
while True:  #Enter the dead cycle
    user_socket, addr = server_socket.accept()
    L.append(user_socket)  #Put all users in the L list
    threading.Thread(target=test, args=(user_socket,)).start()

Client code

Main thread: read the user's keyboard input

Sub thread: read socket data

import socket, threading

user_socket = socket.socket()
user_socket.connect(('192.168.1.88', 30000))

def read_(server_socket):
    while True:  #Enter the dead cycle
        print(s.recv())

threading.Thread(target=read_(server_socket), args(user_socket,)).start()
while True:  #Enter the dead cycle
    line = input('')
    if input('') is None or input('') == 'exit': break
    user_socket.send(line.encode('utf-8'))

Record user information - chat room case (one-to-one)

User 1 - server - User 2

Build a dictionary with user name as key and session id as value. Duplicate user names are not allowed

The selectors module allows socket s to communicate in a non blocking manner

Compared with socket, selectors add conditions to avoid blocking caused by unable to read data

parameter	Conditions that trigger the event
EVENT_READ	When data is readable; When a client is connected
EVENT_WRITE	When writing data

Server code

Create - bind - Listen - set to non blocking - replace the dead loop of socket with the dead loop of selectors

import socket, selectors

def close_skt(skt):
    sel.unregister(skt)
    skt.close()
    L.remove(skt)
        
def read_(skt, mask):   #Listener function for 'data readable' event
    try:
        data = skt.recv()
        if data:
            for user_socket in L:
                user_socket.send(data)
        else:
            close_skt()  
    except:
        close_skt()

def accept_(server_socket, mask):   #Listening function for the event "a client is connected"
    user_socket, addr = server_socket.accept()
    L.append(cuser_socket)
    user_socket.setblocking(False)
    sel.register(user_socket, selectors.EVENT_READ, read_)

server_socket = socket.socket()
server_socket.bind(('192.168.1.88', 30000)) 
server_socket.listen() 
server_socket.setblocking(False)

L = []
sel = selectors.DefaultSelector()
sel.register(server_socket, selectors.EVENT_READ, accept_)

while True:
    events = sel.select()       #Continuous monitoring of events
    for key, mask in events:
        callback = key.data
        callback(key.fileobj, mask)

Client code (omitted)

#Network programming based on UDP protocol (omitted)

Advantages: Fast - no need to establish a connection; Just send it, regardless of whether the other party receives it or not

Disadvantages: unreliable; The size is limited to 64KB

Applications: Games, video conferencing

#Email support

EmailMessage create message

attribute	subject	from	to
meaning	theme	Sender	addressee

• msg[‘to’]. addresses[0].username is the name of the first recipient
  |Method | function|
  |:----|:----|
  |walk() | returns each part of the content in the form of an iteratable object|
  |set_content(str, type, code) | set the content and specify the text, type and code|
  |add_attachemnt(maintype, subtype, filename, cid=img) | add an attachment. Set the main type, subtype (restricted by the main type), file name and resource ID of the attachment (required for body reference)|

from email.message import EmailMessage
import email.utils

msg = EmailMessage()                    #Create mail object
msg.set_content('hi', 'html', 'utf-8')  #Set mail content, format and code
#Add attachment
file = open('1.png', 'rb').read()
a = email.utils.make_msgid()
msg.add_attachment(file, maintype='image', subtype='jpeq', filename='test.png', cid=a)

Email SMTP Lib

Tencent enterprise mailbox test failed. After entering the authorization code, you will still be prompted to enter the authorization code

1. Enter the address and port of the SMTP server to connect to the sending server
2. Enter the account and password and log in to the mailbox

import smtplib
from email.message import EmailMessage

msg = EmailMessage()  #Create message
msg.set_content('Hi')

conn = smtplib.SMTP_SSL('smtp.qq.com', 465)  
conn.login('123@qq.com', '123')
conn.sendmail('123@qq.com', '456@qq.com', msg.as_string())
conn.quit()

Receive mail poplib

Used to connect to pop server. There are two classes, poplib POP3 (basic), poplib.POP3_SSL (SSL based)

Steps:

1. Enter the address and port of the POP server to connect to the receiving server
2. Enter the account and password and log in to the mailbox

import poplib

conn = poplib.POP3_SSL('pop.exmail.qq.com', 995)
conn.user('sales5@lovetoysindustry.com')
conn.pass_('As1234561')

Methods for POP3 objects

method	Return tuple	explain
stat()	(number and size of messages)
list()	(server response code, number, size, octets of each email)	octets: octal number
retr(N)	(server response code, mail content, octets)	N is the mail number, starting from 1

• The message content is a byte list and needs to be spliced: b '\ r\n.join()
  msg = BytesParser(policy=default).parsebytes(data)

Get different parts of the message: msg ['from'] sender

Type of message content (maintype)	meaning
multipart	Container. Including text, attachments, etc
text	text
other	enclosure

Sample code to be supplemented

Parse mail

import os.path
content = msg.walk()

for message in msg.walk():
    a = message.get_content_maintype()
    if a == 'multipart':    print('container')
    elif a == 'text':       print('text', message.get_content())
    else:  #Save attachment to local
        name = message.get_filename()
        fp = open(os.path.join('.', name), 'wb')
        fp.write(message.get_payload(decode=True))