Network request

Among Python's many HTTP clients, the most famous are requests, aiohttp and httpx.

Requests can only send synchronization requests without the help of other third-party libraries; AIO HTTP can only send asynchronous requests; httpx can send both synchronous and asynchronous requests.

So how to choose

• Only synchronous requests are sent, but they can cooperate with multi-threaded mutation steps.
• aiohttp is only used for asynchronous requests, but it can be synchronized with await.
• httpx can send synchronous or asynchronous requests, but the request speed is slightly worse than requests and asynchronous than AIO http

The power of Asyncio. However, any scheme is not perfect and has certain limitations, as does Asyncio.

In actual use, if you want to make good use of Asyncio, especially its powerful functions, you must have corresponding Python library support in many cases.

such as

The requests library is not Asyncio compatible, while the AIO HTTP library is.

requests

Let's start with requests

Installation dependency

pip install requests

response

Type of response

#Gets the string data returned by the interface
r.text

#Get the json data returned by the interface, that is, directly convert the data in json format into json objects
r.json()

#Get the binary data returned by the interface, assuming that if the binary data is a picture, it can continue to be converted into a picture
r.content

#Get the original socket and use r.raw. Please add the parameter stream=True in the requests request
r.raw

Get additional information about the request response

#Get status code
r.status_code

#Get the url of the request
r.url

#Get specified cookie information
r.cookies['token']

#Get the response header information returned to us by the access server
r.headers

#Get the response header information returned to us by the specified access server
r.headers['Content-Type']

#Gets information about the header of the request sent to the server
r.request.headers

request

GET request

get request:

res = requests.get(url,data=data,cookies=cookie,headers=header,verify=False,files=file)

data can be transferred or not. data is in dictionary format. If the url is https, add verify=False. If the url is http, do not add.

Example 1

import requests

if __name__ == '__main__':
    r = requests.get("https://www.psvmc.cn")
    print(r.text)

Example 2

import requests

if __name__ == '__main__':
    data = {'username': 'admin', 'passwd': '123456'}
    r = requests.get("https://www.psvmc.cn/login.json", params=data)
    print(r.status_code)
    print(r.json()["obj"])

POST request

url_post = "https://www.psvmc.cn/login.json"

#Request without any parameters
r = requests.post(url_post)

#For requests that do not contain any parameters, the timeout is set to 10s. If the timeout is not set, the default is 60s
r = requests.post(url_post,timeout=10)

#Request with parameters, dict_param is the parameter dictionary. The default is data=dict_param, using data = means that the post is a form request
#application/x-www-form-urlencoded.
r = requests.post(url_post, data=dict_param)

#Request with parameters, dict_param is a parameter dictionary. If json = is used, it means that the post is a json request
r = requests.post(url_post, json=dict_param)

#For the request with parameters, the body passes the string. Here is the JSON string.
r = requests.post(url_post, data=json.dumps(dict_param))

#Request with parameters, dict_param is the parameter dictionary, sets the timeout of 10s, and carries the headers attribute
r = requests.post(
    url_post,
    data=dict_param,
    timeout=10,
    headers={
        'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)'
    })

#post request upload file
url = 'http://apinum/upload/post'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)

Other types of requests

r = requests.put(url, data =dict_param)
r = requests.delete(url)
r = requests.head(url)
r = requests.options(url)

agent

Proxy access can be considered when cross domain. Whether it is a post request or a get request, you only need to add proxies.

There is no need to consider cross domain issues during client development, and there is no need to set up proxy access.

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get(url_get, proxies=proxies)

Check whether the agent is valid

It works the same as telnet

import telnetlib

if __name__ == '__main__':
    try:
        telnetlib.Telnet('110.242.68.4', port='80', timeout=3)
    except:
        print('ip Invalid!')
    else:
        print('ip Valid!')

Asynchronous request

The code of AIO HTTP coincides with that of httpx asynchronous mode by 90%, except that AsyncClient is replaced by ClientSession.

aiohttp

Installation dependency

pip install aiohttp

Example

import aiohttp
import asyncio


async def main():
    async with aiohttp.ClientSession() as client:
        resp = await client.post(
            'https://www.psvmc.cn/login.json',
            json={'keyword': '123'}
        )
        result = await resp.text()
        print(result)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

report errors

RuntimeError: Event loop is closed

hold

asyncio.run(main())

Change to

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Asynchronous request

import asyncio
import aiohttp
import time


async def download_one(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            print('Read {} from {}'.format(resp.content_length, url))


async def download_all(sites):
    tasks = [asyncio.ensure_future(download_one(site)) for site in sites]
    await asyncio.gather(*tasks)


def main():
    sites = [
        'https://www.psvmc.cn/index.html',
        'https://www.psvmc.cn/login.json',
        'https://www.psvmc.cn/userlist.json'
    ]
    start_time = time.perf_counter()

    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(download_all(sites))
    finally:
        if loop.is_running():
            loop.close()
    end_time = time.perf_counter()
    print('Download {} sites in {} seconds'.format(
        len(sites), end_time - start_time))


if __name__ == '__main__':
    main()

httpx

Details: https://www.psvmc.cn/article/2021-11-26-python-httpx.html

JSON

String to object

import json

# Some JSON:
x =  '{ "name":"Bill", "age":63, "city":"Seatle"}'

# Parse x:
y = json.loads(x)

# The result is a Python Dictionary:
print(y["age"])

Object to string

import json

# Python object (Dictionary):
x = {
  "name": "Bill",
  "age": 63,
  "city": "Seatle"
}

# Convert to JSON:
y = json.dumps(x)

# The result is a JSON string:
print(y)

When Python is converted to JSON, python objects are converted to JSON (JavaScript) equivalents: