There are 3 ways to parse Python web pages, which are worth collecting

Generally speaking, when we crawl the whole source code of the web page, we need to analyze the web page.

There are three normal analytical methods
① : regular match resolution
② : BeatuifulSoup parsing
③ : lxml parsing

Regular match resolution:

In the previous study, we learned the basic usage of reptiles, such as / s,/d,/w, *, +,? However, when parsing the crawled web page, it is not enough to only use these basic usages. Therefore, we need to understand the classical functions of regular matching in Python.


runoob explanation: re Match attempts to match a pattern from the starting position of the string. If the matching is not successful, match will return none. If the matching is successful, re Match will return a matching object, otherwise it is null To view the objects returned by matching, use the method of the group function.



Explanation: pattern is the regular expression for matching, string is the character to be matched, and flags is the flag bit, which is used to match whether it is case sensitive, multi line matching, etc. The specific usage is shown in the following table:

Modifier describe
re.IMake matching case insensitive
re.MMulti line matching, affecting ^ and$
re.SMake Match all characters including line feed (under normal circumstances, if line feed is not matched, this behavior is very dangerous and easy to be bypassed by hackers)
re.UThis flag affects \ w,\W,\b,\B when characters are parsed according to the Unicode character set

Code case:

import re
line="Anyone who dreams there gets"
match=re.match(r'(.*) dreams (.*?).*',line,re.M|re.I)
#match1=re.match(r'(.*) Dreams (.*?)',line,re.M)#Case sensitive, matching will fail
match2=re.match(r'(.*) Dreams (.*?).*',line,re.M|re.I)#Non greedy model
match3=re.match(r'(.*) Dreams (.*).*',line, re.M|re.I)#Greedy model
print("match 0:", who dreams there gets
print("match 1:", who
print("match 2:",' '
print("match2 0:", who dreams there gets
print("match2 1:", who
print("match2 2:",' '
print("match3 0:", who dreams there gets
print("match3 1:", who
print("match3 2:", gets

You can see (. *?) The non greedy pattern will match the space and put it into group(2). By default, group()==group(0) matches the whole sentence, and the groups() function will put the matching result into a list. For the results returned by matching, such as match, we can use start(),end(),span() to view the start and end positions.

The R of the regular matching string r '(. *)' represents raw string and pure matching string. If it is used, the backslash in the backquote will not be specially processed. For example, the newline \ n will not be escaped. Therefore, we can understand the escape in the following regular matching:

'\ \' will be escaped without adding r, because the string has undergone two transitions. One is string escape, which becomes' \ 'and then regular escape.

If you add r, it will become '\' because matching with a pure string cancels the string escape. This is the escape process in regular matching. and re Match differs in re Search scans the entire string and returns the first result. In addition, re Match doesn't make much difference. You can also use m.span(),m.start(),m.end() to view the starting position and so on

Goal: crawl the titles of all python videos in the first bilibili search bar


import requests
import re
headers={"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0'}
title_list=re.findall('<a title=(.*?) href=.*?>',text)
for i in title_list:

Key content: title_list=re.findall(’ ',text)

the second.? Indicates the result of matching conditions. The first (.?) Indicates to extract the results that meet the conditions

In this way, we can extract the title

BeautifulSoup parsing

Beautiful soup can extract data from HTML or XML, which is similar to a toolbox. When using beautiful soup, you can use some parsers to make the speed faster. At present, the better Lxml HTML parser is used as follows:


The essence of beautiful soup is a complex tree structure. There are generally three methods to extract its objects:

① : traverse the document tree, which is similar to the file directory of Linux. Since the tree structure must be from top to bottom, the first node must be the header. The specific method is as follows:

#To get a label that starts with < H1 >, you can use
#Content similar to < H1 id = "name" > Culin < / H1 > will be returned
#If you want to get all the child nodes of the div, you can use
#If you want to get all the child nodes of the div, you can use
#If you want to get the descendant nodes of div, you can use
#If you want to get the parent node of node a, you can use

② : search document tree:

Traversing the document tree has great limitations. Generally, it is not recommended. It is more to search the document tree. The use method is very convenient. The essence lies in find and find_all function, for example, we need to crawl the title of the blog post:

import requestss
from bs4 import BeautifulSoup
#Print the first title
#Print all articles
for i in range(len(all_titile)):
    print("No ",i," is ",all_title[i].a.text.strip())

③ : CSS selector:

CSS can be used as a document tree shrew to extract data, or it can be searched according to the method of searching the document tree. However, I personally think the CSS selector is not as easy to use as selenium, and it is very inconvenient to use, so I won't repeat it

In addition, the lxml parser focuses on xpath, which is also well reflected in selenium. For the choice of xpath, it is more convenient for us to directly check xpath in web page inspection.

Project practice

Crawling shell website, Nanjing housing information, total house price, average house price, location

import requests
import re
from bs4 import BeautifulSoup
headers={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0","Host":
for i in range(1,20):
    for house in houst_list:
        total_money=str(house.find('div',class_='totalPrice').span.text.strip())+'ten thousand'
        text_="title:"+name+"House information:"+houst_info+"Total price:"+total_money+' '+aver_money+adderess
        with open ('haha.txt',"a+") as f:

The result returns a haha Txt, as follows:

Keywords: Python

Added by HDMICable on Sat, 19 Feb 2022 05:29:57 +0200