1. URLError

• First, explain the possible causes of URLError:

  • The network is not connected, that is, the local computer can not access the Internet.

  • Unable to connect to a specific server

  • Server does not exist

• In the code, we need to surround and capture the corresponding exception with try-except statement, the code is as follows:

from urllib.request import Request, urlopen
from fake_useragent import UserAgent
from urllib.error import URLError

url = 'http://www.sxt.cn/index/login/login12353wfeds.html' # Servers available, resources not available
url = 'http://www.sxt12412412.cn/index/login/login12353wfeds.html'
headers = {'User-Agent': UserAgent().chrome}
try:
    req = Request(url, headers=headers)
    resp = urlopen(req)
    info = resp.read().decode()
    print(info)
except URLError as e:
    if len(e.args) != 0:
        print('Address acquisition error!')
    else:
        print(e.code)
print('Climbing completed')

• Skills on debug mode:

• We use the urlopen method to access a non-existent web site. The results are as follows:

[Errno 11004] getaddrinfo failed

2. Use of request Libraries

2.1. Basic Introduction

1. Introduction:

   It is helpful to understand some basic concepts of reptiles and grasp the process of reptiles crawling. After introducing, we need to learn more advanced content and tools to facilitate our crawling. So this section gives a brief introduction to the basic usage of requests libraries.

2. install

   Install with pip:

pip install requests

1. Basic Request

req = requests.get("http://www.baidu.com")
req = requests.post("http://www.baidu.com")
req = requests.put("http://www.baidu.com")
req = requests.delete("http://www.baidu.com")
req = requests.head("http://www.baidu.com")
req = requests.options("http://www.baidu.com")

2.2. get request

• The parameters are dictionaries, and we can also pass json-type parameters:

1. Use of get 01:

import requests
from fake_useragent import UserAgent

url = 'http://www.baidu.com'
headers = {'User-Agent': UserAgent().chrome}
resp = requests.get(url, headers=headers)
resp.encoding='utf-8'
print(resp.text)

1. Use of get 02:

import requests
from fake_useragent import UserAgent

url = 'http://www.baidu.com/s?'
params = {
    'wd': 'Black Horse Programmer'
}
headers = {'User-Agent': UserAgent().chrome}
resp = requests.get(url, headers=headers, params=params)
resp.encoding = 'utf-8'
print(resp.text)

2.3. post request

1. The parameters are dictionaries, and we can also pass json-type parameters:
2. Code Example 01:

import requests
from fake_useragent import UserAgent

url = 'http://www.sxt.cn/index/login/login.html'
args = {
    'user': '17703181473',
    'password': '123456'
}
headers={'User-Agent':UserAgent().chrome}
resp = requests.post(url,headers=headers,data=args)
print(resp.text)

1. Code example 02:

import requests
from fake_useragent import UserAgent

# Sign in
login_url = 'https://www.kuaidaili.com/login/'

headers = {'User-Agent': UserAgent().chrome}
data = {
    'username': '398707160@qq.com',
    'passwd': '123456abc'
}

resp = requests.post(login_url, headers=headers, data=data)
print(resp.text)

2.4. Custom request header

• Camouflage request headers are often used in gathering, and we can use this method to hide:

headers = {'User-Agent': 'python'}
r = requests.get('http://www.zhidaow.com', headers = headers)
print(r.request.headers['User-Agent'])

2.5. Setting timeout time

• Timeout can be set through the timeout property, and if the response content is not available beyond that time, an error will be prompted.

requests.get('http://github.com', timeout=0.001)

2.6. Proxy access

1. In order to avoid blocked IP, proxy is often used. requests also has the corresponding proxies attribute:

import requests

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "https://10.10.1.10:1080",
}

requests.get("http://www.zhidaow.com", proxies=proxies)

1. If the agent needs an account and password, this is the case:

proxies = {
    "http": "http://user:pass@10.10.1.10:3128/",
}

1. Code example:

import requests
from fake_useragent import UserAgent

url = 'http://httpbin.org/get'
headers = {'User-Agent': UserAgent().chrome}

# proxy = {
#     'type': 'type://ip:port',
#     'type': 'type://username:password@ip:port'
# }
proxy = {
    'http':'http://117.191.11.102:8080'
    #'http': 'http://398707160:j8inhg2g@58.87.79.136:16817'

}

resp = requests.get(url, headers=headers, proxies=proxy)
print(resp.text)

2.7. session automatically saves cookies

1. Seeion means to maintain a session, such as: continue to operate (record identity information) after login, while requests are requests for a single request, and the identity information will not be recorded.

# Create a session object 
s = requests.Session()
# Set cookies by issuing get requests with session objects 
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')

1. Code example:

import requests
from fake_useragent import UserAgent

# Sign in
login_url = 'http://www.sxt.cn/index/login/login'

# Personal information
info_url = 'http://www.sxt.cn/index/user.html'
headers = {'User-Agent': UserAgent().chrome}
data = {
    'user': '17703181473',
    'password': '123456'
}
# Open session object and save cookie s in session
session = requests.Session()
resp = session.post(login_url, headers=headers, data=data)
# Get the response content (in strings)
print(resp.text)

info_resp = session.get(info_url, headers=headers)

print(info_resp.text)

2.8. ssl verification

# Disable Security Request Warning
requests.packages.urllib3.disable_warnings()

resp = requests.get(url, verify=False, headers=headers)

2.9. request for information

3. Data extraction

3.1. Regular expression re (the highest lattice; the fastest speed)

1. Extracting data

1. We've figured out how to get the content of the page before, but it's still a step away. How can we extract and sort out so much messy code with text? Let's start with a very powerful tool, regular expressions!

   Regular expression is a logical formula for string operation, which is to form a "regular string" by using pre-defined specific characters and combinations of these specific characters. This "regular string" is used to express a filtering logic for strings.

   Regular expressions are powerful tools for matching strings. There are also the concepts of regular expressions in other programming languages. Python is no exception. With regular expressions, it's easy for us to extract what we want from the returned page content.

2. Rules:

2. Relevant Annotations of Regular Expressions

1. The greedy and non-greedy modes of quantifiers:

   Regular expressions are often used to find matching strings in text
   Quantifiers in Python are greedy by default (or in a few languages they may be non-greedy by default), always trying to match as many characters as possible; non-greedy, on the contrary, always trying to match as few characters as possible.

   • For example, if the regular expression `ab*'is used to find `abbbc', it will find `abbb'. If we use the non-greedy quantifier "ab?", we will find "a".

2. Common methods:

   1. re.match

      • re.match tries to match a pattern from the beginning of the string, and if the match() is not successful, it returns none.
      • Functional grammar:
        re.match(pattern, string, flags=0)

   2. re.search

      • re.search scans the entire string and returns the first successful match.
      • Functional grammar:
        re.search(pattern, string, flags=0)

   3. re.sub

      • Subsubstitution string

      • Grammar:

        re.sub(pattern,replace,string)

   4. re.findall

      • Findall Find All

      • Grammar:

        re.findall(pattern,string,flags=0)

3. Regular expression modifier-optional flag:

   Regular expressions can contain optional token modifiers to control the pattern of matching. The modifier is specified as an optional flag. Multiple flags can be specified by bitwise OR(|). For example, re.I | re.M is set to the I and M flags:

• Code example:

import re

str1 = 'I study Python3.6 everday!'

				############ match ############
print('-' * 30, 'match()', '-' * 30)  
# match from left to right, matching in turn (the previous matching also needs to be matched), if matching is not directly returned to None

# m1 = re.match(r'I', str1)
# m1 = re.match(r'[I]', str1)
# m1 = re.match(r'\bI', str1)
# m1 = re.match(r'\w', str1)
# m1 = re.match(r'\S', str1) 
# m1 = re.match(r'(I)', str1)
# m1 = re.match(r'.', str1)
# m1 = re.match(r'\D', str1)
m1 = re.match(r'\w\s(study)', str1)
print(m1.group(1))

				############ search ############
print('-' * 30, 'search()', '-' * 30)  
# From left to right, scan all, find the first and return the result.
s1 = re.search(r'study', str1)
s1 = re.search(r'y', str1)
print(s1.group())

				############ findall ############
print('-' * 30, 'findall()', '-' * 30)
f1 = re.findall(r'y', str1)
f1 = re.findall(r'Python3.6', str1)
f1 = re.findall(r'P\w*.\d', str1)
print(f1)

				############ sub ############
print('-' * 30, 'sub()', '-' * 30)
su1 = re.sub(r'everday', 'Everday', str1)
su1 = re.sub(r'ev.+', 'Everday', str1)
print(su1)

print('-' * 30, 'test()', '-' * 30)
str2 = '<span><a href="http://Www.bjstx.com "> Silicon Valley sxt</a> </span>"

# t1 = re.findall(r'[\u4e00-\u9fa5]+', str2)
# t1 = re.findall(r'>([\u4e00-\u9fa5]+)<', str2)
# t1 = re.findall(r'>(\S+?)<', str2)
t1 = re.findall(r'<a href=".*">(.+)</a>', str2)
t1 = re.findall(r'<a href="(.*)">.+</a>', str2)
print(t1)
t2 = re.sub(r'span', 'div', str2)
t2 = re.sub(r'<span>(.+)</span>', r'<div>\1</div>', str2)
print(t2)

1. Exercise: Crawl the first three pages of the Encyclopedia of Gongshi, only the content of the paragraph.

import requests
from fake_useragent import UserAgent
import re

with open('duanzi.txt', 'w', encoding='utf-8') as f:
    for i in range(1, 4):
        url = 'https://www.qiushibaike.com/text/page/{}/'.format(i)
        headers = {'User-Agent': UserAgent().chrome}
        resp = requests.get(url, headers=headers)
        html = resp.text
        infos = re.findall(r'<div class="content">\s<span>\s+(.+)', html)
        for info in infos:
            f.write('-' * 30 + '\n')
            f.write(info.replace(r'<br/>','\n'))
            f.write('\n' + '-' * 30 + '\n')

3.2. Beautiful Soup

1. Introduction, Installation and Four Categories

1. Beautiful Soup provides some simple, python-like functions for navigating, searching, and modifying analysis trees. It's a toolbox that provides users with data to grab by parsing documents. Because it's simple, it doesn't need much code to write a complete application.

   Beautiful Soup automatically converts input documents into Unicode encoding and output documents into utf-8 encoding. You don't need to think about coding, unless the document does not specify a coding method, then Beautiful Soup can't automatically identify the coding method. Then, you just need to explain the original encoding.

   Beautiful Soup has become an excellent python interpreter like lxml and html6lib, providing users with flexible different parsing strategies or strong speed

2. Installation: Beautiful Soup 3 is currently out of development. It is recommended to use Beautiful Soup 4 in current projects, but it has been ported to BS4, which means we need import bs4 when importing.

   pip install beautifulsoup4
   pip install lxml
   

   Beautiful Soup supports HTML parsers in Python standard libraries and some third-party parsers. If we don't install it, Python will use Python's default parser. The lxml parser is more powerful, faster and recommended for installation.

1. Create Beautiful Soup Objects

   from bs4 import BeautifulSoup
   bs = BeautifulSoup(html,"lxml")
   

2. Four categories of objects:

   Beautiful Soup converts complex HTML documents into a complex tree structure. Each node is a Python object. All objects can be summarized into four types:

   • Tag

   • NavigableString

   • Beautiful Soup

   • Comment (not often used)

2. Tag

• Popularly, it's a tag in HTML; for example, <div> <title>.

• How to use it:

# Take the following code as an example
<title id='title'>Ada Tam</title>
<div class='info' float='left'>Welcome to SXT</div>
<div class='info' float='right'>
    <span>Good Good Study</span>
    <a href='www.bjsxt.cn'></a>
    <strong><!--Useless--></strong>
</div>

1. Get tags:

# Parsing in lxml
soup = BeautifulSoup(info, 'lxml')
print(soup.title)
# < title > Shangxue </title >

• Note: The same tag can only get the first tag that meets the requirements.

1. Get properties:

# Get all attributes
print(soup.title.attrs)
# class='info' float='left'

# Get the value of a single attribute
print(soup.div.get('class'))
print(soup.div['class'])
print(soup.a['href'])
# info

3. Navigable String Gets Content

print(soup.title.string)
print(soup.title.text)
#Ada Tam

4. BeautifulSoup

• BeautifulSoup object represents the entire content of a document, and most of the time it can be treated as a Tag object, which supports traversing the document tree and searching for most of the methods described in the document tree.

  Because the BeautifulSoup object is not a real HTML or XML tag, it has no name and attribute attributes. But sometimes it's convenient to look at its. name attribute, so the BeautifulSoup object contains a special attribute. name with a value of "[document]".

print(soup.name)
print(soup.head.name)
# [document]
# head

5. Comment

• Comment object is a special type of Navigable String object. In fact, the output still does not include annotation symbols, but if it is not handled properly, it may cause unexpected trouble to our text processing.

if type(soup.strong.string) == Comment:
    print(soup.strong.prettify())
else:
    print(soup.strong.string)

6. Searching Document Tree

• Beautiful Soup defines a number of search methods, focusing on two: find() and find_all(). The parameters and usage of other methods are similar. Please give us a second opinion.

• Filter:

  Before introducing the find_all() method, let's first introduce the types of filters that run through the entire search API. Filters can be used in tag name, node attributes, strings, or their mix.

1. Character string

The simplest filter is a string. When a string parameter is passed into the search method, Beautiful Soup finds the content that matches the string completely. The following example is used to find all of the contents in the document.

# Returns all div Tags
print(soup.find_all('div'))

If a bytecode parameter is passed in, Beautiful Soup will be encoded as UTF-8, and a Unicode code code can be passed in to avoid a Beautiful Soup parsing error.

1. regular expression

If a regular expression is passed in as a parameter, Beautiful Soup matches the content through the match() of the regular expression.

# Returns all div Tags
print (soup.find_all(re.compile("^div")))

1. list

If a list parameter is passed in, Beautiful Soup returns the content that matches any element in the list.

# Returns all matched span, a Tags
print(soup.find_all(['span','a']))

1. keyword

If a parameter with a specified name is not a search for the built-in parameter name, it will be searched as an attribute of the specified name tag. If a parameter with a name ID is included, Beautiful Soup will search for the "id" attribute of each tag.

#Returns the label with id welcom
print(soup.find_all(id='welcom'))

1. True

True can match any value. The following code finds all tag s, but does not return string nodes.

1. Search by CSS

Searching tags according to CSS class names is very useful, but the keyword class identifying CSS class names is a reserved word in Python. Using class as a parameter can lead to grammatical errors. Starting from version 4.1.1 of Beautiful Soup, tags with specified CSS class names can be searched through class_parameter.

# Returns a div whose class equals info
print(soup.find_all('div',class_='info'))

1. Search by property

soup.find_all("div", attrs={"class": "info"})

7. CSS selector (extension)

• Sop. select (parameter):

8. Code examples

# pip install bs4
# pip install lxml
from bs4 import BeautifulSoup
from bs4.element import Comment

str1 = '''
<title id='title'>Ada Tam</title>
<div class='info' float='left'>Welcome to SXT</div>
<div class='info' float='right'>
    <span>Good Good Study</span>
    <a href='www.bjsxt.cn'></a>
    <strong><!--Useless--></strong>
</div>
'''
soup = BeautifulSoup(str1, 'lxml')
print('-' * 30, 'Get Tags', '-' * 30)
print(soup.title)
print(soup.span)
print(soup.div)

print('-' * 30, 'get attribute', '-' * 30)
print(soup.div.attrs)
print(soup.div.get('class'))
print(soup.a['href'])

print('-' * 30, 'Getting content', '-' * 30)
print(type(soup.title.string))
print(soup.title.text)

print(type(soup.strong.string))
print(soup.strong.text)

if type(soup.strong.string) == Comment:
    print('There are notes!')
    print(soup.strong.prettify())

print('-' * 30, 'find_all()', '-' * 30)
print(soup.find_all('div'))
print(soup.find_all(id='title'))
print(soup.find_all(class_='info'))
print(soup.find_all(attrs={'float': 'right'}))

print('-' * 30, 'select()', '-' * 30)
print(soup.select('a'))
print(soup.select('#title'))
print(soup.select('.info'))
print(soup.select('div span'))
print(soup.select('div > span'))

3.3. Xpath

• You can install the Xpath Helper plug-in on Google Chrome.

1. Introduction and installation

1. Beautiful Soup is already a very powerful library, but there are some popular parsing libraries, such as lxml, using Xpath grammar, which is also a more efficient parsing method. If you're not used to Beautiful Soup, try Xpath.

   Official website http://lxml.de/index.html

   w3c http://www.w3school.com.cn/xpath/index.asp

2. Installation:

   pip install lxml
   

2. Xpath grammar

1. XPath is a language for finding information in XML documents. XPath can be used to traverse elements and attributes in XML documents. XPath is the main element of W3C XSLT standard, and XQuery and XPointer are built on XPath expression.

2. Node relationship:

   • Parent
   • Children
   • Sibling
   • Ancestor
   • Descendant

3. Acquisition node:

1. Commonly used path expressions:

1. Wildcards: XPath wildcards can be used to select unknown XML elements.

1. Select several paths: By using the "|" operator in the path expression, you can select several paths

1. Predicates: Predicates are embedded in square brackets to find a particular node or node containing a specified value

1. Xpath operator

4. Use

1. Examples:

from lxml import etree

text = '''
<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a>
     </ul>
 </div>
'''

html = etree.HTML(text)
result = etree.tostring(html)
print(result)

First we use the etree Library of lxml, then we initialize it with etree.HTML, and then we print it out.

Among them, this reflects a very practical function of lxml is to automatically modify the HTML code, you should note that the last li tag, in fact, I deleted the tail tag, is not closed. However, lxml inherits the features of libxml 2 and has the function of automatically modifying HTML code.

So the output is as follows:

<html><body>
<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a></li>
</ul>
</div>

</body></html>

It not only completes the li tag, but also adds body and html tag.
File Reading

In addition to reading strings directly, it also supports reading content from files. For example, we create a new file called hello.html, which contains:

<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a></li>
     </ul>
 </div>

The parse method is used to read files:

from lxml import etree
html = etree.parse('hello.html')
result = etree.tostring(html, pretty_print=True)
print(result)

The same results can also be obtained.

2. XPath uses:

1. Get all < li > tags

from lxml import etree
html = etree.parse('hello.html')
print (type(html))
result = html.xpath('//li')
print (result)
print (len(result))
print (type(result))
print (type(result[0]))

• Operation results:

<type 'lxml.etree._ElementTree'>
[<Element li at 0x1014e0e18>, <Element li at 0x1014e0ef0>, <Element li at 0x1014e0f38>, <Element li at 0x1014e0f80>, <Element li at 0x1014e0fc8>]

<type 'list'>
<type 'lxml.etree._Element'>

It can be seen that the type of etree.parse is ElementTree. After calling xpath, we get a list of five < li > elements, each of which is Element type.

1. Get all class es of the < li > tag

result = html.xpath('//li/@class')
print (result)

• Operation results:

['item-0', 'item-1', 'item-inactive', 'item-1', 'item-0']

1. Get the < a > label with href as link 1. HTML under < li > label

result = html.xpath('//li/a[@href="link1.html"]')
print (result)

• Operation results:

[<Element a at 0x10ffaae18>]

1. Get all < span > tags under < li > tags

   Note: This is not correct.

result = html.xpath('//li/span')

#Because / is used to get child elements, and < span > is not < li > child elements, so use parallel slash bars
result = html.xpath('//li//span')
print(result)

• Operation results:

[<Element span at 0x10d698e18>]

1. Get all class es under the < li > tag, excluding < li >

result = html.xpath('//li/a//@class')
print (resul)t
# Operation results
# ['blod']

1. Get the last < li > href of < a >

result = html.xpath('//li[last()]/a/@href')
print (result)

• Operation results:

['link5.html']

1. Get the content of the penultimate element

result = html.xpath('//li[last()-1]/a')
print (result[0].text)

• Operation results:

fourth item

1. Get the label signature with class as bold

result = html.xpath('//*[@class="bold"]')
print (result[0].tag)

• Operation results:

span

Select the nodes in the XML file:

• Element (element node)
• Attribute (attribute node)
• Text (text node)
• Concat (element node, element node)
• Comment (comment node)
• Root (root node)

5. Code examples

• In the vertical and horizontal web, crawl the first three pages of data, as long as the title of the book

from lxml import etree
import requests
from fake_useragent import UserAgent

url = 'http://book.zongheng.com/store/c1/c0/b0/u0/p1/v9/s1/t0/u0/i1/ALL.html'
headers = {'User-Agent': UserAgent().chrome}
resp = requests.get(url, headers=headers)
html = resp.text

# Constructing Analytical Object etree
e = etree.HTML(html)

# Title
names = e.xpath('//div[@class="bookname"]/a/text()')
# author
authors = e.xpath('//div[@class="bookilnk"]/a[1]/text()')
# Mode 01: If there is no author, it will not correspond.
for i in range(len(names)):
    print('{}:{}'.format(names[i], authors[i]))
# Mode 02: If the number of iterators is different, choose the shorter one.
for n, a in zip(names, authors):
   print('{}:{}'.format(n,a))

('//li[last()-1]/a')
print (result[0].text)

- Operation results:

fourth item

8. Obtain class by bold Tag signature

```python
result = html.xpath('//*[@class="bold"]')
print (result[0].tag)

• Operation results:

span

Select the nodes in the XML file:

• Element (element node)
• Attribute (attribute node)
• Text (text node)
• Concat (element node, element node)
• Comment (comment node)
• Root (root node)

5. Code examples

• In the vertical and horizontal web, crawl the first three pages of data, as long as the title of the book

from lxml import etree
import requests
from fake_useragent import UserAgent

url = 'http://book.zongheng.com/store/c1/c0/b0/u0/p1/v9/s1/t0/u0/i1/ALL.html'
headers = {'User-Agent': UserAgent().chrome}
resp = requests.get(url, headers=headers)
html = resp.text

# Constructing Analytical Object etree
e = etree.HTML(html)

# Title
names = e.xpath('//div[@class="bookname"]/a/text()')
# author
authors = e.xpath('//div[@class="bookilnk"]/a[1]/text()')
# Mode 01: If there is no author, it will not correspond.
for i in range(len(names)):
    print('{}:{}'.format(names[i], authors[i]))
# Mode 02: If the number of iterators is different, choose the shorter one.
for n, a in zip(names, authors):
   print('{}:{}'.format(n,a))