Create autocrawler files with commands
Creating crawler files is based on the master scrapy to create crawler files
Scrapy genspider-l View scrapy's available master for creating crawler files
Available templates: Master description
Create basic crawler files
crawl Create Autocrawler Files
Create crawler files for crawling csv data
Create crawling xml data crawler files
Create a base master crawler, the same goes for the rest
Scrapy genspider-t master name crawler file name to crawl the domain name to create a base master crawler, the same as other
For example: scrapy genspider-t crawl Lagou www.lagou.com
First, configure items.py to receive data fields
default_output_processor = TakeFirst() defaults to use the ItemLoader class to load the items container class to fill in the data, which is a list type and can be retrieved from the list through the TakeFirst() method.
Input_processor = MapCompose (preprocessing function) sets the preprocessing function for the data field, which can be multiple functions
What I don't know in the process of learning can be added to me? python Learning Exchange Button qun，784758214 //There are good learning video tutorials, development tools and e-books in the group. //Share with you the current talent needs of python enterprises and how to learn python from zero foundation, and what to learn # -*- coding: utf-8 -*- # Define here the models for your scraped items # # See documentation in: # http://doc.scrapy.org/en/latest/topics/items.html #Item. py, the file is specially used to receive data information from the crawler, which is equivalent to container file. import scrapy from scrapy.loader.processors import MapCompose,TakeFirst from scrapy.loader import ItemLoader #Importing the ItemLoader class also loads the items container class to fill in the data. class LagouItemLoader(ItemLoader): #Custom Loader inherits the ItemLoader class and calls it on the crawler page to populate the data into the Item class default_output_processor = TakeFirst() #By default, the ItemLoader class is used to load the items container class to fill in the data, which is a list type. The contents of the list can be obtained through the TakeFirst() method. def tianjia(value): #Custom Data Preprocessing Function return 'Hawking Recording Network'+value #Return the processed data to Item class LagouItem(scrapy.Item): #Setting the information container class that the crawler gets title = scrapy.Field( #Receiving title information from Crawlers input_processor = MapCompose(tianjia), #When the name of the data preprocessing function is passed into the MapCompose method for processing, the formal parameter value of the data preprocessing function automatically receives the field title. )
The second step is to write the automatic crawler and use the ItemLoader class to load the items container class to fill the data.
Rule() Sets crawler rules
LinkExtractor() Sets url rules
callback='callback function name'
follow=True means to go deeper in crawling pages
LinkExtractor() makes rule judgment on the url acquired by the crawler
Allo= r'jobs/'is a regular expression, which means that only if it conforms to this url format can it be extracted.
deny= r'jobs/'is a regular expression that indicates that it conforms to this url format, does not extract and discard, as opposed to allow.
Allo_domains= www.lagou.com/ indicates that the connection under this domain name is extracted
deny_domains= www.lagou.com/ denotes that connections under this domain name are not withdrawn and discarded
The restrict_xpaths= xpath expression indicates that the crawler can use the xpath expression to restrict it to extract only the URL s of the specified area of a page
Rest_css= CSS selector, which means that the crawler can use CSS selector to restrict the URL to extract only one page's specified area
Tags='a'means that the crawler searches for url through a tag, which is set by default and can be done by default.
Attrs='href'denotes the href attribute of the tag a, which has been set by default and can be done by default.
Inheriting ItemLoader class with custom Loader class, loading items container class to fill data
ItemLoader() instantiates an ItemLoader object to load the ItemLoader container class and fill in the data, the same usage as ItemLoader inherited by custom Loader
First parameter: Note parentheses for the items container class to fill in the data.
The second parameter is response:
Method under ItemLoader object:
The add_xpath('field name','xpath expression') method fills the specified field with the data obtained from the XPath expression
The add_css('field name','css selector') method fills in the specified field with the data obtained by the CSS selector
The add_value('field name', string content) method fills the specified string data into the specified field
The load_item() method has no parameters and generates all the data. After the load_item() method is yield ed, the data is filled in the fields of the specified class of the items container.
# -*- coding: utf-8 -*- import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule from adc.items import LagouItem,LagouItemLoader #Import items container classes, and ItemLoader classes class LagouSpider(CrawlSpider): #Create reptiles name = 'lagou' #Reptilian name allowed_domains = ['www.luyin.org'] #Initial domain name start_urls = ['http://www.luyin.org/'] start url rules = ( #Configuration crawl list page rule Rule(LinkExtractor(allow=('ggwa/.*')), follow=True), #Configuration crawl content page rule Rule(LinkExtractor(allow=('post/\d+.html.*')), callback='parse_job', follow=True), ) def parse_job(self, response): #Callback function, note: Because the source code of the CrawlS template creates the parse callback function, remember that we can't create the function with the parse name. #Using ItemLoader class to load items container class to fill data item_loader = LagouItemLoader(LagouItem(), response=response) item_loader.add_xpath('title','/html/head/title/text()') article_item = item_loader.load_item() yield article_item
If you are still confused in the world of programming, you can join our Python Learning button qun: 784758214 to see how our predecessors learned. Exchange of experience. From basic Python script to web development, crawler, django, data mining, zero-base to actual project data are sorted out. To every Python buddy! Share some learning methods and small details that need attention. Click to join us. python learner gathering place
Schematic diagram of items.py file and crawler file