Basic knowledge of scratch crawler framework

Catalog

scrapy The concept of

scrapy The role of the framework

scrapy The process of

scrapy Three built-in objects of

scrapy The specific function of each module in

---

scrapy The concept of

Scrapy It's a Python Write the open source web crawler framework . It is designed to crawl network data 、 Framework for extracting structural data .

scrapy The role of the framework

A small amount of code , You can quickly grab

scrapy The process of

  The process can be described as follows ：

1. The first of the reptiles url Constructed as request object --> Crawler middleware --> engine --> Scheduler
2. Dispatcher request--> engine --> Download Middleware ---> Downloader
3. Downloader sends request , obtain response Respond to ----> Download Middleware ----> engine ---> Crawler middleware ---> Reptiles
4. Reptile extract url Address , Assemble into request object ----> Crawler middleware ---> engine ---> Scheduler , Repeat step 2
5. Crawler extraction data ---> engine ---> Pipeline processing and data storage

Be careful ：

• The green line in the figure represents the transfer of data
• Note the location of the middleware in the figure , Determined its function
• Pay attention to the location of the engine , All modules are previously independent of each other , Only interact with the engine

scrapy Three built-in objects of

• request Request object ： from url method post_data headers Etc
• response The response object ： from url body status headers Etc
• item Data objects ： The essence is a dictionary

scrapy The specific function of each module in

Be careful ：

• Crawler middleware and download middleware are only running in different logical locations , The effect is repetitive ： Replace UA etc.