1、 Reptile concept and basic process

One 、 The concept of reptile

• Web crawler —— It is a kind of installation of Yidi port rules , A program or script that automatically obtains Internet information . Due to Internet data 䣌 Diversity and limited resources , According to user needs, we can crawl relevant web pages and analyze what has been called the mainstream crawling strategy

• The crawler can crawl all the data that can be accessed through the browser

• The essence of reptiles is ： Simulation browser open web page , Get the data we want in the web page

Two 、 Basic process of reptile

1. preparation
   • View the analysis target web page through the browser , Learn basic programming specifications
2. get data
   • adopt HTTP The library sends requests to the target site , The request can contain additional header Etc , If the server can respond normally , Get one back Response, This is the page content you want to get
3. Parsing content
   • What you get may be HTML、json Equiform , You can use the page parsing library 、 Regular expressions, etc
4. Save the data
   • There are many ways to save data , Can be saved as text , It can also be saved to a database , Or save a specific format file .

If you think the article is good , You can give me some likes
Pay attention to me , We learn together and make progress together ！