Reptile lesson 1

Introduction to reptiles

From today on, let's learn something about reptiles ！ This is also the first time I have written a blog on this platform to record the learning process . If you have any mistakes, please give us more advice ！ Now let's start together ！！！

（ One ） What is a reptile

1.1 The concept of reptile
Crawlers are used to crawl data , Also called data acquisition program .
The data of the crawler comes from the network , And the network data can have web The server , Database server, cloud storage, etc .
notes ： Of course, it is legal to use crawlers to crawl data , For example, the data you want to crawl must be public and non-profit .
1.2. python The reptiles of
Use python Written crawler script （ Program ） Can complete timing , ration , Designated target （web Site ） Data crawling . It mainly uses many （ single ） Threads / process , Network request Library , Data analysis , data storage , Task scheduling and other related technologies .
python The crawler engineer can complete the interface test , Functional testing and integration testing .

（ Two ） Reptiles and web The relationship between back-end services

The crawler uses the web request Library , Equivalent to client request ,web The back-end server responds to the data according to the request .（ Here's the picture ）

The reptile is going to web The server initiates HTTP request , Accept the response data correctly , Then, according to the type of data （Content-Type） To parse and save data .

The crawler needs to forge the browser before sending the request （User-Agent Specify the request header ）, Then make a request to the server .

（ 3、 ... and ）Python Related libraries of crawler technology

Network request ：

• urllib
• requests
• selenium（UI Automatic test , dynamic js Rendering ）
• appium（ mobile phone app A reptile or UI test ）

Data analysis ：

• re Regular
• xpath
• bs4
• json

data storage ：

• pymysql
• mongodb
• elasticsearch

Multitask Library ：

• Multithreading （threading）/ The thread queue （queue）
• coroutines （asynio,gevent/eventlet）

The crawler frame ：

• scrapy
• scrapy-redis Distributed （ Multi machine crawler ）

（ Four ） Common anti - Crawler strategies

• UA（User-Agent） Strategy
• Login restrictions （cookie） Strategy
• Frequency of requests （IP agent ） Strategy
• Verification Code （ picture - Cloud code , Text or object selection , Slider, etc ） Strategy
• dynamic js（selenium/splash/api Interface ） Strategy

（ 5、 ... and ） The reptile library urllib

You can visit this website to see the detailed urllib Use of Library ！ Later, I will update the specific usage of this library for you .

https://www.runoob.com/python3/python-urllib.html

So that's the end of today's learning ！ See you next time ！