For Xiaobai who just learned to crawl, you can understand it after reading it

Preface

I wrote the library reservation script and found that it attracted a lot of Xiaobai , Then I found that they had learned to crawl , But you may not know the nature and principle of reptiles , You may feel that you have understood the video or tutorial , Then I changed a website and found that I knew the one in the tutorial . Because it is not the implementation of the code that is difficult to write a crawler , How hard it can be to write some little crawler code ？ Not just Baidu . The main difficulty is packet analysis , After reading this article, you can basically crawl all the small websites .

The essence of reptiles

We click... In the browser 、 Operations such as input can be simplified to the following figure .
The following is a simplified login flowchart .
This is a simplified diagram , But you can tell , What users do on the browser is essentially the browser sending packets to the server 、 Receive packets .
Whether it's a reptile （ for example ： Crawling novels ） Or automated scripts （ for example ： Grab seats in the library ） The essence of is to send packets 、 Receive packets , Pictured ： You can see , To log in, we only need to send a key packet , You don't need so many useless packets . Of course this is a simplified version , There are some websites that can send more than one packet when logging in , You need to grab your own bags 、 The contract tests analyzed .
Seeing this, some people may have been unable to bear the excitement to find a website to try , Don't worry , Otherwise, it would be a bit awkward to come back later .

Tips for analyzing packets

If you think that opening a web page or doing an operation is sending a packet, it's naive . Don't believe it f12 Then open Baidu , The data package makes your scalp numb .52 A request , Just , Very ignorant .
But this is a big website , So much is normal , There are not so many small websites in general . So how do you find out what you want most from multiple packets .

1. According to the requested path 、 Name, etc

For example, you need to find the key data package for login , You can find the corresponding keyword for login . for instance login、logincheck What? , There may be some low-level websites that can't be identified denglu In this way . This depends on personal experience , You can see everything if you climb too much .

2. According to the type

Take Baidu as an example , If you are looking for a login package （ I opened the home page , Just for example ）, Like these js、gif、css、png、 And I don't know plain Basically useless . After all, log in and you take care of those pictures 、gif What are you doing , If there is a picture verification code, just select the element to find it .

Conclusion

Seeing this can basically help Xiaobai further understand reptiles , It's easy to analyze some simple packages . Don't tell me that you won't write a program to simulate the contract after analyzing the package , See if I can beat you or not .
If you are crawling data, you have to xpath Equal resolution 、 If the package is complex and cannot be analyzed, you can use selenium. These on their own Baidu , I am too lazy to write on the Internet .
If you think it's useful, give it a compliment .