Reptile initial and project

1.robots.txt agreement ：

        Specify which data in the website can be crawled , Which cannot be crawled （ A gentleman's agreement ）
Such as the input ：https://www.sohu.com/robots.txt To view the agreement

2.http agreement ：

Concept ： A form of server client interaction . Be similar to , The slang between yangzirong and Zuoshan Diao , Heavenly King Gedi tiger , pagoda will stop river monster （ Encrypted calls ）

Common request header information ：

• 1.User-Agent： The identity represented by the request carrier （ The user agent ）
  

        Chrome Click... In the browser F12、network（ The Internet ）,Ctrl+r,header, Then you can get the information on the graph .

• 2.Connection： After the request , Disconnect or stay connected

Common response header information ：

• Content-Type： The data type of the server response back to the client

3.https agreement ：（s：security）

        Security Hypertext transfer protocol , Including data encryption

encryption ：

• 1. Symmetric key encryption
  

disadvantages ： The secret key may be blocked by a third-party organization , Cause the disclosure of encrypted information .

• 2. Asymmetric key encryption
  shortcoming ： Low efficiency , Communication speed slows down ; The public key may be replaced by a third party

• 3. Certificate key encryption
  

It can ensure that the public key obtained by the client must be generated by the server

4.requests modular

       Python Zhongsheng's network request based module , Very powerful , Simple and convenient , Very efficient ,requests Modules are half of the reptiles

effect ： Used to simulate the browser to send a request .

Usage mode ：

• 1. Appoint url（ website ）
• 2. Initiate request
• 3. Get response data
• 4. Persistent storage

Environmental installation ：

pip install -i http://pypi.douban.com/simple --trusted-host pypi.douban.com requests

import requests

#  Appoint url
url = 'https://www.baidu.com/'

#  Initiate request 
response = requests.get(url=url) # get Method will return a response object 

#  Get response data 
pageText = response.text  # text What is returned is the response data in the form of string 

#  Persistent storage 
with open('C:\\Users\\Administrator\\Desktop\\baiduPachong.html','w',encoding='utf-8') as fp:
    fp.write(pageText)
print("end")

4. Netease cloud music crawler

import requests
import re #  Regular expressions , Self contained , No installation required 
import os #  File operation module 

filename = 'music1\\'

if not os.path.exists(filename):
    os.mkdir(filename)

#  modify id Access to different music lists 
url = 'https://music.163.com/#/discover/toplist?id=3778678'
# header Request header   It's Camouflage Python Code   hold Python The code disguises itself as a browser to access the server 
#  After the server receives the request , Will return the response data to us （response）
headers = {
    
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'
}
response = requests.get(url = url,headers=headers)
#print(response.text)

#  A content extracted from regular expression   The return is a list   Each element is a tuple 
html_data = re.findall('<li><a href="/song\?id=(\d+)">(.*?)</a>',response.text)
for num_id, title in html_data:
    # https://music.163.com/song/media/outer/url?id=1830419924.mp3
    music_url = 'https://music.163.com/song/media/outer/url?id={num_id}.mp3'
    #  Send a request for the music playing address   Get binary data content 
    music_content = requests.get(url=music_url, headers=headers).content
    with open(filename + title + '.mp3',mode='wb') as f:
        f.write(music_content)
        
    print(html_data)