Requests crawler multi page crawling to KFC restaurant location

Crawling of paging data - KFC restaurant location information

1 analysis

The address displayed after entering the address is the same as the original address

Explain that pressing the query button initiates Ajax request

• The location information refreshed from the current page must be through ajax Request the requested data

. Locate the... Based on the packet capturing tool ajax Requested packet , From this packet ：

• Requested url
• Request mode
• The parameter that the request carries
• See response data

At first, you choose to grab bags ALL, But the analysis shows that what is sent here is Ajax request , So this time I choose Fetch/XHR, This is for checking Ajax Requested

open F12, choice Fetch/XHR, Click query to view the results

The discovery request method is post The way

Return value or json Format

2 Crawl to a page of data

import requests

headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=keyword'

data = {
    
    'cname': '',
    'pid': '',
    'keyword': ' Beijing ',
    'pageIndex': '1',
    'pageSize': '10',
}
# data Parameter is post Method to deal with parameter dynamic parameters 
response = requests.post(url=url, headers=headers, data=data)
page_text = response.json()

for dic in page_text['Table1']:
    title = dic['storeName']
    addr = dic['addressDetail']
    print(title, addr)

3 Crawling multiple pages of data

When you click on the second page , To discover the requested data pageIndex Change into 2, When you click on the third page, it changes to 3.

So you can crawl all pages by writing a cycle

What only needs to be changed in each cycle is pageIndex The value of the parameter , Because the data is required to be of string type , So avoid making mistakes and force it to convert

import requests

headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=keyword'

for page in range(1, 9):
    data = {
    
        'cname': '',
        'pid': '',
        'keyword': ' Beijing ',
        'pageIndex': str(page),
        'pageSize': '10',
    }
    # data Parameter is post Method to deal with parameter dynamic parameters 
    response = requests.post(url=url, headers=headers, data=data)
    page_text = response.json()

    for dic in page_text['Table1']:
        title = dic['storeName']
        addr = dic['addressDetail']
        print(' The first ', page, ' page ：', title, addr)

Follow the column for more details