Crawler grabs the idea of reading on wechat

I have an idea recently , Capture the notes and ideas of reading on wechat in recent years .

Tool preparation ：

1、 Cloud gathering reptile  （ Data collection ）

2、charles （ Grab the bag + agent ）

Grab the bag

 

First you have to let your iPhone Use MAC Internet proxy .

start-up  charles, He has a proxy function .

And then use iPhone Just connect this agent to the Internet . This step is omitted , There are many online tutorials .

The interface for capturing packets is shown in the figure ：

 

This process will not be described in detail . Is not the point .

First you have to get the requested url、cookie, There's another one called  skey Of thing .

Fetching the data

Request address ：

https://i.weread.qq.com/review/list?count=20&listMode=1&listType=6&maxIdx=1579514807&mine=1&rangeType=1&userVid=2107157

 

The data is shown below ：

yes json data , The structure is very simple .

How to turn the page ？

url Inside  maxIdx  Parameters , Is what you get every time json The last record of data createTime , Put this in to construct the next page request address .

The whole capture flow chart is as follows ：

 

 

 

Total captured 666 An idea , The generated word cloud is as follows ：

 

Here are my notes ：

 

This is the original book excerpt ：