Incremental crawler in distributed crawler

Incremental reptiles ： Overview of detecting website data updates , Then the updated data is crawled
The core ： duplicate removal
Record sheet ： Store the captured data identification redis Of set Make data update table .

The idea is to go when crawling redis Please confirm ,url Whether there is , as follows ：

li_list=response.xpath('./span[3]/ul/li')
for li in li_list
    detail-url="http://baidu.com"+li.xpath('/li/@href').extract_first()
    ex=self.conn.sadd('urls',detail-url)
    if ex==1:
       #ex On behalf of the successful return , There are data updates , No repetition 
       yield scrapy Request(detail-url)
    else:
        print(" No updated data ")