Python 爬虫框架Scrapy ITEM PIPELINE

Typical uses of item pipelines are: cleansing HTML data validating scraped data (checking that the items contain certain fields) checking for duplicates (and dropping them) storing the scraped item in a database ITEM PIPELINE作用: 清理HTML数据 验证爬取的数据(检查item包含某些字段) 去重(并丢弃)【预防数据去重,真正去重是在url,即请求阶段做】 将爬取结果保存