我是Scrapy的新手,我正在尝试从已删除项目中的链接中删除新页面.具体来说,我想从谷歌搜索结果中删除DropBox上的一些文件共享链接,并将这些链接存储在JSON文件中.获取这些链接后,我想为每个链接打开一个新页面,以验证链接是否有效.如果它有效,我也想将文件名存储到JSON文件中.
我使用带有’链接’,’文件名’,’状态’,’err_msg’属性的DropBoxItem来存储每个被删除的项目,我尝试在解析函数中为每个被删除的链接发起异步请求.但似乎永远不会调用parse_file_page函数.有谁知道如何实现这样的两步爬行？
class DropBoxSpider(Spider):
name = "dropBox"
allowed_domains = ["google.com"]
start_urls = [
"https://www.google.com/#filter=0&q=site:www.dropBox.com/s/&start=0"
]
def parse(self,response):
sel = Selector(response)
sites = sel.xpath("//h3[@class='r']")
items = []
for site in sites:
item = DropBoxItem()
link = site.xpath('a/@href').extract()
item['link'] = link
link = ''.join(link)
#I want to parse a new page with url=link here
new_request = Request(link,callback=self.parse_file_page)
new_request.Meta['item'] = item
items.append(item)
return items
def parse_file_page(self,response):
#item passed from request
item = response.Meta['item']
#selector
sel = Selector(response)
content_area = sel.xpath("//div[@id='shmodel-content-area']")
filename_area = content_area.xpath("div[@class='filename shmodel-filename']")
if filename_area:
filename = filename_area.xpath("span[@id]/text()").extract()
if filename:
item['filename'] = filename
item['status'] = "normal"
else:
err_area = content_area.xpath("div[@class='err']")
if err_area:
err_msg = err_area.xpath("h3/text()").extract()
item['err_msg'] = err_msg
item['status'] = "error"
return item
感谢@ScrapyNovice的回答.我修改了代码.现在看起来像
def parse(self,response):
sel = Selector(response)
sites = sel.xpath("//h3[@class='r']")
#items = []
for site in sites:
item = DropBoxItem()
link = site.xpath('a/@href').extract()
item['link'] = link
link = ''.join(link)
print 'link!!!!!!=',link
new_request = Request(link,callback=self.parse_file_page)
new_request.Meta['item'] = item
yield new_request
#items.append(item)
yield item
return
#return item #Note,when I simply return item here,got an error msg "SyntaxError: 'return' with argument inside generator"
def parse_file_page(self,response):
#item passed from request
print 'parse_file_page!!!'
item = response.Meta['item']
#selector
sel = Selector(response)
content_area = sel.xpath("//div[@id='shmodel-content-area']")
filename_area = content_area.xpath("div[@class='filename shmodel-filename']")
if filename_area:
filename = filename_area.xpath("span[@id]/text()").extract()
if filename:
item['filename'] = filename
item['status'] = "normal"
item['err_msg'] = "none"
print 'filename=',filename
else:
err_area = content_area.xpath("div[@class='err']")
if err_area:
err_msg = err_area.xpath("h3/text()").extract()
item['filename'] = "null"
item['err_msg'] = err_msg
item['status'] = "error"
print 'err_msg',err_msg
else:
item['filename'] = "null"
item['err_msg'] = "unknown_err"
item['status'] = "error"
print 'unknown err'
return item
控制流程实际上变得非常奇怪.当我使用“scrapy crawl dropBox -o items_dropBox.json -t json”来抓取本地文件(谷歌搜索结果的下载页面)时,我可以看到输出像
2014-05-31 08:40:35-0400 [scrapy] INFO: Scrapy 0.22.2 started (bot: tutorial)
2014-05-31 08:40:35-0400 [scrapy] INFO: Optional features available: ssl,http11
2014-05-31 08:40:35-0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders','Feed_FORMAT': 'json','SPIDER_MODULES': ['tutorial.spiders'],'Feed_URI': 'items_dropBox.json','BOT_NAME': 'tutorial'}
2014-05-31 08:40:35-0400 [scrapy] INFO: Enabled extensions: FeedExporter,LogStats,TelnetConsole,CloseSpider,WebService,CoreStats,SpiderState
2014-05-31 08:40:35-0400 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware,DownloadTimeoutMiddleware,UserAgentMiddleware,RetryMiddleware,DefaultHeadersMiddleware,MetaRefreshMiddleware,HttpCompressionMiddleware,RedirectMiddleware,CookiesMiddleware,ChunkedTransferMiddleware,DownloaderStats
2014-05-31 08:40:35-0400 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware,OffsiteMiddleware,RefererMiddleware,UrlLengthMiddleware,DepthMiddleware
2014-05-31 08:40:35-0400 [scrapy] INFO: Enabled item pipelines:
2014-05-31 08:40:35-0400 [dropBox] INFO: Spider opened
2014-05-31 08:40:35-0400 [dropBox] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)
2014-05-31 08:40:35-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-05-31 08:40:35-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-05-31 08:40:35-0400 [dropBox] DEBUG: Crawled (200) Box_s/dropBox_s_1-Google.html> (referer: None)
link!!!!!!= http://www.google.com/intl/en/webmasters/#utm_source=en-wmxmsg&utm_medium=wmxmsg&utm_campaign=bm&authuser=0
link!!!!!!= https://www.dropBox.com/s/
2014-05-31 08:40:35-0400 [dropBox] DEBUG: Filtered offsite request to 'www.dropBox.com': Box.com/s/>
link!!!!!!= https://www.dropBox.com/s/awg9oeyychug66w
link!!!!!!= http://www.dropBox.com/s/kfmoyq9y4vrz8fm
link!!!!!!= https://www.dropBox.com/s/pvsp4uz6gejjhel
.... many links here
link!!!!!!= https://www.dropBox.com/s/gavgg48733m3918/MailCheck.xlsx
link!!!!!!= http://www.dropBox.com/s/9x8924gtb52ksn6/Phonesky.apk
2014-05-31 08:40:35-0400 [dropBox] DEBUG: Scraped from <200 file:///home/xin/Downloads/dropBox_s/dropBox_s_1-Google.html>
{'link': [u'http://www.dropBox.com/s/9x8924gtb52ksn6/Phonesky.apk']}
2014-05-31 08:40:35-0400 [dropBox] DEBUG: Crawled (200) Box_s/dropBox_s_1-Google.html)
parse_file_page!!!
unknown err
2014-05-31 08:40:35-0400 [dropBox] DEBUG: Scraped from <200 http://www.google.com/intl/en/webmasters/>
{'err_msg': 'unknown_err','filename': 'null','link': [u'http://www.google.com/intl/en/webmasters/#utm_source=en-wmxmsg&utm_medium=wmxmsg&utm_campaign=bm&authuser=0'],'status': 'error'}
2014-05-31 08:40:35-0400 [dropBox] INFO: Closing spider (finished)
2014-05-31 08:40:35-0400 [dropBox] INFO: Stored json Feed (2 items) in: items_dropBox.json
2014-05-31 08:40:35-0400 [dropBox] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 558,'downloader/request_count': 2,'downloader/request_method_count/GET': 2,'downloader/response_bytes': 449979,'downloader/response_count': 2,'downloader/response_status_count/200': 2,'finish_reason': 'finished','finish_time': datetime.datetime(2014,5,31,12,40,35,348058),'item_scraped_count': 2,'log_count/DEBUG': 7,'log_count/INFO': 8,'request_depth_max': 1,'response_received_count': 2,'scheduler/dequeued': 2,'scheduler/dequeued/memory': 2,'scheduler/enqueued': 2,'scheduler/enqueued/memory': 2,'start_time': datetime.datetime(2014,249309)}
2014-05-31 08:40:35-0400 [dropBox] INFO: Spider closed (finished)
现在json文件只有：
[{"link": ["http://www.dropBox.com/s/9x8924gtb52ksn6/Phonesky.apk"]},{"status": "error","err_msg": "unknown_err","link": ["http://www.google.com/intl/en/webmasters/#utm_source=en-wmxmsg&utm_medium=wmxmsg&utm_campaign=bm&authuser=0"],"filename": "null"}]
最佳答案
您正在创建一个请求并很好地设置回调,但您从不对它做任何事情.
原文链接：https://www.f2er.com/python/439784.html for site in sites:
item = DropBoxItem()
link = site.xpath('a/@href').extract()
item['link'] = link
link = ''.join(link)
#I want to parse a new page with url=link here
new_request = Request(link,callback=self.parse_file_page)
new_request.Meta['item'] = item
yield new_request
# Don't do this here because you're adding your Item twice.
#items.append(item)
在更多的设计级别,您将所有已删除的项目存储在parse()末尾的项目中,但是管道通常希望接收单个项目,而不是它们的数组.摆脱items数组,您将能够使用内置于Scrapy的JSON Feed Export以JSON格式存储结果.
更新：
尝试返回项目时收到错误消息的原因是因为在函数中使用yield会将其转换为生成器.这允许您重复调用该函数.每次达到收益时,它都会返回您正在收益的值,但会记住它的状态以及它正在执行的行.下次调用生成器时,它会从上次停止的位置继续执行.如果它没有产生的东西,它会引发一个StopIteration异常.在Python 2中,不允许在同一函数中混合yield和return.
你不想从parse()中产生任何项目,因为它们在那时仍然缺少文件名,状态等.
你在解析()中的请求是在dropBox.com上,对吗？请求没有通过,因为dropBox不在spider的allowed_domains中. (因此日志消息：DEBUG：过滤现场请求’www.dropBox.com’：< GET https://www.dropBox.com/s/\u0026gt;) 实际有效且未过滤的一个请求会导致http://www.google.com/intl/zh-CN/webmasters/#utm_source=en-wmxmsg\u0026amp;utm_medium=wmxmsg\u0026amp;utm_campaign=bm\u0026amp;authuser=0,这是谷歌的一个页面,而不是DropBox的.您可能希望在parse()方法中使用urlparse检查链接的域之前检查它的请求. 至于你的结果：第一个JSON对象
{"link": ["http://www.dropBox.com/s/9x8924gtb52ksn6/Phonesky.apk"]}
是你在parse()方法中调用yield项的地方.只有一个因为你的产量不在任何类型的循环中,所以当生成器恢复执行时,它会运行下一行：return,它退出生成器.您会注意到此项缺少您在parse_file_page()方法中填写的所有字段.这就是为什么你不想在你的parse()方法中产生任何项目.
你的第二个JSON对象
{
"status": "error","filename": "null"
}
是试图解析谷歌的一个页面的结果,就好像它是你期望的DropBox页面一样.您在parse()方法中产生了多个请求,除了其中一个之外的所有请求都指向dropBox.com.所有DropBox链接都被删除,因为它们不在您的allowed_domains中,因此您获得的唯一响应是页面上的另一个链接,该链接与您的xpath选择器匹配并且来自您的allowed_sites中的一个站点. (这是谷歌网站管理员链接)这就是为什么你只看到parse_file_page！一次在你的输出中.
我建议学习更多关于生成器的知识,因为它们是使用Scrapy的基本部分. second Google result for “python generator tutorial” looks like a very good place to start.