我正在写蜘蛛来抓取网页.我知道asyncio可能是我最好的选择.所以我使用协同程序异步处理工作.现在我抓住了关于如何通过键盘中断退出程序的问题.所有工作完成后,该程序可能会关闭.源代码可以在python 3.5中运行,并在下面附上.

import asyncio
import aiohttp
from contextlib import suppress

class Spider(object):
    def __init__(self):
        self.max_tasks = 2
        self.task_queue = asyncio.Queue(self.max_tasks)
        self.loop = asyncio.get_event_loop()
        self.counter = 1

    def close(self):
        for w in self.workers:
            w.cancel()

    async def fetch(self,url):
        try:
            async with aiohttp.ClientSession(loop = self.loop) as self.session:
                with aiohttp.Timeout(30,loop = self.session.loop):
                    async with self.session.get(url) as resp:
                        print('get response from url: %s' % url)
        except:
            pass
        finally:
            pass

    async def work(self):
        while True:
            url = await self.task_queue.get()
            await self.fetch(url)
            self.task_queue.task_done()

    def assign_work(self):
        print('[*]assigning work...')
        url = 'https://www.python.org/'
        if self.counter > 10:
            return 'done'
        for _ in range(self.max_tasks):
            self.counter += 1
            self.task_queue.put_nowait(url)

    async def crawl(self):
        self.workers = [self.loop.create_task(self.work()) for _ in range(self.max_tasks)]
        while True:
            if self.assign_work() == 'done':
                break
            await self.task_queue.join()
        self.close()

def main():
    loop = asyncio.get_event_loop()
    spider = Spider()
    try:
        loop.run_until_complete(spider.crawl())
    except KeyboardInterrupt:
        print ('Interrupt from keyboard')
        spider.close()
        pending  = asyncio.Task.all_tasks()
        for w in pending:
            w.cancel()
            with suppress(asyncio.CancelledError):
                loop.run_until_complete(w)
    finally:
        loop.stop()
        loop.run_forever()
        loop.close()

if __name__ == '__main__':
    main()

但如果我在运行时按“Ctrl C”,可能会出现一些奇怪的错误.我的意思是有时程序可以通过’Ctrl C’优雅地关闭.没有错误消息.但是,在某些情况下,程序将在按下“Ctrl C”后仍然运行,并且在完成所有工作之前不会停止.如果我在那一刻按下“Ctrl C”,“任务已被销毁,但它正在等待！”会在那里.

我已经阅读了一些关于asyncio的主题,并在main()中添加了一些代码来优雅地关闭协同程序.但它不起作用.别人有类似的问题吗？