如果需要并发 http 请求怎么办呢?requests库是同步阻塞的,必须等到结果才会发第二个请求,这里需使用http请求异步库 aiohttp。
aiohttp 用于 asyncio 和 Python 的异步 HTTP 客户端/服务器。 使用pip安装对应的包。当前使用版本v3.8.1
pip install aiohttp
如果使用requests 库,发10个请求访问我的博客,那么这10个请求是串行的。
import requests
import time
url = "https://www.cnblogs.com/yoyoketang/"
start_time = time.time()
for i in range(10):
r = requests.get(url)
print(r)
print('总耗时:', time.time()-start_time)
运行结果,总共耗时1秒左右
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
总耗时:1.0870192050933838
使用asyncio + aiohttp 并发请求
import asyncio
from aiohttp import ClientSession
import time
async def bai_du(url):
print(f'启动时间: {time.time()}')
async with ClientSession() as session:
async with session.get(url) as response:
res = await response.text()
return res
async def main():
url = "https://www.cnblogs.com/yoyoketang/"
task_list = []
for i in range(10):
task = asyncio.create_task(bai_du(url))
task_list.append(task)
done, pending = await asyncio.wait(task_list, timeout=None)
# 得到执行结果
for done_task in done:
print(f"{time.time()} 得到执行结果 {done_task.result()}")
# asyncio.run(main())
start_time = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print("总耗时: ", time.time()-start_time)
运行结果
启动时间: 1646028822.3952098
启动时间: 1646028822.4161537
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
....
总耗时: 0.17400002479553223
从运行结果可以看到,启动时间非常接近,也就是并发的请求,总过耗时 0.17 秒。
需注意的是这里使用 asyncio.run(main())
会报错RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000238675A8F70>
Traceback (most recent call last):
File "D:\python3.8\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "D:\python3.8\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "D:\python3.8\lib\asyncio\base_events.py", line 719, in call_soon
self._check_closed()
File "D:\python3.8\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
解决办法,把执行方式 asyncio.run(main())
改成
# asyncio.run(main())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
注意原因是asyncio.run()会自动关闭循环,并且调用ProactorBasePipeTransport._del报错, 而asyncio.run_until_complete()不会. 详情参考https://zhuanlan.zhihu.com/p/365815189