使用Python和AsyncIO获取JSON
使用Python和AsyncIO获取JSON
提问于 2018-11-07 23:06:20
不久前，我开始学习异步。我遇到了一个问题。我的代码不会终止。我搞不懂。请救救我！
import signal
import sys
import asyncio
import aiohttp
import json
loop = asyncio.get_event_loop()
client = aiohttp.ClientSession(loop=loop)
async def get_json(client, url):
async with client.get(url) as response:
assert response.status == 200
return await response.read()
async def get_reddit_cont(subreddit, client):
data1 = await get_json(client, 'https://www.reddit.com/r/' + subreddit + '/top.json?sort=top&t=day&limit=50')
jn = json.loads(data1.decode('utf-8'))
print('DONE:', subreddit)
def signal_handler(signal, frame):
loop.stop()
client.close()
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
for key in {'python':1, 'programming':2, 'compsci':3}:
asyncio.ensure_future(get_reddit_cont(key, client))
loop.run_forever()结果：
DONE: compsci
DONE: programming
DONE: python
...我试图取得一些成就，但结果并不稳定。
future = []
for key in {'python':1, 'programming':2, 'compsci':3}:
future=asyncio.ensure_future(get_reddit_cont(key, client))
loop.run_until_complete(future)结果(1项任务而不是3项任务)：
DONE: compsci
[Finished in 1.5s] 我用这种方式解决了我的问题：
加上：
async with aiohttp.ClientSession () as a client:AT：
async def get_reddit_cont (subreddit, client): 和：
if __name__ == '__main__':
loop = asyncio.get_event_loop()
futures = [get_reddit_cont(subreddit,client) for subreddit in range(1,6)]
result = loop.run_until_complete(asyncio.gather(*futures))但是，当代码完成后，我会得到这样的信息：
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x034021F0>
[Finished in 1.0s]我不明白为什么会这样。
但是，当我尝试执行"for key“大约60次或更多次时，我会得到一个错误：
..。 aiohttp.client_exceptions.ClientOSError: WinError 10054远程主机强制终止现有连接
回答 3
Stack Overflow用户
回答已采纳
发布于 2018-11-07 23:28:03
答案就在你的代码里。这是loop.run_forever()的线索。因此，您需要调用loop.stop()。我将使用诸如if子句或while循环之类的条件。
if we_have_what_we_need:
signal_handler(signal, frame)或
while we_dont_have_what_we_need:
loop.forever()第一个将在满足条件时停止您的代码。后者将继续下去，直到满足条件为止。
更新
我们也可以使用；
(Python文档)
loop.run_until_complete(future)运行到未来(未来的一个实例)已经完成。 如果参数是coroutine对象，则隐式地将其作为asyncio.Task运行。 返回未来的结果或提出其异常。
loop.run_forever()运行事件循环，直到调用stop()为止。
Stack Overflow用户
发布于 2018-11-08 02:50:09
以下是一些建议的修改，并在评论中添加了上下文。
除非您真的有一个独特的用例，或者只是为了学习而进行实验，否则可能不应该有使用signal的理由-- asyncio具有顶级函数，可以让您决定何时关闭和终止事件循环。
import asyncio
import logging
import sys
import aiohttp
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG,
format='%(asctime)s:%(message)s')
URL = 'https://www.reddit.com/r/{subreddit}/top.json?sort=top&t=day&limit=50'
async def get_json(client: aiohttp.ClientSession, url: str) -> dict:
# If you're going to be making repeated requests, use this
# over .get(), which is just a wrapper around `.request()` and
# involves an unneeded lookup
async with client.request('GET', url) as response:
# Raise if the response code is >= 400.
# Some 200 codes may still be "ok".
# You can also pass raise_for_status within
# client.request().
response.raise_for_status()
# Let your code be fully async. The call to json.loads()
# is blocking and won't take full advantage.
#
# And it does largely the same thing you're doing now:
# https://github.com/aio-libs/aiohttp/blob/76268e31630bb8615999ec40984706745f7f82d1/aiohttp/client_reqrep.py#L985
j = await response.json()
logging.info('DONE: got %s, size %s', url, j.__sizeof__())
return j
async def get_reddit_cont(keys, **kwargs) -> list:
async with aiohttp.ClientSession(**kwargs) as session:
# Use a single session as a context manager.
# this enables connection pooling, which matters a lot when
# you're only talking to one site
tasks = []
for key in keys:
# create_task: Python 3.7+
task = asyncio.create_task(
get_json(session, URL.format(subreddit=key)))
tasks.append(task)
# The result of this will be a list of dictionaries
# It will only return when all of your subreddits
# have given you a response & been decoded
#
# To process greedily, use asyncio.as_completed()
return await asyncio.gather(*tasks, return_exceptions=True)
if __name__ == '__main__':
default = ('python', 'programming', 'compsci')
keys = sys.argv[1:] if len(sys.argv) > 1 else default
sys.exit(asyncio.run(get_reddit_cont(keys=keys)))输出：
$ python3 asyncreddit.py
2018-11-07 21:44:49,495:Using selector: KqueueSelector
2018-11-07 21:44:49,653:DONE: got https://www.reddit.com/r/compsci/top.json?sort=top&t=day&limit=50, size 216
2018-11-07 21:44:49,713:DONE: got https://www.reddit.com/r/python/top.json?sort=top&t=day&limit=50, size 216
2018-11-07 21:44:49,947:DONE: got https://www.reddit.com/r/programming/top.json?sort=top&t=day&limit=50, size 216编辑:从你的问题：
但是，当代码完成后，我得到消息：
Unclosed client session
这是因为您需要.close() client对象，就像文件对象一样。你可以用两种方法：
- 显式调用它：
client.close()。将其包装在try/finally块中更安全，以确保无论发生什么，它都已关闭 - 或者(更简单的方法)，使用客户机作为异步上下文管理器，如这个答案所示。这意味着，在
async with块结束后，会话将通过其.__aexit__()方法自动关闭。
connector是基础TCPConnector，它是会话的一个属性。它处理连接池，这是您的代码中最终打开的部分。
Stack Overflow用户
发布于 2018-11-09 03:15:19
我用这样的方法解决了这个问题：
import asyncio
import aiohttp
import json
async def get_json(client, url):
async with client.get(url) as response:
assert response.status == 200
return await response.read()
async def get_reddit_cont(subreddit):
async with aiohttp.ClientSession(loop=loop) as client:
data1 = await get_json(client, 'https://www.reddit.com/r/' + subreddit + '/top.json?sort=top&t=day&limit=50')
jn = json.loads(data1.decode('utf-8'))
print('DONE:', subreddit)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
futures = [get_reddit_cont(subreddit) for subreddit in {'python':1, 'programming':2, 'compsci':3}]
result = loop.run_until_complete(asyncio.gather(*futures))页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/53199248
复制相关文章
相似问题
腾讯云开发者
