异步函数内部的子进程调用不会并行发生
异步函数内部的子进程调用不会并行发生
提问于 2022-07-26 16:16:49
我需要在我的应用程序中实现两个级别的“异步性”。
假设我输入的数据是：
input = {
"set1": ["subset11", "subset12"],
"set2": ["subset21", "subset22"],
...
}我需要用这样的方式来处理它们：
parallel
- Within中的
- 处理所有集合，并行处理子集，然后对
进行总结。
所以我要做的是：
# top level function
async def process_all_sets(input):
tasks = [process_set(payload) for key, payload in input.items()]
results = await asyncio.gather(*tasks)
# process a single set
async def process_set(payload):
tasks = [process_subset(item) for item in payload]
results = await asyncio.gather(*tasks)
# here, loop over results and do some summarization
# and return it
return summary
# process a single subset
async def process_subset(subset):
# need to run a subprocess here, it make take several minutes
subprocess.run("some_command_based_on_subset")
# do whatever needs to be done after subprocess completes
# and return result
return result由于asyncio应该在一个集合内并行地运行多个process_subset调用，所以我希望看到多个subprocess.run调用同时被调用。但我一次只看到一个电话。
那么并行化在哪里呢？
回答 1
Stack Overflow用户
发布于 2022-07-26 16:21:07
哈。我从subprocess.run切换到了asyncio.create_subprocess_shell，现在它可以正常工作了！
# process a single subset
async def process_subset(subset):
# need to run a subprocess here, it make take several minutes
process = await asyncio.create_subprocess_shell("some_command_based_on_subset")
await process.wait()
# do whatever needs to be done after subprocess completes
# and return result
return result页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/73126744
复制相关文章
相似问题
腾讯云开发者
