按池强制退出python多线程
按池强制退出python多线程
提问于 2021-11-05 21:44:48
我正在使用多线程来运行消耗内存的进程。为了避免内存交换，我想在内存超过一定数量时终止所有多线程进程。
Python是3.9，我使用的是Jupyter notebook。
以下示例代码用于在内存使用量超过50 is时终止进程。
from multiprocessing import Pool, cpu_count
def my_func():
a = MemoryConsumingObject()
memory_usage_giga = os.popen('free -g').readlines()[1].split()[1:][1]
if memory_usage_giga > 50:
# I want quit multi-threading process! How do I quit?
pass
return a
my_list = list(np.arange(1,1000))
with Pool(cpu_count()) as p:
result = p.imap(my_func, my_list)回答 1
Stack Overflow用户
回答已采纳
发布于 2021-11-06 07:53:59
您可以使用Event在工作进程之间进行通信。您不能轻易地终止池(在这种情况下，工作进程不能直接与主进程通信)，但是当满足某些条件时，您可以阻止工作进程进行内存分配。在下面的脚本中，我通过生成一个随机数来模拟何时停止工作。一旦满足了停止的条件，工作人员仍然会被调用来处理要完成的工作，但是如果满足停止条件，他们就会停止而不实际执行任何操作。
from os import getpid
from multiprocessing import Pool, cpu_count, Event
from random import random
event = Event()
def my_func(arg):
pid = getpid()
if event.is_set():
print('%s received %s, but aborting as stop condition was seen' % (pid, arg))
return None
r = random()
if r < 0.3:
print('%s received %s, %.2f stop condition met, aborting and letting everyone else know' % (pid, arg, r))
event.set()
return
print("%s received: %s, %.2f stop condition not met, do work" % (getpid(), arg, r))
# do the actual work here
return r
my_list = list(range(10))
def main():
with Pool(cpu_count()) as p:
result = p.map(my_func, my_list)
p.close()
p.join()
print(list(result))
if __name__ == '__main__':
main()一个示例run打印如下：
18786 received: 0, 0.67 stop condition not met, do work
18786 received: 1, 0.38 stop condition not met, do work
18786 received: 2, 0.51 stop condition not met, do work
18786 received: 3, 0.34 stop condition not met, do work
18786 received: 4, 0.72 stop condition not met, do work
18786 received: 5, 0.82 stop condition not met, do work
18786 received 6, 0.00 stop condition met, aborting and letting everyone else know
18786 received 7, but aborting as stop condition was seen
18786 received 8, but aborting as stop condition was seen
18786 received 9, but aborting as stop condition was seen
[0.6733142995176965, 0.3830673860039788, 0.5053762644489409, 0.3437097578584267, 0.7211013474170365, 0.816546904830295, None, None, None, None]请注意，这些进程可能并不总是以您期望的顺序运行-毕竟这是并发处理。在我的例子中，CPU计数只有2，因此一个工作进程得到了所有的数据。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/69859572
复制相关文章
相似问题
腾讯云开发者
