内容来源于 Stack Overflow,并遵循CC BY-SA 3.0许可协议进行翻译与使用
我希望同时运行多个Program.py实例,同时限制同时运行的实例数量(例如,我的系统上可用的CPU核数量)。例如,如果我有10个内核,并且总共必须运行1000次Program.py,那么在任何给定的时间内,只会创建和运行10个实例。
我尝试过使用MultiProcessing模块、多线程和队列,但我觉得没有什么可以让自己得到一个简单的实现。我最大的问题是找到一种方法来限制同时运行的进程的数量。
Basic Pool:
from multiprocessing import Pool def do_work(val): # could instantiate some other library class, # call out to the file system, # or do something simple right here. return "FOO: %s" % val pool = Pool(4) work = get_work_args() results = pool.map(do_work, work)
使用流程管理器和生产者
from multiprocessing import Process, Manager import time import itertools def do_work(in_queue, out_list): while True: item = in_queue.get() # exit signal if item == None: return # fake work time.sleep(.5) result = item out_list.append(result) if __name__ == "__main__": num_workers = 4 manager = Manager() results = manager.list() work = manager.Queue(num_workers) # start for workers pool = [] for i in xrange(num_workers): p = Process(target=do_work, args=(work, results)) p.start() pool.append(p) # produce data # this could also be started in a producer process # instead of blocking iters = itertools.chain(get_work_args(), (None,)*num_workers) for item in iters: work.put(item) for p in pool: p.join() print results