首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >是否可以在Python中对返回某些内容的函数进行多进程处理?

是否可以在Python中对返回某些内容的函数进行多进程处理?
EN

Stack Overflow用户
提问于 2012-05-29 19:12:59
回答 4查看 84.3K关注 0票数 28

在Python中,我见过许多调用多处理的例子,但目标只是打印一些东西。我有一个场景,目标返回2个变量,我稍后需要用到它们。例如:

代码语言:javascript
复制
def foo(some args):
   a = someObject
   b = someObject
   return a,b

p1=multiprocess(target=foo,args(some args))
p2=multiprocess(target=foo,args(some args))
p3=multiprocess(target=foo,args(some args))

这次又是什么?我可以执行.start和.join,但是如何检索各个结果呢?我需要捕获我执行的所有作业的返回值a,b,然后处理它。

EN

回答 4

Stack Overflow用户

回答已采纳

发布于 2012-05-29 19:15:28

是的,当然--你可以使用很多方法。最简单的方法之一是共享Queue。请看这里的示例:http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/

票数 22
EN

Stack Overflow用户

发布于 2012-05-29 19:19:42

我直接从文档中复制这个例子,因为我不能给你一个直接的链接。请注意,它会打印出done_queue的结果,但您可以使用它做任何您喜欢的事情。

代码语言:javascript
复制
#
# Simple example which uses a pool of workers to carry out some tasks.
#
# Notice that the results will probably not come out of the output
# queue in the same in the same order as the corresponding tasks were
# put on the input queue.  If it is important to get the results back
# in the original order then consider using `Pool.map()` or
# `Pool.imap()` (which will save on the amount of code needed anyway).
#
# Copyright (c) 2006-2008, R Oudkerk
# All rights reserved.
#

import time
import random

from multiprocessing import Process, Queue, current_process, freeze_support

#
# Function run by worker processes
#

def worker(input, output):
    for func, args in iter(input.get, 'STOP'):
        result = calculate(func, args)
        output.put(result)

#
# Function used to calculate result
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % \
        (current_process().name, func.__name__, args, result)

#
# Functions referenced by tasks
#

def mul(a, b):
    time.sleep(0.5*random.random())
    return a * b

def plus(a, b):
    time.sleep(0.5*random.random())
    return a + b

#
#
#

def test():
    NUMBER_OF_PROCESSES = 4
    TASKS1 = [(mul, (i, 7)) for i in range(20)]
    TASKS2 = [(plus, (i, 8)) for i in range(10)]

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    # Submit tasks
    for task in TASKS1:
        task_queue.put(task)

    # Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

    # Get and print results
    print 'Unordered results:'
    for i in range(len(TASKS1)):
        print '\t', done_queue.get()

    # Add more tasks using `put()`
    for task in TASKS2:
        task_queue.put(task)

    # Get and print some more results
    for i in range(len(TASKS2)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')


if __name__ == '__main__':
    freeze_support()
    test()

它最初来自multiprocessing module docs

票数 10
EN

Stack Overflow用户

发布于 2012-05-29 19:40:37

它不会在windows上工作,但这是我的多进程函数装饰器,它返回一个队列,您可以从队列中轮询和收集返回的数据

代码语言:javascript
复制
import os
from Queue import Queue
from multiprocessing import Process

def returning_wrapper(func, *args, **kwargs):
    queue = kwargs.get("multiprocess_returnable")
    del kwargs["multiprocess_returnable"]
    queue.put(func(*args, **kwargs))

class Multiprocess(object):
    """Cute decorator to run a function in multiple processes."""
    def __init__(self, func):
        self.func = func
        self.processes = []

    def __call__(self, *args, **kwargs):
        num_processes = kwargs.get("multiprocess_num_processes", 2) # default to two processes.
        return_obj = kwargs.get("multiprocess_returnable", Queue()) # default to stdlib Queue
        kwargs["multiprocess_returnable"] = return_obj
        for i in xrange(num_processes):
            pro = Process(target=returning_wrapper, args=tuple([self.func] + list(args)), kwargs=kwargs)
            self.processes.append(pro)
            pro.start()
        return return_obj


@Multiprocess
def info():
    print 'module name:', __name__
    print 'parent process:', os.getppid()
    print 'process id:', os.getpid()
    return 4 * 22

data = info()
print data.get(False)
票数 4
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/10797998

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档