blocks|key|2795094|text|如果您使用的是Linux+(或任何兼容POSIX的系统)，则可以将此数组定义为全局变量。当multiprocessing启动一个新的子进程时，它正在使用Linux上的fork()。新产生的子进程会自动与其父进程共享内存，只要它不更改内存(copy-on-write机制)。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2795095|既然您说“我不需要任何类型的锁，因为数组(实际上是一个矩阵)将是只读的”，利用这种行为将是一种非常简单但非常有效的方法:所有子进程在读取这个大型的numpy数组时将访问物理内存中的相同数据。|2795096|不要将您的数组交给Process()构造函数，这将指示multiprocessing将数据pickle到子对象，在您的情况下，这将是非常低效或不可能的。在Linux上，在fork()之后，子元素是父元素的完全副本，使用相同的物理内存，所以您需要做的就是确保可以从您移交给Process()的target函数中访问“包含”矩阵的Python变量。这通常可以通过一个“全局”变量来实现。|2795097|示例代码：|2795098|from+multiprocessing+import+Process
from+numpy+import+random


global_array+=+random.random(10**4)


def+child():
++++print+sum(global_array)


def+main():
++++processes+=+[Process(target=child)+for+_+in+xrange(10)]
++++for+p+in+processes:
++++++++p.start()
++++for+p+in+processes:
++++++++p.join()


if+__name__+==+"__main__":
++++main()|code-block|syntax|javascript|2795099|在不支持fork()的Windows上，multiprocessing使用win32应用编程接口调用CreateProcess。它从任何给定的可执行文件创建一个全新的进程。这就是为什么在Windows上，如果需要在父进程运行时创建的数据，则需要将数据保留到子进程。|2795100|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Copy-on-write^0|19|F|2B|6|3B|D|0|0|0|9|9|R|F|19|6|2D|6|3R|9|41|6|0|0|0|4|6|K|F|1E|D|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]|$9|13|A|14|B|C]]|D|@$9|15|A|16|1|17]]|E|$]]|$1|F|3|G|5|6|7|18|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|19|8|@$9|1A|A|1B|B|C]|$9|1C|A|1D|B|C]|$9|1E|A|1F|B|C]|$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]|$9|1K|A|1L|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|1M|8|@]|D|@]|E|$]]|$1|L|3|M|5|N|7|1N|8|@]|D|@]|E|$O|P]]|$1|Q|3|R|5|6|7|1O|8|@$9|1P|A|1Q|B|C]|$9|1R|A|1S|B|C]|$9|1T|A|1U|B|C]]|D|@]|E|$]]|$1|S|3|-4|5|6|7|1V|8|@]|D|@]|E|$]]]|T|$U|$5|V|W|X|E|$Y|Z]]]]

If you are on Linux (or any POSIX-compliant system), you can define this array as a global variable. <code>multiprocessing</code> is using <code>fork()</code> on Linux when it starts a new child process. A newly spawned child process automatically shares the memory with its parent as long as it does not change it (<a href="http://en.wikipedia.org/wiki/Copy-on-write" rel="noreferrer">copy-on-write</a> mechanism).

Since you are saying "I don't need any kind of locks, since the array (actually a matrix) will be read-only" taking advantage of this behavior would be a very simple and yet extremely efficient approach: all child processes will access the same data in physical memory when reading this large numpy array.

Don't hand your array to the <code>Process()</code> constructor, this will instruct <code>multiprocessing</code> to <code>pickle</code> the data to the child, which would be extremely inefficient or impossible in your case. On Linux, right after <code>fork()</code> the child is an exact copy of the parent using the same physical memory, so all you need to do is making sure that the Python variable 'containing' the matrix is accessible from within the <code>target</code> function that you hand over to <code>Process()</code>. This you can typically achieve with a 'global' variable.

Example code:

<pre><code>from multiprocessing import Process
from numpy import random


global_array = random.random(10**4)


def child():
 print sum(global_array)


def main():
 processes = [Process(target=child) for _ in xrange(10)]
 for p in processes:
 p.start()
 for p in processes:
 p.join()


if __name__ == "__main__":
 main()
</code></pre>

On Windows -- which does not support <code>fork()</code> -- <code>multiprocessing</code> is using the win32 API call <code>CreateProcess</code>. It creates an entirely new process from any given executable. That's why on Windows one is required to pickle data to the child if one needs data that has been created during runtime of the parent.

blocks|key|3007755|text|如果你的数组那么大，你可以使用numpy.memmap。例如，如果您有一个存储在磁盘中的数组，比如'test.array'，那么即使在“写入”模式下，也可以使用并发进程来访问其中的数据，但是您的情况会更简单，因为您只需要“读取”模式。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3007756|创建数组：|3007757|a+=+np.memmap('test.array',+dtype='float32',+mode='w%2B',+shape=(100000,1000))|code-block|syntax|javascript|3007758|然后，您可以使用与普通数组相同的方式填充此数组。例如：|3007759|a[:10,:100]=1.
a[10:,100:]=2.|3007760|删除变量a时，数据将存储到磁盘中。|3007761|稍后，您可以使用多个进程来访问test.array中的数据|3007762|#+read-only+mode
b+=+np.memmap('test.array',+dtype='float32',+mode='r',+shape=(100000,1000))

#+read+and+writing+mode
c+=+np.memmap('test.array',+dtype='float32',+mode='r%2B',+shape=(100000,1000))|3007763|相关答案：|3007764|3007765|Working+with+big+data+in+python+and+numpy,+not+enough+ram,+how+to+save+partial+results+on+disc?|unordered-list-item|3007766|Is+it+possible+to+map+a+discontiuous+data+on+disk+to+an+array+with+python?|3007767|3007768|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/a/16633274/832621|1|https://stackoverflow.com/a/16597695/832621^0|F|C|1D|C|0|0|0|0|0|4|1|0|F|A|0|0|0|0|0|2N|0|0|0|22|1|0|0^^$0|@$1|2|3|4|5|6|7|1F|8|@$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1K|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|1L|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1M|8|@]|D|@]|E|$]]|$1|O|3|P|5|J|7|1N|8|@]|D|@]|E|$K|L]]|$1|Q|3|R|5|6|7|1O|8|@$9|1P|A|1Q|B|C]]|D|@]|E|$]]|$1|S|3|T|5|6|7|1R|8|@$9|1S|A|1T|B|C]]|D|@]|E|$]]|$1|U|3|V|5|J|7|1U|8|@]|D|@]|E|$K|L]]|$1|W|3|X|5|6|7|1V|8|@]|D|@]|E|$]]|$1|Y|3|-4|5|6|7|1W|8|@]|D|@]|E|$]]|$1|Z|3|10|5|11|7|1X|8|@]|D|@$9|1Y|A|1Z|1|20]]|E|$]]|$1|12|3|13|5|11|7|21|8|@]|D|@$9|22|A|23|1|24]]|E|$]]|$1|14|3|-4|5|6|7|25|8|@]|D|@]|E|$]]|$1|15|3|-4|5|6|7|26|8|@]|D|@]|E|$]]]|16|$17|$5|18|19|1A|E|$1B|1C]]|1D|$5|18|19|1A|E|$1B|1E]]]]

If your array is that big you can use <code>numpy.memmap</code>. For example, if you have an array stored in disk, say <code>'test.array'</code>, you can use simultaneous processes to access the data in it even in "writing" mode, but your case is simpler since you only need "reading" mode.

Creating the array:

<pre><code>a = np.memmap('test.array', dtype='float32', mode='w+', shape=(100000,1000))
</code></pre>

You can then fill this array in the same way you do with an ordinary array. For example:

<pre><code>a[:10,:100]=1.
a[10:,100:]=2.
</code></pre>

The data is stored into disk when you delete the variable <code>a</code>.

Later on you can use multiple processes that will access the data in <code>test.array</code>:

<pre><code># read-only mode
b = np.memmap('test.array', dtype='float32', mode='r', shape=(100000,1000))

# read and writing mode
c = np.memmap('test.array', dtype='float32', mode='r+', shape=(100000,1000))
</code></pre>

Related answers:

<ul>
<li><a href="https://stackoverflow.com/a/16633274/832621">Working with big data in python and numpy, not enough ram, how to save partial results on disc?</a></li>
<li><a href="https://stackoverflow.com/a/16597695/832621">Is it possible to map a discontiuous data on disk to an array with python?</a></li>
</ul>

blocks|key|3007861|text|您可能会对我编写的一小段代码感兴趣：github.com/vmlaker/benchmark-sharedmem|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3007862|唯一感兴趣的文件是main.py。这是numpy-sharedmem的一个基准测试--代码只是通过管道将数组(+numpy或sharedmem)传递给衍生的进程。工作人员只需对数据调用sum()。我只对比较两种实现之间的数据通信时间感兴趣。|style|CODE|3007863|我还编写了另一个更复杂的代码：github.com/vmlaker/sherlock。|3007864|在这里，我使用numpy-sharedmem模块通过OpenCV进行实时图像处理--图像是NumPy数组，根据OpenCV的更新的cv2应用编程接口。图像，实际上是其引用，通过从multiprocessing.Manager创建的字典对象在进程之间共享(与使用队列或管道相反)。与使用普通的NumPy数组相比，我获得了很大的性能改进。|3007865|管道与队列|BOLD|3007866|根据我的经验，使用管道的IPC比使用队列的IPC更快。这是有道理的，因为队列增加了锁定，使其对多个生产者/消费者来说是安全的。管道不需要，但是如果你只有两个进程来回通信，那么使用管道是安全的，或者，正如文档所说：|3007867|3007868|+...同时使用管道的不同末端的进程不会有损坏的风险。|blockquote|3007869|3007870|3007871|sharedmem+安全|3007872|sharedmem模块的主要问题是在不正常的程序退出时可能发生内存泄漏。在冗长的讨论here中对此进行了描述。虽然在2011年4月10日Sturla提到了内存泄漏的修复，但从那时起，我仍然经历了泄漏，使用了两个repos，Sturla+Molden自己在GitHub+(github.com/sturlamolden/sharedmem-numpy)上的和Chris+Lee-Messer在Bitbucket+(bitbucket.org/cleemesser/numpy-sharedmem)上的。|3007873|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/vmlaker/benchmark-sharedmem|1|https://bitbucket.org/cleemesser/numpy-sharedmem|2|https://github.com/vmlaker/sherlock|3|4|http://docs.python.org/2/library/multiprocessing.html#multiprocessing-managers|5|http://grokbase.com/t/python/python-list/1144s75ps4/multiprocessing-shared-memory-vs-pickled-copies|6|https://github.com/sturlamolden/sharedmem-numpy|7^0|I|12|0|0|9|7|1K|5|1Q|9|2K|5|J|F|1|0|F|R|2|0|1T|3|2H|N|7|F|3|2H|N|4|0|0|5|0|0|0|0|0|0|0|9|A|2|0|9|0|0|9|16|4|5|3R|13|6|5Q|14|7|0^^$0|@$1|2|3|4|5|6|7|1K|8|@]|9|@$A|1L|B|1M|1|1N]]|C|$]]|$1|D|3|E|5|6|7|1O|8|@$A|1P|B|1Q|F|G]|$A|1R|B|1S|F|G]|$A|1T|B|1U|F|G]|$A|1V|B|1W|F|G]]|9|@$A|1X|B|1Y|1|1Z]]|C|$]]|$1|H|3|I|5|6|7|20|8|@]|9|@$A|21|B|22|1|23]]|C|$]]|$1|J|3|K|5|6|7|24|8|@$A|25|B|26|F|G]|$A|27|B|28|F|G]]|9|@$A|29|B|2A|1|2B]|$A|2C|B|2D|1|2E]]|C|$]]|$1|L|3|M|5|6|7|2F|8|@$A|2G|B|2H|F|N]]|9|@]|C|$]]|$1|O|3|P|5|6|7|2I|8|@]|9|@]|C|$]]|$1|Q|3|-4|5|6|7|2J|8|@]|9|@]|C|$]]|$1|R|3|S|5|T|7|2K|8|@]|9|@]|C|$]]|$1|U|3|-4|5|6|7|2L|8|@]|9|@]|C|$]]|$1|V|3|-4|5|6|7|2M|8|@]|9|@]|C|$]]|$1|W|3|X|5|6|7|2N|8|@$A|2O|B|2P|F|N]|$A|2Q|B|2R|F|N]|$A|2S|B|2T|F|G]]|9|@]|C|$]]|$1|Y|3|Z|5|6|7|2U|8|@$A|2V|B|2W|F|G]]|9|@$A|2X|B|2Y|1|2Z]|$A|30|B|31|1|32]|$A|33|B|34|1|35]]|C|$]]|$1|10|3|-4|5|6|7|36|8|@]|9|@]|C|$]]]|11|$12|$5|13|14|15|C|$16|17]]|18|$5|13|14|15|C|$16|19]]|1A|$5|13|14|15|C|$16|1B]]|1C|$5|13|14|15|C|$16|19]]|1D|$5|13|14|15|C|$16|1E]]|1F|$5|13|14|15|C|$16|1G]]|1H|$5|13|14|15|C|$16|1I]]|1J|$5|13|14|15|C|$16|19]]]]

You may be interested in a tiny piece of code I wrote: <a href="https://github.com/vmlaker/benchmark-sharedmem" rel="noreferrer">github.com/vmlaker/benchmark-sharedmem</a>

The only file of interest is <code>main.py</code>. It's a benchmark of <a href="https://bitbucket.org/cleemesser/numpy-sharedmem" rel="noreferrer">numpy-sharedmem</a> -- the code simply passes arrays (either <code>numpy</code> or <code>sharedmem</code>) to spawned processes, via Pipe. The workers just call <code>sum()</code> on the data. I was only interested in comparing the data communication times between the two implementations.

I also wrote another, more complex code: <a href="https://github.com/vmlaker/sherlock" rel="noreferrer">github.com/vmlaker/sherlock</a>.

Here I use the <a href="https://bitbucket.org/cleemesser/numpy-sharedmem" rel="noreferrer">numpy-sharedmem</a> module for real-time image processing with OpenCV -- the images are NumPy arrays, as per OpenCV's newer <code>cv2</code> API. The images, actually references thereof, are shared between processes via the dictionary object created from <a href="http://docs.python.org/2/library/multiprocessing.html#multiprocessing-managers" rel="noreferrer"><code>multiprocessing.Manager</code></a> (as opposed to using Queue or Pipe.) I'm getting great performance improvements when compared with using plain NumPy arrays.

Pipe vs. Queue:

In my experience, IPC with Pipe is faster than Queue. And that makes sense, since Queue adds locking to make it safe for multiple producers/consumers. Pipe doesn't. But if you only have two processes talking back-and-forth, it's safe to use Pipe, or, as the docs read: 

<blockquote>
 ... there is no risk of corruption from processes using different ends of the pipe at the same time.
</blockquote>

<code>sharedmem</code> safety:

The main issue with <code>sharedmem</code> module is the possibility of memory leak upon ungraceful program exit. This is described in a lengthy discussion <a href="http://grokbase.com/t/python/python-list/1144s75ps4/multiprocessing-shared-memory-vs-pickled-copies" rel="noreferrer">here</a>. Although on Apr 10, 2011 Sturla mentions a fix to memory leak, I have still experienced leaks since then, using both repos, Sturla Molden's own on GitHub (<a href="https://github.com/sturlamolden/sharedmem-numpy" rel="noreferrer">github.com/sturlamolden/sharedmem-numpy</a>) and Chris Lee-Messer's on Bitbucket (<a href="https://bitbucket.org/cleemesser/numpy-sharedmem" rel="noreferrer">bitbucket.org/cleemesser/numpy-sharedmem</a>).

blocks|key|180469|text|@Velimir给出了一个很好的答案。我想我可以添加一些注释和一个小示例。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|180470|(我在sharedmem上找不到太多文档--这些是我自己的实验结果。)|180471|180472|您是否需要在子进程启动时或启动后传递句柄？如果只是前者，那么您可以只对Process使用target和args参数。这可能比使用全局变量更好。|ordered-list-item|offset|length|style|CODE|180473|从你链接的讨论页面上看，对64位Linux的支持似乎是不久前添加到sharedmem中的，所以这可能不是问题。|180474|我不知道这个问题。|180475|No。请参阅下面的示例。|180476|180477|示例|180478|#!/usr/bin/env+python
from+multiprocessing+import+Process
import+sharedmem
import+numpy

def+do_work(data,+start):
++++data[start]+=+0;

def+split_work(num):
++++n+=+20
++++width++=+n/num
++++shared+=+sharedmem.empty(n)
++++shared[:]+=+numpy.random.rand(1,+n)[0]
++++print+"values+are+%25s"+%25+shared

++++processes+=+[Process(target=do_work,+args=(shared,+i*width))+for+i+in+xrange(num)]
++++for+p+in+processes:
++++++++p.start()
++++for+p+in+processes:
++++++++p.join()

++++print+"values+are+%25s"+%25+shared
++++print+"type+is+%25s"+%25+type(shared[0])

if+__name__+==+'__main__':
++++split_work(4)|code-block|syntax|javascript|180479|输出|180480|values+are+[+0.81397784++0.59667692++0.10761908++0.6736734+++0.46349645++0.98340718
++0.44056863++0.10701816++0.67167752++0.29158274++0.22242552++0.14273156
++0.34912309++0.43812636++0.58484507++0.81697513++0.57758441++0.4284959
++0.7292129+++0.06063283]
values+are+[+0.++++++++++0.59667692++0.10761908++0.6736734+++0.46349645++0.
++0.44056863++0.10701816++0.67167752++0.29158274++0.++++++++++0.14273156
++0.34912309++0.43812636++0.58484507++0.++++++++++0.57758441++0.4284959
++0.7292129+++0.06063283]
type+is+<type+'numpy.float64'>|180481|此related+question可能会很有用。|180482|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/7894791/use-numpy-array-in-shared-memory-for-multiprocessing^0|0|0|0|Z|7|18|6|1F|4|0|0|0|0|0|0|0|0|0|1|G|0|0^^$0|@$1|2|3|4|5|6|7|1D|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1E|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|1F|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|1G|8|@$H|1H|I|1I|J|K]|$H|1J|I|1K|J|K]|$H|1L|I|1M|J|K]]|9|@]|A|$]]|$1|L|3|M|5|G|7|1N|8|@]|9|@]|A|$]]|$1|N|3|O|5|G|7|1O|8|@]|9|@]|A|$]]|$1|P|3|Q|5|G|7|1P|8|@]|9|@]|A|$]]|$1|R|3|-4|5|6|7|1Q|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|1R|8|@]|9|@]|A|$]]|$1|U|3|V|5|W|7|1S|8|@]|9|@]|A|$X|Y]]|$1|Z|3|10|5|6|7|1T|8|@]|9|@]|A|$]]|$1|11|3|12|5|W|7|1U|8|@]|9|@]|A|$X|Y]]|$1|13|3|14|5|6|7|1V|8|@]|9|@$H|1W|I|1X|1|1Y]]|A|$]]|$1|15|3|-4|5|6|7|1Z|8|@]|9|@]|A|$]]]|16|$17|$5|18|19|1A|A|$1B|1C]]]]

@Velimir Mlaker gave a great answer. I thought I could add some bits of comments and a tiny example.

(I couldn't find much documentation on sharedmem - these are the results of my own experiments.)

<ol>
<li>Do you need to pass the handles when the subprocess is starting, or after it has started? If it's just the former, you can just use the <code>target</code> and <code>args</code> arguments for <code>Process</code>. This is potentially better than using a global variable.</li>
<li>From the discussion page you linked, it appears that support for 64-bit Linux was added to sharedmem a while back, so it could be a non-issue.</li>
<li>I don't know about this one.</li>
<li>No. Refer to example below.</li>
</ol>

<h3>Example</h3>

<pre><code>#!/usr/bin/env python
from multiprocessing import Process
import sharedmem
import numpy

def do_work(data, start):
 data[start] = 0;

def split_work(num):
 n = 20
 width = n/num
 shared = sharedmem.empty(n)
 shared[:] = numpy.random.rand(1, n)[0]
 print "values are %s" % shared

 processes = [Process(target=do_work, args=(shared, i*width)) for i in xrange(num)]
 for p in processes:
 p.start()
 for p in processes:
 p.join()

 print "values are %s" % shared
 print "type is %s" % type(shared[0])

if __name__ == '__main__':
 split_work(4)
</code></pre>

<h3>Output</h3>

<pre><code>values are [ 0.81397784 0.59667692 0.10761908 0.6736734 0.46349645 0.98340718
 0.44056863 0.10701816 0.67167752 0.29158274 0.22242552 0.14273156
 0.34912309 0.43812636 0.58484507 0.81697513 0.57758441 0.4284959
 0.7292129 0.06063283]
values are [ 0. 0.59667692 0.10761908 0.6736734 0.46349645 0.
 0.44056863 0.10701816 0.67167752 0.29158274 0. 0.14273156
 0.34912309 0.43812636 0.58484507 0. 0.57758441 0.4284959
 0.7292129 0.06063283]
type is &lt;type 'numpy.float64'&gt;
</code></pre>

This <a href="https://stackoverflow.com/questions/7894791/use-numpy-array-in-shared-memory-for-multiprocessing">related question</a> might be useful.

blocks|key|181920|text|您可能还会发现，查看pyro的文档很有用，因为如果您可以对任务进行适当的分区，您可以使用它在不同的机器上以及同一机器的不同内核上执行不同的部分。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|181921|entityMap|0|LINK|mutability|MUTABLE|url|http://pythonhosted.org/Pyro4/^0|A|4|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

You may also find it useful to take a look at the documentation for <a href="http://pythonhosted.org/Pyro4/" rel="nofollow">pyro</a> as if you can partition your task appropriately you could use it to execute different sections on different machines as well as on different cores in the same machine.

blocks|key|182842|text|为什么不使用多线程呢？主进程的资源可以由其线程在本地共享，因此多线程显然是共享主进程拥有的对象的一种更好的方式。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|182843|如果你担心python的GIL机制，也许你可以求助于numba的nogil。|offset|length|style|CODE|182844|entityMap|0|LINK|mutability|MUTABLE|url|https://numba.pydata.org/numba-doc/latest/user/jit.html#nogil^0|0|Q|5|W|5|W|5|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@$D|R|E|S|F|G]|$D|T|E|U|F|G]]|9|@$D|V|E|W|1|X]]|A|$]]|$1|H|3|-4|5|6|7|Y|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

Why not use multithreading?
Resources of main process can be shared by its threads natively, thus multithreading is obviously a better way to share objects owned by the main process.
If you worry about python's GIL mechanism, maybe you can resort to the <a href="https://numba.pydata.org/numba-doc/latest/user/jit.html#nogil" rel="nofollow noreferrer"><code>nogil</code></a> of <code>numba</code>.

I have a 60GB SciPy Array (Matrix) I must share between 5+ <code>multiprocessing</code> <code>Process</code> objects. I've seen numpy-sharedmem and read <a href="http://grokbase.com/t/python/python-list/1144s75ps4/multiprocessing-shared-memory-vs-pickled-copies">this discussion</a> on the SciPy list. There seem to be two approaches--<code>numpy-sharedmem</code> and using a <code>multiprocessing.RawArray()</code> and mapping NumPy <code>dtype</code>s to <code>ctype</code>s. Now, <code>numpy-sharedmem</code> seems to be the way to go, but I've yet to see a good reference example. I don't need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I'd like to avoid a copy. It sounds like the correct method is to create the only copy of the array as a <code>sharedmem</code> array, and then pass it to the <code>Process</code> objects? A couple of specific questions:

<ol>
<li>What's the best way to actually pass the sharedmem handles to sub-<code>Process()</code>es? Do I need a queue just to pass one array around? Would a pipe be better? Can I just pass it as an argument to the <code>Process()</code> subclass's init (where I'm assuming it's pickled)?</li>
<li>In the discussion I linked above, there's mention of <code>numpy-sharedmem</code> not being 64bit-safe? I'm definitely using some structures that aren't 32-bit addressable.</li>
<li>Are there tradeoff's to the <code>RawArray()</code> approach? Slower, buggier?</li>
<li>Do I need any ctype-to-dtype mapping for the numpy-sharedmem method?</li>
<li>Does anyone have an example of some OpenSource code doing this? I'm a very hands-on learned and it's hard to get this working without any kind of good example to look at.</li>
</ol>

If there's any additional info I can provide to help clarify this for others, please comment and I'll add. Thanks!

This needs to run on Ubuntu Linux and Maybe Mac OS, but portability isn't a huge concern.

Share Large, Read-Only Numpy Array Between Multiprocessing Processes

图像处理

Bitbucket

Windows

Python

GitHub

Ubuntu

Linux

我有一个60 5+的SciPy阵列(矩阵)，必须在5+ multiprocessing Process对象之间共享。我在SciPy列表上看过numpy-sharedmem和。似乎有两种方法--numpy-sharedmem和使用multiprocessing.RawArray()以及将NumPy dtypes映射到ct...

问在多进程之间共享大型只读Numpy数组
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在多进程之间共享大型只读Numpy数组EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在多进程之间共享大型只读Numpy数组
EN