blocks|key|4688791|text|CPython使用全局解释器锁，这使得并行编程比C%2B%2B更有趣|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4688792|本主题提供了几个有关挑战的有用示例和说明：|4688793|Python+Global+Interpreter+Lock+(GIL)+workaround+on+multi-core+systems+using+taskset+on+Linux?|offset|length|4688794|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/990102/python-global-interpreter-lock-gil-workaround-on-multi-core-systems-using-task^0|0|0|0|2L|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@$F|S|G|T|1|U]]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

CPython uses the Global Interpreter Lock which makes parallel programing a bit more interesting than C++

This topic has several useful examples and descriptions of the challenge:

<a href="https://stackoverflow.com/questions/990102/python-global-interpreter-lock-gil-workaround-on-multi-core-systems-using-task">Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?</a>

blocks|key|182949|text|您可以使用multiprocessing模块。在这种情况下，我可能会使用一个处理池：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|182950|from+multiprocessing+import+Pool
pool+=+Pool()
result1+=+pool.apply_async(solve1,+[A])++++#+evaluate+"solve1(A)"+asynchronously
result2+=+pool.apply_async(solve2,+[B])++++#+evaluate+"solve2(B)"+asynchronously
answer1+=+result1.get(timeout=10)
answer2+=+result2.get(timeout=10)|code-block|syntax|javascript|182951|这将产生可以为您执行常规工作的进程。因为我们没有传递processes，所以它将为您机器上的每个CPU核心生成一个进程。每个CPU核心可以同时执行一个进程。|style|CODE|182952|如果你想把一个列表映射到一个函数，你可以这样做：|182953|args+=+[A,+B]
results+=+pool.map(solve1,+args)|182954|不要使用线程，因为GIL会锁定python对象上的所有操作。|182955|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/2/library/multiprocessing.html|1|https://wiki.python.org/moin/GlobalInterpreterLock^0|5|F|0|0|0|Q|9|0|0|0|9|3|1|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@$A|13|B|14|1|15]]|C|$]]|$1|D|3|E|5|F|7|16|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|17|8|@$A|18|B|19|K|L]]|9|@]|C|$]]|$1|M|3|N|5|6|7|1A|8|@]|9|@]|C|$]]|$1|O|3|P|5|F|7|1B|8|@]|9|@]|C|$G|H]]|$1|Q|3|R|5|6|7|1C|8|@]|9|@$A|1D|B|1E|1|1F]]|C|$]]|$1|S|3|-4|5|6|7|1G|8|@]|9|@]|C|$]]]|T|$U|$5|V|W|X|C|$Y|Z]]|10|$5|V|W|X|C|$Y|11]]]]

You can use the <a href="http://docs.python.org/2/library/multiprocessing.html">multiprocessing</a> module. For this case I might use a processing pool:

<pre><code>from multiprocessing import Pool
pool = Pool()
result1 = pool.apply_async(solve1, [A]) # evaluate "solve1(A)" asynchronously
result2 = pool.apply_async(solve2, [B]) # evaluate "solve2(B)" asynchronously
answer1 = result1.get(timeout=10)
answer2 = result2.get(timeout=10)
</code></pre>

This will spawn processes that can do generic work for you. Since we did not pass <code>processes</code>, it will spawn one process for each CPU core on your machine. Each CPU core can execute one process simultaneously.

If you want to map a list to a single function you would do this:

<pre><code>args = [A, B]
results = pool.map(solve1, args)
</code></pre>

Don't use threads because the <a href="https://wiki.python.org/moin/GlobalInterpreterLock">GIL</a> locks any operations on python objects.

blocks|key|4688893|text|在某些情况下，可以使用Numba自动并行化循环，尽管它只适用于Python子集：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4688894|from+numba+import+njit,+prange

@njit(parallel=True)
def+prange_test(A):
++++s+=+0
++++#+Without+"parallel=True"+in+the+jit-decorator
++++#+the+prange+statement+is+equivalent+to+range
++++for+i+in+prange(A.shape[0]):
++++++++s+%2B=+A[i]
++++return+s|code-block|syntax|javascript|4688895|不幸的是，Numba似乎只适用于Numpy数组，而不适用于其他Python对象。从理论上讲，也可以先使用compile+Python+to+C%2B%2B，然后使用automatically+parallelize+it+using+the+Intel+C%2B%2B+compiler，尽管我还没有尝试过。|4688896|entityMap|0|LINK|mutability|MUTABLE|url|https://numba.pydata.org/numba-doc/dev/user/parallel.html|1|https://stackoverflow.com/questions/4650243/convert-python-program-to-c-c-code|2|https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=2ahUKEwiuucmL8oflAhURna0KHcCKCrIQFjABegQIBhAD&url=https%253A%252F%252Fsoftware.intel.com%252Fen-us%252Fcpp-compiler-developer-guide-and-reference-automatic-parallelization&usg=AOvVaw0tMzwSvV4bIh7nPa0q2O1s^0|B|5|0|0|0|1G|L|1|26|1L|2|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|F|7|10|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|11|8|@]|9|@$A|12|B|13|1|14]|$A|15|B|16|1|17]]|C|$]]|$1|K|3|-4|5|6|7|18|8|@]|9|@]|C|$]]]|L|$M|$5|N|O|P|C|$Q|R]]|S|$5|N|O|P|C|$Q|T]]|U|$5|N|O|P|C|$Q|V]]]]

In some cases, it's possible to automatically parallelize loops using <a href="https://numba.pydata.org/numba-doc/dev/user/parallel.html" rel="nofollow noreferrer">Numba</a>, though it only works with a small subset of Python:

<pre><code>from numba import njit, prange

@njit(parallel=True)
def prange_test(A):
 s = 0
 # Without "parallel=True" in the jit-decorator
 # the prange statement is equivalent to range
 for i in prange(A.shape[0]):
 s += A[i]
 return s
</code></pre>

Unfortunately, it seems that Numba only works with Numpy arrays, but not with other Python objects. In theory, it might also be possible to <a href="https://stackoverflow.com/questions/4650243/convert-python-program-to-c-c-code">compile Python to C++</a> and then <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=2&amp;ved=2ahUKEwiuucmL8oflAhURna0KHcCKCrIQFjABegQIBhAD&amp;url=https%3A%2F%2Fsoftware.intel.com%2Fen-us%2Fcpp-compiler-developer-guide-and-reference-automatic-parallelization&amp;usg=AOvVaw0tMzwSvV4bIh7nPa0q2O1s" rel="nofollow noreferrer">automatically parallelize it using the Intel C++ compiler</a>, though I haven't tried this yet.

blocks|key|183026|text|您可以使用joblib库进行并行计算和多处理。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|183027|from+joblib+import+Parallel,+delayed|code-block|syntax|javascript|183028|您可以简单地创建一个想要并行运行的函数foo，并基于以下代码实现并行处理：|183029|output+=+Parallel(n_jobs=num_cores)(delayed(foo)(i)+for+i+in+input)|183030|其中，可以从multiprocessing库中获取num_cores，如下所示：|183031|import+multiprocessing

num_cores+=+multiprocessing.cpu_count()|183032|如果您有一个具有多个输入参数的函数，并且您只想通过列表迭代其中一个参数，则可以使用functools库中的partial函数，如下所示：|183033|from+joblib+import+Parallel,+delayed
import+multiprocessing
from+functools+import+partial
def+foo(arg1,+arg2,+arg3,+arg4):
++++'''
++++body+of+the+function
++++'''
++++return+output
input+=+[11,32,44,55,23,0,100,...]+#+arbitrary+list
num_cores+=+multiprocessing.cpu_count()
foo_+=+partial(foo,+arg2=arg2,+arg3=arg3,+arg4=arg4)
#+arg1+is+being+fetched+from+input+list
output+=+Parallel(n_jobs=num_cores)(delayed(foo_)(i)+for+i+in+input)|183034|您可以通过两个示例here找到对python和R多处理的完整解释。|183035|entityMap|0|LINK|mutability|MUTABLE|url|https://medium.com/@najari.vahab/parallel-computing-in-r-and-python-1c916a2803d7^0|5|6|0|0|J|3|0|0|6|F|P|9|0|0|15|9|1H|7|0|0|9|4|0|0^^$0|@$1|2|3|4|5|6|7|16|8|@$9|17|A|18|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|19|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|1A|8|@$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|M|3|N|5|H|7|1D|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|1E|8|@$9|1F|A|1G|B|C]|$9|1H|A|1I|B|C]]|D|@]|E|$]]|$1|Q|3|R|5|H|7|1J|8|@]|D|@]|E|$I|J]]|$1|S|3|T|5|6|7|1K|8|@$9|1L|A|1M|B|C]|$9|1N|A|1O|B|C]]|D|@]|E|$]]|$1|U|3|V|5|H|7|1P|8|@]|D|@]|E|$I|J]]|$1|W|3|X|5|6|7|1Q|8|@]|D|@$9|1R|A|1S|1|1T]]|E|$]]|$1|Y|3|-4|5|6|7|1U|8|@]|D|@]|E|$]]]|Z|$10|$5|11|12|13|E|$14|15]]]]

You can use <code>joblib</code> library to do parallel computation and multiprocessing. 

<pre><code>from joblib import Parallel, delayed
</code></pre>

You can simply create a function <code>foo</code> which you want to be run in parallel and based on the following piece of code implement parallel processing:

<pre><code>output = Parallel(n_jobs=num_cores)(delayed(foo)(i) for i in input)
</code></pre>

Where <code>num_cores</code> can be obtained from <code>multiprocessing</code> library as followed:

<pre><code>import multiprocessing

num_cores = multiprocessing.cpu_count()
</code></pre>

If you have a function with more than one input argument, and you just want to iterate over one of the arguments by a list, you can use the the <code>partial</code> function from <code>functools</code> library as follow:

<pre><code>from joblib import Parallel, delayed
import multiprocessing
from functools import partial
def foo(arg1, arg2, arg3, arg4):
 '''
 body of the function
 '''
 return output
input = [11,32,44,55,23,0,100,...] # arbitrary list
num_cores = multiprocessing.cpu_count()
foo_ = partial(foo, arg2=arg2, arg3=arg3, arg4=arg4)
# arg1 is being fetched from input list
output = Parallel(n_jobs=num_cores)(delayed(foo_)(i) for i in input)
</code></pre>

You can find a complete explanation of the python and R multiprocessing with couple of examples <a href="https://medium.com/@najari.vahab/parallel-computing-in-r-and-python-1c916a2803d7" rel="nofollow noreferrer">here</a>.

For C++, we can use OpenMP to do parallel programming; however, OpenMP will not work for Python. What should I do if I want to parallel some parts of my python program?
The structure of the code may be considered as:
<pre><code>solve1(A)
solve2(B)
</code></pre>
Where <code>solve1</code> and <code>solve2</code> are two independent function. How to run this kind of code in parallel instead of in sequence in order to reduce the running time?
The code is:
<pre><code>def solve(Q, G, n):
 i = 0
 tol = 10 ** -4

 while i &lt; 1000:
 inneropt, partition, x = setinner(Q, G, n)
 outeropt = setouter(Q, G, n)

 if (outeropt - inneropt) / (1 + abs(outeropt) + abs(inneropt)) &lt; tol:
 break
 
 node1 = partition[0]
 node2 = partition[1]
 
 G = updateGraph(G, node1, node2)

 if i == 999:
 print &quot;Maximum iteration reaches&quot;
 print inneropt
</code></pre>
Where <code>setinner</code> and <code>setouter</code> are two independent functions. That's where I want to parallel...

How to do parallel programming in Python?

解释器

Python

对于C++，我们可以使用OpenMP进行并行编程；但是，OpenMP不适用于Python.如果我想并行我的python程序的某些部分，我应该怎么做？代码的结构可以被认为是：solve1(A)solve2(B)其中solve1和solve2是两个独立的函数。如何并行运行这类代码，而不是按顺序运行，以减少运行时间？代码是：...

问如何在Python中进行并行编程？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Python中进行并行编程？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Python中进行并行编程？
EN