blocks|key|408668|text|在我看来，让一个多线程C应用程序尝试从多个线程到单个CPython实例的多个Python线程进行通信是有风险的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|408669|只要只有一个C线程与Python通信，就不必担心锁定问题，即使Python应用程序是多线程的。如果您需要多个python线程，您可以这样设置应用程序，让多个C线程通过一个队列与单个C线程进行通信，该C线程将把它们分发给多个Python线程。|408670|另一种可能适合您的替代方法是让多个CPython实例为每个需要它的C线程提供一个实例(当然，Python程序之间的通信应该通过C程序进行)。|408671|另一种选择可能是Stackless+Python解释器。这消除了GIL，但我不确定您是否会遇到将它绑定到多个线程的其他问题。stackless是我的(单线程)C应用程序的临时替代品。|408672|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

Having a multi-threaded C app trying to communicate from multiple threads to multiple Python threads of a single CPython instance looks risky to me.

As long as only one C thread communicates with Python you should not have to worry about locking even if the Python application is multi-threading.
If you need multiple python threads you can set the application up this way and have multiple C threads communicate via a queue with that single C thread that farms them out to multiple Python threads.

An alternative that might work for you is to have multiple CPython instances one for each C thread that needs it (of course communication between Python programs should be via the C program).

Another alternative might the Stackless Python interpreter. That does away with the GIL, but I am not sure you run into other problems binding it to multiple threads. stackless was a drop-in replacement for my (single-threaded) C application.

blocks|key|3457853|text|最终我发现了这一点。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3457854|之后|3457855|PyEval_InitThreads();|code-block|syntax|javascript|3457856|你需要打电话给|3457857|PyEval_SaveThread();|3457858|同时适当地释放主线程的GIL。|3457859|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|S|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|T|8|@]|9|@]|A|$]]|$1|K|3|L|5|F|7|U|8|@]|9|@]|A|$G|H]]|$1|M|3|N|5|6|7|V|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

Eventually I figured it out. 
After 

<pre><code>PyEval_InitThreads();
</code></pre>

You need to call 

<pre><code>PyEval_SaveThread();
</code></pre>

While properly release the GIL for the main thread.

blocks|key|3244621|text|我遇到了完全相同的问题，现在可以按照您上面的建议，在PyEval_InitThreads()之后立即使用PyEval_SaveThread()来解决这个问题。然而，我的实际问题是，我在PyInitialise()之后使用PyEval_InitThreads()，这会导致PyGILState_Ensure()在从不同的后续本机线程调用时阻塞。总而言之，这是我现在要做的：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3244622|3244623|有一个全局变量：|ordered-list-item|3244624|3244625|static+int+=+0；|3244626|从主线程加载本地C扩展并启动Python解释器：|unordered-list-item|3244627|3244628|Py_Initialize()+|3244629|从多个其他线程我的应用程序并发地对Python/C应用程序接口进行了大量调用：|3244630|3244631|如果(!+gil_init+)+{gil_init=+1；PyEval_InitThreads()；PyEval_SaveThread()；}状态=+PyGILState_Ensure()；//调用Python/C+API函数...PyGILState_Release(state);|3244632|From主线程停止Python解释器|3244633|3244634|Py_Finalize()|3244635|我尝试过的所有其他解决方案要么会导致随机的Python信号错误，要么会使用PyGILState_Ensure()导致死锁/阻塞。|3244636|Python文档确实应该更清楚地说明这一点，并至少为嵌入和扩展用例提供一个示例。|3244637|entityMap^0|Q|K|1G|J|2K|E|32|K|3R|J|0|0|0|0|0|0|0|0|0|0|0|0|0|0|11|J|0|0^^$0|@$1|2|3|4|5|6|7|18|8|@$9|19|A|1A|B|C]|$9|1B|A|1C|B|C]|$9|1D|A|1E|B|C]|$9|1F|A|1G|B|C]|$9|1H|A|1I|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|1J|8|@]|D|@]|E|$]]|$1|G|3|H|5|I|7|1K|8|@]|D|@]|E|$]]|$1|J|3|-4|5|6|7|1L|8|@]|D|@]|E|$]]|$1|K|3|L|5|6|7|1M|8|@]|D|@]|E|$]]|$1|M|3|N|5|O|7|1N|8|@]|D|@]|E|$]]|$1|P|3|-4|5|6|7|1O|8|@]|D|@]|E|$]]|$1|Q|3|R|5|6|7|1P|8|@]|D|@]|E|$]]|$1|S|3|T|5|O|7|1Q|8|@]|D|@]|E|$]]|$1|U|3|-4|5|6|7|1R|8|@]|D|@]|E|$]]|$1|V|3|W|5|6|7|1S|8|@]|D|@]|E|$]]|$1|X|3|Y|5|O|7|1T|8|@]|D|@]|E|$]]|$1|Z|3|-4|5|6|7|1U|8|@]|D|@]|E|$]]|$1|10|3|11|5|6|7|1V|8|@]|D|@]|E|$]]|$1|12|3|13|5|6|7|1W|8|@$9|1X|A|1Y|B|C]]|D|@]|E|$]]|$1|14|3|15|5|6|7|1Z|8|@]|D|@]|E|$]]|$1|16|3|-4|5|6|7|20|8|@]|D|@]|E|$]]]|17|$]]

I had exactly the same problem and it is now solved by using <code>PyEval_SaveThread()</code> immediately after <code>PyEval_InitThreads()</code>, as you suggest above. However, my actual problem was that I used <code>PyEval_InitThreads()</code> after <code>PyInitialise()</code> which then caused <code>PyGILState_Ensure()</code> to block when called from different, subsequent native threads. In summary, this is what I do now:

<ol>
<li>There is global variable: 

<pre><code>static int gil_init = 0; 
</code></pre></li>
<li>From a main thread load the native C extension and start the Python interpreter:

<pre><code>Py_Initialize() 
</code></pre></li>
<li>From multiple other threads my app concurrently makes a lot of calls into the Python/C API:

<pre><code>if (!gil_init) {
 gil_init = 1;
 PyEval_InitThreads();
 PyEval_SaveThread();
}
state = PyGILState_Ensure();
// Call Python/C API functions... 
PyGILState_Release(state);
</code></pre></li>
<li>From the main thread stop the Python interpreter

<pre><code>Py_Finalize()
</code></pre></li>
</ol>

All other solutions I've tried either caused random Python sigfaults or deadlock/blocking using <code>PyGILState_Ensure()</code>. 

The Python documentation really should be more clear on this and at least provide an example for both the embedding and extension use cases.

blocks|key|3457901|text|注意，@forman的答案中的if+(!gil_init)+{代码只运行一次，所以它可以在主线程中很好地完成，这允许我们删除标志(gil_init应该是原子的或以其他方式同步的)。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3457902|PyEval_InitThreads()只有在CPython+3.6和更早的版本中才有意义，并且在CPython+3.9中已被弃用，因此必须用宏保护它。|3457903|考虑到所有这些，我目前使用的是：|3457904|在主线程中，运行|3457905|Py_Initialize();
PyEval_InitThreads();+//+only+on+Python+3.6+or+older!
/*+tstate+=+*/+PyEval_SaveThread();+//+maybe+save+the+return+value+if+you+need+it+later|code-block|syntax|javascript|3457906|现在，无论何时需要调用Python，都可以|3457907|state+=+PyGILState_Ensure();
//+Call+Python/C+API+functions...++++
PyGILState_Release(state);|3457908|最后，从主线程停止Python解释器|3457909|PyGILState_Ensure();+//+PyEval_RestoreThread(tstate);+seems+to+work+just+as+well
Py_Finalize()|3457910|entityMap^0|F|G|1T|8|0|0|K|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]|$9|13|A|14|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|15|8|@$9|16|A|17|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|18|8|@]|D|@]|E|$]]|$1|J|3|K|5|6|7|19|8|@]|D|@]|E|$]]|$1|L|3|M|5|N|7|1A|8|@]|D|@]|E|$O|P]]|$1|Q|3|R|5|6|7|1B|8|@]|D|@]|E|$]]|$1|S|3|T|5|N|7|1C|8|@]|D|@]|E|$O|P]]|$1|U|3|V|5|6|7|1D|8|@]|D|@]|E|$]]|$1|W|3|X|5|N|7|1E|8|@]|D|@]|E|$O|P]]|$1|Y|3|-4|5|6|7|1F|8|@]|D|@]|E|$]]]|Z|$]]

Note that the <code>if (!gil_init) {</code> code in @forman's answer runs only once, so it can be just as well done in the main thread, which allows us to drop the flag (<code>gil_init</code> would properly have to be atomic or otherwise synchronized).
<code>PyEval_InitThreads()</code> is meaningful only in CPython 3.6 and older, and has been deprecated in CPython 3.9, so it has to be guarded with a macro.
Given all this, what I am currently using is the following:
In the main thread, run all of
<pre class="lang-c prettyprint-override"><code>Py_Initialize();
PyEval_InitThreads(); // only on Python 3.6 or older!
/* tstate = */ PyEval_SaveThread(); // maybe save the return value if you need it later
</code></pre>
Now, whenever you need to call into Python, do
<pre class="lang-c prettyprint-override"><code>state = PyGILState_Ensure();
// Call Python/C API functions... 
PyGILState_Release(state);
</code></pre>
Finally, from the main thread, stop the Python interpreter
<pre class="lang-c prettyprint-override"><code>PyGILState_Ensure(); // PyEval_RestoreThread(tstate); seems to work just as well
Py_Finalize()
</code></pre>

I'm embedding the python interpreter in a multithreaded C application and I'm a little confused as to what APIs I should use to ensure thread safety.

From what I gathered, when embedding python it is up to the embedder to take care of the GIL lock before calling any other Python C API call. This is done with these functions:

<pre><code>gstate = PyGILState_Ensure();
// do some python api calls, run python scripts
PyGILState_Release(gstate);
</code></pre>

But this alone doesn't seem to be enough. I still got random crashes since it doesn't seem to provide mutual exclusion for the Python APIs.

After reading some more docs I also added: 

<pre><code>PyEval_InitThreads();
</code></pre>

right after the call to <code>Py_IsInitialized()</code> but that's where the confusing part comes. The docs state that this function:

<blockquote>
 Initialize and acquire the global interpreter lock
</blockquote>

This suggests that when this function returns, the GIL is supposed to be locked and should be unlocked somehow. but in practice this doesn't seem to be required. With this line in place my multithreaded worked perfectly and mutual exclusion was maintained by the <code>PyGILState_Ensure/Release</code> functions. 
When I tried adding <code>PyEval_ReleaseLock()</code> after <code>PyEval_ReleaseLock()</code> the app dead-locked pretty quickly in a subsequent call to <code>PyImport_ExecCodeModule()</code>.

So what am I missing here?

Embedding python in multithreaded C application

解释器

Python

我将python解释器嵌入到一个多线程的C应用程序中，对于应该使用什么API来确保线程安全，我感到有点困惑。据我所知，在嵌入python时，在调用任何其他Python C API调用之前，需要由嵌入器来处理GIL锁。这是通过以下函数完成的：gstate = PyGILState_Ensure();// do some ...

问在多线程C应用程序中嵌入python
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在多线程C应用程序中嵌入pythonEN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在多线程C应用程序中嵌入python
EN