blocks|key|4111908|text|@a_guest的代码：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4111909|def+foo1(data,+thresh):
++++data+=+np.asarray(data)
++++thresh+=+np.asarray(thresh)
++++condition+=+(
+++++++(data.min(axis=1)[:,+None]+<+thresh)
+++++++&+(data.max(axis=1)[:,+None]+>+thresh)
+++++++)
++++result+=+{v:+data[c].tolist()+for+c,+v+in+zip(condition.T,+thresh)}
++++return+result|code-block|syntax|javascript|4111910|这段代码为thresh中的每个条目创建一次字典条目。|offset|length|style|CODE|4111911|操作码，使用default_dict+(来自collections)进行了一点简化：|4111912|def+foo3(list_tuple,+threeshold):
++++result+=+defaultdict(list)
++++for+tuple+in+list_tuple:
++++++++for+value+in+threeshold:
++++++++++++if+max(tuple)>value+and+min(tuple)<value:
++++++++++++++++result[value].append(tuple)
++++return+result|4111913|它为满足条件的每个条目更新一次字典条目。|4111914|和他的样本数据：|4111915|In+[27]:+foo1(data,thresh)
Out[27]:+{0:+[],+1:+[[0,+1,+2]],+2:+[],+3:+[],+4:+[[3,+4,+5]]}
In+[28]:+foo3(data.tolist(),+thresh.tolist())
Out[28]:+defaultdict(list,+{1:+[[0,+1,+2]],+4:+[[3,+4,+5]]})|4111916|时间测试：|4111917|In+[29]:+timeit+foo1(data,thresh)
66.1+µs+±+197+ns+per+loop+(mean+±+std.+dev.+of+7+runs,+10000+loops+each)
#+In+[30]:+timeit+foo3(data,thresh)
#+161+µs+±+242+ns+per+loop+(mean+±+std.+dev.+of+7+runs,+10000+loops+each)
In+[31]:+timeit+foo3(data.tolist(),thresh.tolist())
30.8+µs+±+56.4+ns+per+loop+(mean+±+std.+dev.+of+7+runs,+10000+loops+each)|4111918|对数组的迭代比对列表的迭代慢。tolist()的时间很短；列表的np.asarray较长。|4111919|对于较大的数据样本，array版本速度更快：|4111920|In+[42]:+data+=+np.random.randint(0,50,(3000,3))
++++...:+thresh+=+np.arange(50)
In+[43]:+
In+[43]:+timeit+foo1(data,thresh)
16+ms+±+391+µs+per+loop+(mean+±+std.+dev.+of+7+runs,+100+loops+each)
In+[44]:+%25%25timeit+x,y+=+data.tolist(),+thresh.tolist()+
++++...:+foo3(x,y)
++++...:+
83.6+ms+±+68.6+µs+per+loop+(mean+±+std.+dev.+of+7+runs,+10+loops+each)|4111921|entityMap^0|0|0|5|6|0|6|C|M|B|0|0|0|0|0|0|0|F|8|W|A|0|A|5|0|0^^$0|@$1|2|3|4|5|6|7|18|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|19|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|1A|8|@$I|1B|J|1C|K|L]]|9|@]|A|$]]|$1|M|3|N|5|6|7|1D|8|@$I|1E|J|1F|K|L]|$I|1G|J|1H|K|L]]|9|@]|A|$]]|$1|O|3|P|5|D|7|1I|8|@]|9|@]|A|$E|F]]|$1|Q|3|R|5|6|7|1J|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|1K|8|@]|9|@]|A|$]]|$1|U|3|V|5|D|7|1L|8|@]|9|@]|A|$E|F]]|$1|W|3|X|5|6|7|1M|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|D|7|1N|8|@]|9|@]|A|$E|F]]|$1|10|3|11|5|6|7|1O|8|@$I|1P|J|1Q|K|L]|$I|1R|J|1S|K|L]]|9|@]|A|$]]|$1|12|3|13|5|6|7|1T|8|@$I|1U|J|1V|K|L]]|9|@]|A|$]]|$1|14|3|15|5|D|7|1W|8|@]|9|@]|A|$E|F]]|$1|16|3|-4|5|6|7|1X|8|@]|9|@]|A|$]]]|17|$]]

@a_guest's code:

<pre><code>def foo1(data, thresh):
 data = np.asarray(data)
 thresh = np.asarray(thresh)
 condition = (
 (data.min(axis=1)[:, None] &lt; thresh)
 &amp; (data.max(axis=1)[:, None] &gt; thresh)
 )
 result = {v: data[c].tolist() for c, v in zip(condition.T, thresh)}
 return result
</code></pre>

This code creates a dictionary entry once for each item in <code>thresh</code>.

The OP code, simplified a bit with <code>default_dict</code> (from <code>collections</code>):

<pre><code>def foo3(list_tuple, threeshold):
 result = defaultdict(list)
 for tuple in list_tuple:
 for value in threeshold:
 if max(tuple)&gt;value and min(tuple)&lt;value:
 result[value].append(tuple)
 return result
</code></pre>

This one updates a dictionary entry once for each item that meets the criteria.

And with his sample data:

<pre><code>In [27]: foo1(data,thresh)
Out[27]: {0: [], 1: [[0, 1, 2]], 2: [], 3: [], 4: [[3, 4, 5]]}
In [28]: foo3(data.tolist(), thresh.tolist())
Out[28]: defaultdict(list, {1: [[0, 1, 2]], 4: [[3, 4, 5]]})
</code></pre>

time tests:

<pre><code>In [29]: timeit foo1(data,thresh)
66.1 µs ± 197 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# In [30]: timeit foo3(data,thresh)
# 161 µs ± 242 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [31]: timeit foo3(data.tolist(),thresh.tolist())
30.8 µs ± 56.4 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>

Iteration on arrays is slower than with lists. Time for <code>tolist()</code> is minimal; <code>np.asarray</code> for lists is longer.

With a larger data sample, the <code>array</code> version is faster:

<pre><code>In [42]: data = np.random.randint(0,50,(3000,3))
 ...: thresh = np.arange(50)
In [43]: 
In [43]: timeit foo1(data,thresh)
16 ms ± 391 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [44]: %%timeit x,y = data.tolist(), thresh.tolist() 
 ...: foo3(x,y)
 ...: 
83.6 ms ± 68.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre>

First sorry for my not perfect English.

My problem is simple to explain, I think.

<pre><code>result={}
list_tuple=[(float,float,float),(float,float,float),(float,float,float)...]#200k tuples
threshold=[float,float,float...] #max 1k values
for tuple in list_tuple:
 for value in threeshold:
 if max(tuple)&gt;value and min(tuple)&lt;value:
 if value in result:
 result[value].append(tuple)
 else:
 result[value]=[]
 result[value].append(tuple) 
</code></pre>

list_tuple contains arround 200k tuples, i have to do this operation very fast(2/3 seconds max on a normal pc).

My first attemp was to do this in cython with prange() (so i could have benefits from the cython optimization and from the paralell execution), but the problem is (as always), GIL: in prange() i can manage lists and tuples using cython memviews, but i can't insert my result in a dict.

In cython i also tried using unordered_map of the c++ std, but now the problem is that i can't make a vector of array in c++ (that would the value of my dict).

The second problem is similar:

<pre><code>list_tuple=[((float,float),(float,float)),((float,float),(float,float))...]#200k tuples of tuples

result={list_tuple[0][0]:[]}

for tuple in list_tuple:
 if tuple[0] in result:
 result[tuple[0]].append(tuple)
 else:
 result[tuple[0]]=[]
</code></pre>

Here i have also another problem,if a want to use prange() i have to use a custom hash function to use an array as key of a c++ unordered_map

As you can see my snippets are very simple to run in paralell.

I thought to try with numba, but probably will be the same because of GIL, and i prefer to use cython because i need a binary code (this library could be a part of a commercial software so only binary libraries are allowed).

In general i would like avoid c/c++ function, what i hope to find is a way to manage something like dicts/lists in parallel,with the cython performance, remaining as much as possible in the Python domain; but i'm open to every advice.

Thanks

Python, efficient paralell operation using a dict

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

首先，为我的英语不太好而道歉。我想，我的问题很容易解释。result={}list_tuple=[(float,float,float),(float,float,float),(float,float,float)...]#200k tuplesthreshold=[float,float,float...] #ma...

Python，使用dict的高效并行操作-腾讯云开发者社区-腾讯云

问Python，使用dict的高效并行操作
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python，使用dict的高效并行操作EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python，使用dict的高效并行操作
EN