blocks|key|619528|text|我觉得np.isnan(np.min(X))应该按你说的做。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|619529|entityMap^0|3|J|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|K|8|@]|D|@]|E|$]]]|G|$]]

I think <code>np.isnan(np.min(X))</code> should do what you want.

blocks|key|619618|text|即使有一个公认的答案，我也想演示一下(在Vista上使用Python+2.7.2和Numpy+1.6.0+)：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|619619|In+[]:+x=+rand(1e5)
In+[]:+%25timeit+isnan(x.min())
10000+loops,+best+of+3:+200+us+per+loop
In+[]:+%25timeit+isnan(x.sum())
10000+loops,+best+of+3:+169+us+per+loop
In+[]:+%25timeit+isnan(dot(x,+x))
10000+loops,+best+of+3:+134+us+per+loop

In+[]:+x[5e4]=+NaN
In+[]:+%25timeit+isnan(x.min())
100+loops,+best+of+3:+4.47+ms+per+loop
In+[]:+%25timeit+isnan(x.sum())
100+loops,+best+of+3:+6.44+ms+per+loop
In+[]:+%25timeit+isnan(dot(x,+x))
10000+loops,+best+of+3:+138+us+per+loop|code-block|syntax|javascript|619620|因此，真正有效的方法可能在很大程度上取决于操作系统。无论如何，基于dot(.)的似乎是最稳定的。|offset|length|style|CODE|619621|entityMap^0|0|0|X|6|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@$I|R|J|S|K|L]]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Even there exist an accepted answer, I'll like to demonstrate the following (with Python 2.7.2 and Numpy 1.6.0 on Vista):

<pre><code>In []: x= rand(1e5)
In []: %timeit isnan(x.min())
10000 loops, best of 3: 200 us per loop
In []: %timeit isnan(x.sum())
10000 loops, best of 3: 169 us per loop
In []: %timeit isnan(dot(x, x))
10000 loops, best of 3: 134 us per loop

In []: x[5e4]= NaN
In []: %timeit isnan(x.min())
100 loops, best of 3: 4.47 ms per loop
In []: %timeit isnan(x.sum())
100 loops, best of 3: 6.44 ms per loop
In []: %timeit isnan(dot(x, x))
10000 loops, best of 3: 138 us per loop
</code></pre>

Thus, the really efficient way might be heavily dependent on the operating system. Anyway <code>dot(.)</code> based seems to be the most stable one.

blocks|key|619662|text|与此相关的问题是如何找到NaN的第一个匹配项。据我所知，这是最快的处理方法：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|619663|index+=+next((i+for+(i,n)+in+enumerate(iterable)+if+n!=n),+None)|code-block|syntax|javascript|619664|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Related to this is the question of how to find the first occurrence of NaN. This is the fastest way to handle that that I know of:

<pre><code>index = next((i for (i,n) in enumerate(iterable) if n!=n), None)
</code></pre>

blocks|key|835316|text|如果你熟悉numba，它允许创建一个快速短路(一旦找到NaN就停止)功能：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|835317|import+numba+as+nb
import+math

@nb.njit
def+anynan(array):
++++array+=+array.ravel()
++++for+i+in+range(array.size):
++++++++if+math.isnan(array[i]):
++++++++++++return+True
++++return+False|code-block|syntax|javascript|835318|如果没有NaN，函数实际上可能比np.min慢，我认为这是因为np.min对大型数组使用了多进程：|style|CODE|835319|import+numpy+as+np
array+=+np.random.random(2000000)

%25timeit+anynan(array)++++++++++#+100+loops,+best+of+3:+2.21+ms+per+loop
%25timeit+np.isnan(array.sum())++#+100+loops,+best+of+3:+4.45+ms+per+loop
%25timeit+np.isnan(array.min())++#+1000+loops,+best+of+3:+1.64+ms+per+loop|835320|但是如果数组中有一个NaN，特别是如果它的位置是低索引，那么它的速度要快得多：|835321|array+=+np.random.random(2000000)
array[100]+=+np.nan

%25timeit+anynan(array)++++++++++#+1000000+loops,+best+of+3:+1.93+µs+per+loop
%25timeit+np.isnan(array.sum())++#+100+loops,+best+of+3:+4.57+ms+per+loop
%25timeit+np.isnan(array.min())++#+1000+loops,+best+of+3:+1.65+ms+per+loop|835322|使用Cython或C扩展可以实现类似的结果，这些扩展稍微复杂一些(或者像bottleneck.anynan一样容易获得)，但最终与我的anynan函数的功能相同。|835323|entityMap|0|LINK|mutability|MUTABLE|url|/questions/tagged/numba|1|https://github.com/kwgoodman/bottleneck^0|5|5|0|0|0|4|3|G|6|V|6|0|0|0|0|10|H|1V|6|10|H|1|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@$A|15|B|16|1|17]]|C|$]]|$1|D|3|E|5|F|7|18|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|19|8|@$A|1A|B|1B|K|L]|$A|1C|B|1D|K|L]|$A|1E|B|1F|K|L]]|9|@]|C|$]]|$1|M|3|N|5|F|7|1G|8|@]|9|@]|C|$G|H]]|$1|O|3|P|5|6|7|1H|8|@]|9|@]|C|$]]|$1|Q|3|R|5|F|7|1I|8|@]|9|@]|C|$G|H]]|$1|S|3|T|5|6|7|1J|8|@$A|1K|B|1L|K|L]|$A|1M|B|1N|K|L]]|9|@$A|1O|B|1P|1|1Q]]|C|$]]|$1|U|3|-4|5|6|7|1R|8|@]|9|@]|C|$]]]|V|$W|$5|X|Y|Z|C|$10|11]]|12|$5|X|Y|Z|C|$10|13]]]]

If you're comfortable with <a href="/questions/tagged/numba" class="post-tag" title="show questions tagged &#39;numba&#39;" rel="tag">numba</a> it allows to create a fast short-circuit (stops as soon as a NaN is found) function:

<pre><code>import numba as nb
import math

@nb.njit
def anynan(array):
 array = array.ravel()
 for i in range(array.size):
 if math.isnan(array[i]):
 return True
 return False
</code></pre>

If there is no <code>NaN</code> the function might actually be slower than <code>np.min</code>, I think that's because <code>np.min</code> uses multiprocessing for large arrays:

<pre><code>import numpy as np
array = np.random.random(2000000)

%timeit anynan(array) # 100 loops, best of 3: 2.21 ms per loop
%timeit np.isnan(array.sum()) # 100 loops, best of 3: 4.45 ms per loop
%timeit np.isnan(array.min()) # 1000 loops, best of 3: 1.64 ms per loop
</code></pre>

But in case there is a NaN in the array, especially if it's position is at low indices, then it's much faster:

<pre><code>array = np.random.random(2000000)
array[100] = np.nan

%timeit anynan(array) # 1000000 loops, best of 3: 1.93 µs per loop
%timeit np.isnan(array.sum()) # 100 loops, best of 3: 4.57 ms per loop
%timeit np.isnan(array.min()) # 1000 loops, best of 3: 1.65 ms per loop
</code></pre>

Similar results may be achieved with Cython or a C extension, these are a bit more complicated (or easily avaiable as <a href="https://github.com/kwgoodman/bottleneck" rel="nofollow noreferrer"><code>bottleneck.anynan</code></a>) but ultimatly do the same as my <code>anynan</code> function.

blocks|key|835502|text|除了@nico-schlömer和@mseifert的回答之外，我还计算了提前停止的numba测试has_nan的性能，并与解析完整数组的一些函数进行了比较。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|835503|在我的机器上，对于没有nans的数组，盈亏平衡发生在大约10%5E4个元素。|835504|​|835505|📷|atomic|835506|835507|import+perfplot
import+numpy+as+np
import+numba
import+math

def+min(a):
++++return+np.isnan(np.min(a))

def+dot(a):
++++return+np.isnan(np.dot(a,+a))

def+einsum(a):
++++return+np.isnan(np.einsum("i->",+a))

@numba.njit
def+has_nan(a):
++++for+i+in+range(a.size+-+1):
++++++++if+math.isnan(a[i]):
++++++++++++return+True
++++return+False


def+array_with_missing_values(n,+p):
++++"""+Return+array+of+size+n,++p+:+nans+(+%25+of+array+length+)
++++Ex+:+n=1e6,+p=1+:+1e4+nan+assigned+at+random+positions+"""
++++a+=+np.random.rand(n)
++++p+=+np.random.randint(0,+len(a),+int(p*len(a)/100))
++++a[p]+=+np.nan
++++return+a


#%25%25
perfplot.show(
++++setup=lambda+n:+array_with_missing_values(n,+0),
++++kernels=[min,+dot,+has_nan],
++++n_range=[2+**+k+for+k+in+range(20)],
++++logx=True,
++++logy=True,
++++xlabel="len(a)",
)|code-block|syntax|javascript|835508|如果阵列具有nans，会发生什么情况？我调查了阵列的nan覆盖的影响。|835509|对于长度为1,000,000的数组，如果数组中有大约10%5E-3%25的nans+(因此大约有10个nans)，则has_nan会成为更好的选择。|835510|835511|835512|835513|#%25%25
N+=+1000000++#+100000
perfplot.show(
++++setup=lambda+p:+array_with_missing_values(N,+p),
++++kernels=[min,+dot,+has_nan],
++++n_range=np.array([2+**+k+for+k+in+range(20)])+/+2**20+*+0.01,+
++++logy=True,
++++xlabel=f"%25+of+nan+in+array+(N+=+{N})",
)|835514|如果在您的应用程序中，大多数阵列都有nan，而您正在寻找没有的阵列，那么has_nan是最好的方法。否则，dot似乎是最好的选择。|835515|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/2a3559925b458472c1639eb0b04c90fa.png|imageAlt|1|https://ask.qcloudimg.com/http-save/yehe-900000/1da35e05a452551ab56a2183415acb21.png^0|1D|7|0|0|0|0|1|0|0|0|0|0|1I|7|0|0|0|1|1|0|0|0|I|3|10|7|1H|3|0^^$0|@$1|2|3|4|5|6|7|1E|8|@$9|1F|A|1G|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1H|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|1I|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1J|8|@]|D|@$9|1K|A|1L|1|1M]]|E|$]]|$1|M|3|I|5|6|7|1N|8|@]|D|@]|E|$]]|$1|N|3|O|5|P|7|1O|8|@]|D|@]|E|$Q|R]]|$1|S|3|T|5|6|7|1P|8|@]|D|@]|E|$]]|$1|U|3|V|5|6|7|1Q|8|@$9|1R|A|1S|B|C]]|D|@]|E|$]]|$1|W|3|I|5|6|7|1T|8|@]|D|@]|E|$]]|$1|X|3|K|5|L|7|1U|8|@]|D|@$9|1V|A|1W|1|1X]]|E|$]]|$1|Y|3|I|5|6|7|1Y|8|@]|D|@]|E|$]]|$1|Z|3|10|5|P|7|1Z|8|@]|D|@]|E|$Q|R]]|$1|11|3|12|5|6|7|20|8|@$9|21|A|22|B|C]|$9|23|A|24|B|C]|$9|25|A|26|B|C]]|D|@]|E|$]]|$1|13|3|-4|5|6|7|27|8|@]|D|@]|E|$]]]|14|$15|$5|16|17|18|E|$19|1A|1B|-4]]|1C|$5|16|17|18|E|$19|1D|1B|-4]]]]

Adding to @nico-schlömer and @mseifert 's answers, I computed the performance of a numba-test <code>has_nan</code> with early stops, compared to some of the functions that will parse the full array.
On my machine, for an array without nans, the break-even happens for ~10^4 elements.
<a href="https://i.stack.imgur.com/MVp3y.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MVp3y.png" alt="has_nan_vs_full_parse_methods" /></a>
<pre class="lang-py prettyprint-override"><code>
import perfplot
import numpy as np
import numba
import math

def min(a):
 return np.isnan(np.min(a))

def dot(a):
 return np.isnan(np.dot(a, a))

def einsum(a):
 return np.isnan(np.einsum(&quot;i-&gt;&quot;, a))

@numba.njit
def has_nan(a):
 for i in range(a.size - 1):
 if math.isnan(a[i]):
 return True
 return False


def array_with_missing_values(n, p):
 &quot;&quot;&quot; Return array of size n, p : nans ( % of array length )
 Ex : n=1e6, p=1 : 1e4 nan assigned at random positions &quot;&quot;&quot;
 a = np.random.rand(n)
 p = np.random.randint(0, len(a), int(p*len(a)/100))
 a[p] = np.nan
 return a


#%%
perfplot.show(
 setup=lambda n: array_with_missing_values(n, 0),
 kernels=[min, dot, has_nan],
 n_range=[2 ** k for k in range(20)],
 logx=True,
 logy=True,
 xlabel=&quot;len(a)&quot;,
)

</code></pre>
What happens if the array has nans ? I investigated the impact of the nan-coverage of the array.
For arrays of length 1,000,000, <code>has_nan</code> becomes a better option is there are ~10^-3 % nans (so ~10 nans) in the array.
<a href="https://i.stack.imgur.com/AZgzg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AZgzg.png" alt="impact of nan-coverage of array" /></a>
<pre><code>
#%%
N = 1000000 # 100000
perfplot.show(
 setup=lambda p: array_with_missing_values(N, p),
 kernels=[min, dot, has_nan],
 n_range=np.array([2 ** k for k in range(20)]) / 2**20 * 0.01, 
 logy=True,
 xlabel=f&quot;% of nan in array (N = {N})&quot;,
)
</code></pre>
If in your application most arrays have <code>nan</code> and you're looking for ones without, then <code>has_nan</code> is the best approach.
Else; <code>dot</code> seems to be the best option.

I'm looking for the fastest way to check for the occurrence of NaN (<code>np.nan</code>) in a NumPy array <code>X</code>. <code>np.isnan(X)</code> is out of the question, since it builds a boolean array of shape <code>X.shape</code>, which is potentially gigantic.

I tried <code>np.nan in X</code>, but that seems not to work because <code>np.nan != np.nan</code>. Is there a fast and memory-efficient way to do this at all?

(To those who would ask "how gigantic": I can't tell. This is input validation for library code.)

Fast check for NaN in NumPy

Python

我正在寻找最快的方法来检查NaN (np.nan)在NumPy数组X中的出现情况。np.isnan(X)是不可能的，因为它构建了一个形状为X.shape的布尔数组，这可能是巨大的。我试过np.nan in X，但似乎不起作用，因为np.nan != np.nan。有没有一种既快速又节省内存的方法来做到这一点？(对于那些...

问NumPy中NaN的快速检查
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问NumPy中NaN的快速检查EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问NumPy中NaN的快速检查
EN