blocks|key|629527|text|我是hdf5的铁杆粉丝，因为它能存储大的数值数组。在python中有两个处理hdf5的选项：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|629528|http://www.pytables.org/|offset|length|629529|http://www.h5py.org/|629530|这两种方法都可以有效地与numpy数组一起使用。|629531|entityMap|0|LINK|mutability|MUTABLE|url|1^0|0|0|O|0|0|0|K|1|0|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|S|8|@]|9|@$D|T|E|U|1|V]]|A|$]]|$1|F|3|G|5|6|7|W|8|@]|9|@$D|X|E|Y|1|Z]]|A|$]]|$1|H|3|I|5|6|7|10|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|11|8|@]|9|@]|A|$]]]|K|$L|$5|M|N|O|A|$P|C]]|Q|$5|M|N|O|A|$P|G]]]]

I'm a big fan of hdf5 for storing large numpy arrays. There are two options for dealing with hdf5 in python:

<a href="http://www.pytables.org/" rel="noreferrer">http://www.pytables.org/</a>

<a href="http://www.h5py.org/" rel="noreferrer">http://www.h5py.org/</a>

Both are designed to work with numpy arrays efficiently.

blocks|key|414395|text|savez()将数据保存为zip文件，压缩和解压缩该文件可能需要一些时间。您可以使用save()+&+load()函数：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|414396|f+=+file("tmp.bin","wb")
np.save(f,a)
np.save(f,b)
np.save(f,c)
f.close()

f+=+file("tmp.bin","rb")
aa+=+np.load(f)
bb+=+np.load(f)
cc+=+np.load(f)
f.close()|code-block|syntax|javascript|414397|要将多个数组保存到一个文件中，只需先打开该文件，然后按顺序保存或加载这些数组。|414398|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

savez() save data in a zip file, It may take some time to zip &amp; unzip the file. You can use save() &amp; load() function:

<pre><code>f = file("tmp.bin","wb")
np.save(f,a)
np.save(f,b)
np.save(f,c)
f.close()

f = file("tmp.bin","rb")
aa = np.load(f)
bb = np.load(f)
cc = np.load(f)
f.close()
</code></pre>

To save multiple arrays in one file, you just need to open the file first, and then save or load the arrays in sequence.

blocks|key|629617|text|另一种有效地存储numpy数组的方法是Bloscpack|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|629618|#!/usr/bin/python
import+numpy+as+np
import+bloscpack+as+bp
import+time

n+=+10000000

a+=+np.arange(n)
b+=+np.arange(n)+*+10
c+=+np.arange(n)+*+-0.5
tsizeMB+=+sum(i.size*i.itemsize+for+i+in+(a,b,c))+/+2**20.

blosc_args+=+bp.DEFAULT_BLOSC_ARGS
blosc_args['clevel']+=+6
t+=+time.time()
bp.pack_ndarray_file(a,+'a.blp',+blosc_args=blosc_args)
bp.pack_ndarray_file(b,+'b.blp',+blosc_args=blosc_args)
bp.pack_ndarray_file(c,+'c.blp',+blosc_args=blosc_args)
t1+=+time.time()+-+t
print+"store+time+=+%25.2f+(%25.2f+MB/s)"+%25+(t1,+tsizeMB+/+t1)

t+=+time.time()
a1+=+bp.unpack_ndarray_file('a.blp')
b1+=+bp.unpack_ndarray_file('b.blp')
c1+=+bp.unpack_ndarray_file('c.blp')
t1+=+time.time()+-+t
print+"loading+time+=+%25.2f+(%25.2f+MB/s)"+%25+(t1,+tsizeMB+/+t1)|code-block|syntax|javascript|629619|我的笔记本电脑(一个相对较旧的带有Core2处理器的MacBook+Air+)的输出：|629620|$+python+store-blpk.py
store+time+=+0.19+(1216.45+MB/s)
loading+time+=+0.25+(898.08+MB/s)|629621|这意味着它可以非常快地存储，即瓶颈通常是磁盘。然而，由于压缩比在这里相当好，所以有效速度乘以压缩比。以下是这76+MB阵列的大小：|629622|$+ll+-h+*.blp
-rw-r--r--++1+faltet++staff+++921K+Mar++6+13:50+a.blp
-rw-r--r--++1+faltet++staff+++2.2M+Mar++6+13:50+b.blp
-rw-r--r--++1+faltet++staff+++1.4M+Mar++6+13:50+c.blp|629623|请注意，Blosc压缩机的使用是实现这一目标的基础。相同的脚本，但使用'clevel‘=0(即禁用压缩)：|629624|$+python+bench/store-blpk.py
store+time+=+3.36+(68.04+MB/s)
loading+time+=+2.61+(87.80+MB/s)|629625|显然是磁盘性能的瓶颈。|629626|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/Blosc/bloscpack|1|http://www.blosc.org/^0|J|9|0|0|0|0|0|0|0|4|5|1|0|0|0^^$0|@$1|2|3|4|5|6|7|16|8|@]|9|@$A|17|B|18|1|19]]|C|$]]|$1|D|3|E|5|F|7|1A|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|1B|8|@]|9|@]|C|$]]|$1|K|3|L|5|F|7|1C|8|@]|9|@]|C|$G|H]]|$1|M|3|N|5|6|7|1D|8|@]|9|@]|C|$]]|$1|O|3|P|5|F|7|1E|8|@]|9|@]|C|$G|H]]|$1|Q|3|R|5|6|7|1F|8|@]|9|@$A|1G|B|1H|1|1I]]|C|$]]|$1|S|3|T|5|F|7|1J|8|@]|9|@]|C|$G|H]]|$1|U|3|V|5|6|7|1K|8|@]|9|@]|C|$]]|$1|W|3|-4|5|6|7|1L|8|@]|9|@]|C|$]]]|X|$Y|$5|Z|10|11|C|$12|13]]|14|$5|Z|10|11|C|$12|15]]]]

Another possibility to store numpy arrays efficiently is <a href="https://github.com/Blosc/bloscpack" rel="noreferrer">Bloscpack</a>:

<pre><code>#!/usr/bin/python
import numpy as np
import bloscpack as bp
import time

n = 10000000

a = np.arange(n)
b = np.arange(n) * 10
c = np.arange(n) * -0.5
tsizeMB = sum(i.size*i.itemsize for i in (a,b,c)) / 2**20.

blosc_args = bp.DEFAULT_BLOSC_ARGS
blosc_args['clevel'] = 6
t = time.time()
bp.pack_ndarray_file(a, 'a.blp', blosc_args=blosc_args)
bp.pack_ndarray_file(b, 'b.blp', blosc_args=blosc_args)
bp.pack_ndarray_file(c, 'c.blp', blosc_args=blosc_args)
t1 = time.time() - t
print "store time = %.2f (%.2f MB/s)" % (t1, tsizeMB / t1)

t = time.time()
a1 = bp.unpack_ndarray_file('a.blp')
b1 = bp.unpack_ndarray_file('b.blp')
c1 = bp.unpack_ndarray_file('c.blp')
t1 = time.time() - t
print "loading time = %.2f (%.2f MB/s)" % (t1, tsizeMB / t1)
</code></pre>

and the output for my laptop (a relatively old MacBook Air with a Core2 processor):

<pre><code>$ python store-blpk.py
store time = 0.19 (1216.45 MB/s)
loading time = 0.25 (898.08 MB/s)
</code></pre>

that means that it can store really fast, i.e. the bottleneck is typically the disk. However, as the compression ratios are pretty good here, the effective speed is multiplied by the compression ratios. Here are the sizes for these 76 MB arrays:

<pre><code>$ ll -h *.blp
-rw-r--r-- 1 faltet staff 921K Mar 6 13:50 a.blp
-rw-r--r-- 1 faltet staff 2.2M Mar 6 13:50 b.blp
-rw-r--r-- 1 faltet staff 1.4M Mar 6 13:50 c.blp
</code></pre>

Please note that the use of the <a href="http://www.blosc.org" rel="noreferrer">Blosc</a> compressor is fundamental for achieving this. The same script but using 'clevel' = 0 (i.e. disabling compression):

<pre><code>$ python bench/store-blpk.py
store time = 3.36 (68.04 MB/s)
loading time = 2.61 (87.80 MB/s)
</code></pre>

is clearly bottlenecked by the disk performance.

blocks|key|629653|text|查找时间很慢，因为当您调用load方法时，使用mmap时不会将数组内容加载到内存中。当需要特定数据时，数据被延迟加载。在你的例子中，这发生在lookup中。但是第二次查找不会那么慢。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|629654|当你有一个很大的数组时，这是mmap的一个很好的特性，你不需要把整个数据加载到内存中。|629655|为了解决您可以使用joblib的问题，您可以使用joblib.dump转储任何您想要的对象，甚至可以使用两个或更多的numpy+arrays，请参见示例|629656|firstArray+=+np.arange(100)
secondArray+=+np.arange(50)
#+I+will+put+two+arrays+in+dictionary+and+save+to+one+file
my_dict+=+{'first'+:+firstArray,+'second'+:+secondArray}
joblib.dump(my_dict,+'file_name.dat')|code-block|syntax|javascript|629657|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/joblib/joblib^0|D|4|N|4|0|E|4|0|O|B|1M|C|9|6|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|11|8|@$9|12|A|13|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|14|8|@$9|15|A|16|B|C]|$9|17|A|18|B|C]]|D|@$9|19|A|1A|1|1B]]|E|$]]|$1|J|3|K|5|L|7|1C|8|@]|D|@]|E|$M|N]]|$1|O|3|-4|5|6|7|1D|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]]]

The lookup time is slow because when you use <code>mmap</code> to does not load content of array to memory when you invoke <code>load</code> method. Data is lazy loaded when particular data is needed. 
And this happens in lookup in your case. But second lookup won`t be so slow.

This is nice feature of <code>mmap</code> when you have a big array you do not have to load whole data into memory.

To solve your can use <a href="https://github.com/joblib/joblib" rel="nofollow noreferrer">joblib</a> you can dump any object you want using <code>joblib.dump</code> even two or more <code>numpy arrays</code>, see the example

<pre><code>firstArray = np.arange(100)
secondArray = np.arange(50)
# I will put two arrays in dictionary and save to one file
my_dict = {'first' : firstArray, 'second' : secondArray}
joblib.dump(my_dict, 'file_name.dat')
</code></pre>

blocks|key|414531|text|我比较了存储numpy数组的几种方法的性能(空间和时间)。它们中很少有支持每个文件多个数组的，但也许它还是有用的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|414532|​|414533|📷|atomic|offset|length|414534|414535|Npy和二进制文件对于密集数据来说既快又小。如果数据是稀疏的或非常结构化的，您可能希望使用npz进行压缩，这将节省大量空间，但会消耗一些加载时间。|414536|如果可移植性是个问题，那么二进制比npy更好。如果人类的可读性很重要，那么你将不得不牺牲大量的性能，但使用csv可以很好地实现这一点(当然，csv也非常可移植)。|414537|更多细节和代码请访问the+github+repo。|414538|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/76750e286bb19efae160f98d9b993b3d.png|imageAlt|1|LINK|MUTABLE|url|https://github.com/mverleg/array_storage_benchmark^0|0|0|0|1|0|0|0|0|0|A|F|1|0^^$0|@$1|2|3|4|5|6|7|13|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|14|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|15|8|@]|9|@$G|16|H|17|1|18]]|A|$]]|$1|I|3|C|5|6|7|19|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|1A|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1B|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|1C|8|@]|9|@$G|1D|H|1E|1|1F]]|A|$]]|$1|P|3|-4|5|6|7|1G|8|@]|9|@]|A|$]]]|Q|$R|$5|S|T|U|A|$V|W|X|-4]]|Y|$5|Z|T|10|A|$11|12]]]]

I've compared performance (space and time) for a number of ways to store numpy arrays. Few of them support multiple arrays per file, but perhaps it's useful anyway.

<a href="https://i.stack.imgur.com/4d6yo.png"><img src="https://i.stack.imgur.com/4d6yo.png" alt="benchmark for numpy array storage"></a>

Npy and binary files are both really fast and small for dense data. If the data is sparse or very structured, you might want to use npz with compression, which'll save a lot of space but cost some load time.

If portability is an issue, binary is better than npy. If human readability is important, then you'll have to sacrifice a lot of performance, but it can be achieved fairly well using csv (which is also very portable of course).

More details and the code are available at <a href="https://github.com/mverleg/array_storage_benchmark">the github repo</a>.

blocks|key|414598|text|“最好”取决于你的目标是什么。正如其他人所说，二进制文件具有最大的可移植性，但问题是您需要了解数据是如何存储的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|414599|Darr以一种基于平面二进制和文本文件的自我记录的方式保存您的numpy数组。这最大化了广泛的可读性。它还自动包含了如何用各种数据科学语言读取数组的代码，比如numpy本身，也包括R、Matlab、Julia等。|offset|length|414600|披露:这个库是我写的。|414601|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/gbeckers/Darr^0|0|0|4|0|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@$D|R|E|S|1|T]]|A|$]]|$1|F|3|G|5|6|7|U|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

'Best' depends on what your goal is. As others have said, a binary is maximally portable, but the problem is that you need to know about how the data is stored.
<a href="https://github.com/gbeckers/Darr" rel="nofollow noreferrer">Darr</a> saves your numpy array in a self-documented way based on flat binary and text files. This maximizes wide readability. It also automatically includes code on how to read your array in a variety of data science languages, such as numpy itself, but also R, Matlab, Julia etc.
Disclosure: I wrote the library.

I am looking for a fast way to preserve large numpy arrays. I want to save them to the disk in a binary format, then read them back into memory relatively fastly. cPickle is not fast enough, unfortunately. 

I found <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html#numpy.savez">numpy.savez</a> and <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html#numpy.load">numpy.load</a>. But the weird thing is, numpy.load loads a npy file into "memory-map". That means regular manipulating of arrays really slow. For example, something like this would be really slow: 

<pre><code>#!/usr/bin/python
import numpy as np;
import time; 
from tempfile import TemporaryFile

n = 10000000;

a = np.arange(n)
b = np.arange(n) * 10
c = np.arange(n) * -0.5

file = TemporaryFile()
np.savez(file,a = a, b = b, c = c);

file.seek(0)
t = time.time()
z = np.load(file)
print "loading time = ", time.time() - t

t = time.time()
aa = z['a']
bb = z['b']
cc = z['c']
print "assigning time = ", time.time() - t;
</code></pre>

more precisely, the first line will be really fast, but the remaining lines that assign the arrays to <code>obj</code> are ridiculously slow:

<pre><code>loading time = 0.000220775604248
assining time = 2.72940087318
</code></pre>

Is there any better way of preserving numpy arrays? Ideally, I want to be able to store multiple arrays in one file.

best way to preserve numpy arrays on disk

内存映射

Julia

我正在寻找一种快速的方法来保存大的numpy数组。我想将它们以二进制格式保存到磁盘上，然后相对较快地将它们读回内存。不幸的是，cPickle不够快。我找到和了。但奇怪的是，numpy.load将npy文件加载到“内存映射”中。这意味着常规的数组操作非常慢。例如，像这样的东西会非常慢：#!/usr/bin/pythoni...

问在磁盘上保留numpy数组的最佳方法
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在磁盘上保留numpy数组的最佳方法EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在磁盘上保留numpy数组的最佳方法
EN