blocks|key|81081|text|泡菜效果很好！|type|unstyled|depth|inlineStyleRanges|entityRanges|data|81082|import+pandas+as+pd
df.to_pickle('123.pkl')++++#to+save+the+dataframe,+df+to+123.pkl
df1+=+pd.read_pickle('123.pkl')+#to+load+123.pkl+back+to+the+dataframe+df|code-block|syntax|javascript|81083|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Pickle works good!

<pre><code>import pandas as pd
df.to_pickle('123.pkl') #to save the dataframe, df to 123.pkl
df1 = pd.read_pickle('123.pkl') #to load 123.pkl back to the dataframe df
</code></pre>

blocks|key|298284|text|Pandas+DataFrames具有to_pickle函数，该函数对于保存DataFrame非常有用：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|298285|import+pandas+as+pd

a+=+pd.DataFrame({'A':[0,1,0,1,0],'B':[True,+True,+False,+False,+False]})
print+a
#++++A++++++B
#+0++0+++True
#+1++1+++True
#+2++0++False
#+3++1++False
#+4++0++False

a.to_pickle('my_file.pkl')

b+=+pd.read_pickle('my_file.pkl')
print+b
#++++A++++++B
#+0++0+++True
#+1++1+++True
#+2++0++False
#+3++1++False
#+4++0++False|code-block|syntax|javascript|298286|entityMap^0|J|9|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

Pandas DataFrames have the <code>to_pickle</code> function which is useful for saving a DataFrame:

<pre><code>import pandas as pd

a = pd.DataFrame({'A':[0,1,0,1,0],'B':[True, True, False, False, False]})
print a
# A B
# 0 0 True
# 1 1 True
# 2 0 False
# 3 1 False
# 4 0 False

a.to_pickle('my_file.pkl')

b = pd.read_pickle('my_file.pkl')
print b
# A B
# 0 0 True
# 1 1 True
# 2 0 False
# 3 1 False
# 4 0 False
</code></pre>

blocks|key|298312|text|Numpy文件格式对于数值数据来说非常快|type|unstyled|depth|inlineStyleRanges|entityRanges|data|298313|我更喜欢使用numpy文件，因为它们既快又容易操作。这是一个简单的基准测试，用于保存和加载包含1列1百万个点的数据帧。|298314|import+numpy+as+np
import+pandas+as+pd

num_dict+=+{'voltage':+np.random.rand(1000000)}
num_df+=+pd.DataFrame(num_dict)|code-block|syntax|javascript|298315|使用ipython的%25%25timeit魔术函数|offset|length|style|CODE|298316|%25%25timeit
with+open('num.npy',+'wb')+as+np_file:
++++np.save(np_file,+num_df)|298317|输出为|298318|100+loops,+best+of+3:+5.97+ms+per+loop|298319|要将数据加载回数据帧，请执行以下操作|298320|%25%25timeit
with+open('num.npy',+'rb')+as+np_file:
++++data+=+np.load(np_file)

data_df+=+pd.DataFrame(data)|298321|298322|100+loops,+best+of+3:+5.12+ms+per+loop|298323|还不错!|298324|缺点|298325|如果您使用python+2保存numpy文件，然后尝试使用python+3打开(反之亦然)，则会出现问题。|298326|entityMap^0|0|0|0|A|8|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|19|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1A|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|1B|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|1C|8|@$K|1D|L|1E|M|N]]|9|@]|A|$]]|$1|O|3|P|5|F|7|1F|8|@]|9|@]|A|$G|H]]|$1|Q|3|R|5|6|7|1G|8|@]|9|@]|A|$]]|$1|S|3|T|5|F|7|1H|8|@]|9|@]|A|$G|H]]|$1|U|3|V|5|6|7|1I|8|@]|9|@]|A|$]]|$1|W|3|X|5|F|7|1J|8|@]|9|@]|A|$G|H]]|$1|Y|3|R|5|6|7|1K|8|@]|9|@]|A|$]]|$1|Z|3|10|5|F|7|1L|8|@]|9|@]|A|$G|H]]|$1|11|3|12|5|6|7|1M|8|@]|9|@]|A|$]]|$1|13|3|14|5|6|7|1N|8|@]|9|@]|A|$]]|$1|15|3|16|5|6|7|1O|8|@]|9|@]|A|$]]|$1|17|3|-4|5|6|7|1P|8|@]|9|@]|A|$]]]|18|$]]

<h3>Numpy file formats are pretty fast for numerical data</h3>

I prefer to use numpy files since they're fast and easy to work with.
Here's a simple benchmark for saving and loading a dataframe with 1 column of 1million points.

<pre><code>import numpy as np
import pandas as pd

num_dict = {'voltage': np.random.rand(1000000)}
num_df = pd.DataFrame(num_dict)
</code></pre>

using ipython's <code>%%timeit</code> magic function

<pre><code>%%timeit
with open('num.npy', 'wb') as np_file:
 np.save(np_file, num_df)
</code></pre>

the output is 

<pre><code>100 loops, best of 3: 5.97 ms per loop
</code></pre>

to load the data back into a dataframe

<pre><code>%%timeit
with open('num.npy', 'rb') as np_file:
 data = np.load(np_file)

data_df = pd.DataFrame(data)
</code></pre>

the output is

<pre><code>100 loops, best of 3: 5.12 ms per loop
</code></pre>

NOT BAD!

<h3>CONS</h3>

There's a problem if you save the numpy file using python 2 and then try opening using python 3 (or vice versa).

blocks|key|298344|text|您可以使用羽化格式的文件。它的速度非常快。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|298345|df.to_feather('filename.ft')|code-block|syntax|javascript|298346|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can use feather format file. It is extremely fast.

<pre><code>df.to_feather('filename.ft')
</code></pre>

blocks|key|81178|text|Arctic是一个用于Pandas、numpy和其他数字数据的高性能数据存储。它位于MongoDB之上。对于OP来说，这可能有些过分了，但对于其他偶然发现这篇文章的人来说，值得一提的是|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|81179|entityMap|0|LINK|mutability|MUTABLE|url|https://pypi.org/project/arctic/^0|0|6|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

<a href="https://pypi.org/project/arctic/" rel="nofollow noreferrer">Arctic</a> is a high performance datastore for Pandas, numpy and other numeric data. It sits on top of MongoDB. Perhaps overkill for the OP, but worth mentioning for other folks stumbling across this post

Right now I'm importing a fairly large <code>CSV</code> as a dataframe every time I run the script. Is there a good solution for keeping that dataframe constantly available in between runs so I don't have to spend all that time waiting for the script to run?

How to reversibly store and load a Pandas dataframe to/from disk

数据存储

MongoDB

现在，我每次运行脚本时都会导入一个相当大的CSV作为数据帧。有没有一个好的解决方案可以让数据帧在两次运行之间持续可用，这样我就不必花费所有的时间来等待脚本运行？

问如何可逆地将Pandas数据帧存储到磁盘或从磁盘加载
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何可逆地将Pandas数据帧存储到磁盘或从磁盘加载EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何可逆地将Pandas数据帧存储到磁盘或从磁盘加载
EN