blocks|key|406425|text|您可以尝试使用gzip库：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|406426|import+gzip
f+=+gzip.open('Your+File',+'wb')
file_content+=+f.read()
print+(file_content)|code-block|syntax|javascript|406427|python:+read+lines+from+compressed+text+files|offset|length|406428|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/10566558/python-read-lines-from-compressed-text-files^0|0|0|0|19|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|U|8|@]|9|@$I|V|J|W|1|X]]|A|$]]|$1|K|3|-4|5|6|7|Y|8|@]|9|@]|A|$]]]|L|$M|$5|N|O|P|A|$Q|R]]]]

You can try using gzip library:

<pre><code>import gzip
f = gzip.open('Your File', 'wb')
file_content = f.read()
print (file_content)
</code></pre>

<a href="https://stackoverflow.com/questions/10566558/python-read-lines-from-compressed-text-files">python: read lines from compressed text files</a>

blocks|key|809616|text|gzip本质上是一种相当慢的压缩方法，而且(如您所说)不支持随机访问。这意味着，到达位置x的唯一方法是从头开始扫描整个文件，这就是为什么Dask在这种情况下不支持尝试并行化。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|809617|如果您至少想使用并行解析，那么最好的做法是首先解压缩整个文件，这样分块机制才有意义。您还可以将其拆分成几个文件，然后压缩每个文件，以便所需的总空间是相似的。|809618|请注意，在理论上，一些支持分块随机访问的压缩机制，但我们还没有找到任何具有足够社区支持的压缩机制来在Dask中实现它们。|809619|不过，最好的答案是以parquet或orc格式存储数据，这种格式具有内部压缩和分区功能。|809620|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

gzip is, inherently, a pretty slow compression method, and (as you say) does not support random access. This means, that the only way to get to position x is to scan through the file from the start, which is why Dask does not support trying to parallelise in this case.

Your best best, if you want to make use of parallel parsing at least, is first to decompress the whole file, so that the chunking mechanism does make sense. You could also break it into several files, and compress each one, so that the total space required is similar.

Note that, in theory, some compression mechanisms that support block-wise random access, but we have not found any with sufficient community support to implement them in Dask.

The best answer, though, is to store your data in parquet or orc format, which has internal compression and partitioning.

blocks|key|803860|text|一种选择是为python使用package+datatable：https://github.com/h2oai/datatable|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|803861|例如，使用函数fread，它可以比pandas+(到gzip)的读/写速度快得多|803862|import+datatable+as+dt
df+=+dt.fread('file.csv.gz')|code-block|syntax|javascript|803863|稍后，可以将其convert到pandas+dataframe：|803864|df1+=+df.to_pandas()|803865|目前，datatable仅为available+on+Linux/Mac。|803866|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/h2oai/datatable|1|https://datatable.readthedocs.io/en/latest/using-datatable.html#convert-a-frame|2|https://datatable.readthedocs.io/en/latest/install.html#install-on-mac-os-x-and-linux^0|W|Y|0|0|0|0|7|7|1|0|0|E|M|2|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@$A|13|B|14|1|15]]|C|$]]|$1|D|3|E|5|6|7|16|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|17|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|18|8|@]|9|@$A|19|B|1A|1|1B]]|C|$]]|$1|M|3|N|5|H|7|1C|8|@]|9|@]|C|$I|J]]|$1|O|3|P|5|6|7|1D|8|@]|9|@$A|1E|B|1F|1|1G]]|C|$]]|$1|Q|3|-4|5|6|7|1H|8|@]|9|@]|C|$]]]|R|$S|$5|T|U|V|C|$W|X]]|Y|$5|T|U|V|C|$W|Z]]|10|$5|T|U|V|C|$W|11]]]]

One option is to use package datatable for python:
<a href="https://github.com/h2oai/datatable" rel="nofollow noreferrer">https://github.com/h2oai/datatable</a>

It can read/write significantly faster than pandas (to gzip) using the function fread, for example

<pre><code>import datatable as dt
df = dt.fread('file.csv.gz')
</code></pre>

Later, one can <a href="https://datatable.readthedocs.io/en/latest/using-datatable.html#convert-a-frame" rel="nofollow noreferrer">convert</a> it to pandas dataframe:

<pre><code>df1 = df.to_pandas()
</code></pre>

Currently datatable is only <a href="https://datatable.readthedocs.io/en/latest/install.html#install-on-mac-os-x-and-linux" rel="nofollow noreferrer">available on Linux/Mac</a>.

I have couple (each of 3.5GB) of gzip files, as of now I am using Pandas to read those files, but it is very slow, I have tried Dask also, but it seems it does not support gzip file breaking. Is there any better way to quickly load these massive gzip files?

Dask and Pandas code:

<pre><code>df = dd.read_csv(r'file', sample = 200000000000,compression='gzip')
</code></pre>

I expect it to read the whole file as quickly as possible.

How to read compressed(.gz) file faster using Pandas/Dask?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有几个gzip文件(每个3.5 as )，到目前为止，我使用Pandas来读取这些文件，但它非常慢，我也尝试过Dask，但它似乎不支持gzip文件中断。有没有更好的方法来快速加载这些庞大的gzip文件？Dask和Pandas代码：df = dd.read_csv(r'file', sample = 200000000...

问如何使用Pandas/Dask更快地读取压缩(.gz)文件？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何使用Pandas/Dask更快地读取压缩(.gz)文件？EN