首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >如何用python在zip中打开一个csv?

如何用python在zip中打开一个csv?
EN

Stack Overflow用户
提问于 2018-06-22 23:19:36
回答 1查看 960关注 0票数 2

我一直在使用一个用户定义的函数来打开包含在ZIP文件中的CSV文件,这对我来说运行得非常好。

How to scrape .csv files from a url, when they are saved in a .zip file in Python?

现在我试图打开一个包含在ZIP中的CSV文件,该文件包含在另一个ZIP中,并且遇到了一些问题。

我没有使用来自CSV的数据获得数据帧的预期输出,而是得到了这个错误:

代码语言:javascript
复制
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte

这在某种程度上是有意义的,因为我正在尝试用read_csv()打开一个压缩文件

代码语言:javascript
复制
import pandas as pd

def fetch_multi_csv_zip_from_url(url, filenames=(), *args, **kwargs):
    assert kwargs.get('compression') is None
    req = urlopen(url)
    zip_file = zipfile.ZipFile(BytesIO(req.read()))

    if filenames:
        names = zip_file.namelist()
        for filename in filenames:
            if filename not in names:
                raise ValueError(
                    'filename {} not in {}'.format(filename, names))
    else:
        filenames = zip_file.namelist()

    return {name: pd.read_csv(zip_file.open(name), *args, **kwargs)
            for name in filenames}

try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen
from io import BytesIO
import zipfile

final_links_list =['http://www.nemweb.com.au/REPORTS/ARCHIVE/Dispatch_SCADA/PUBLIC_DISPATCHSCADA_20170523.zip', 'http://www.nemweb.com.au/REPORTS/ARCHIVE/Dispatch_SCADA/PUBLIC_DISPATCHSCADA_20170524.zip']
l = len(final_links_list)

for j in range(0,l):
    print(j)
    dfs = fetch_multi_csv_zip_from_url(final_links_list[j])

这是我一直在使用的代码,我想我必须更改以以下开头的代码行:

代码语言:javascript
复制
return {name: pd.read_csv(zip_file.open(name)

因为它不再返回csv文件,而是返回zip文件。

EN

回答 1

Stack Overflow用户

回答已采纳

发布于 2018-06-23 23:24:20

这可以通过一些递归来完成。如果发现ZIP中的文件是ZIP文件,则进行递归调用以解压缩CSV文件:

代码语言:javascript
复制
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

from io import BytesIO
import zipfile

import pandas as pd

# Dictionary holding all the dataframes from all zip/zip/csvs
dfs = {}


def zip_to_dfs(data):
    zip_file = zipfile.ZipFile(BytesIO(data))

    for name in zip_file.namelist():
        if name.lower().endswith('.csv'):
            dfs[name] = pd.read_csv(zip_file.open(name))
        elif name.lower().endswith('.zip'):
            zip_to_dfs(zip_file.open(name).read())


def get_zip_data_from_url(url):
    req = urlopen(url)
    zip_to_dfs(req.read())


final_links_list = [
    'http://www.nemweb.com.au/REPORTS/ARCHIVE/Dispatch_SCADA/PUBLIC_DISPATCHSCADA_20170523.zip', 
    'http://www.nemweb.com.au/REPORTS/ARCHIVE/Dispatch_SCADA/PUBLIC_DISPATCHSCADA_20170524.zip']

for link in final_links_list:
    print(link)
    get_zip_data_from_url(link)

# Display the first couple of dataframes    
for name, df in sorted(dfs.items())[:2]:
    print('\n', name, '\n')
    print(df)

这将显示以下内容:

代码语言:javascript
复制
http://www.nemweb.com.au/REPORTS/ARCHIVE/Dispatch_SCADA/PUBLIC_DISPATCHSCADA_20170524.zip

 PUBLIC_DISPATCHSCADA_201705240010_0000000283857084.CSV 

     C     NEMP.WORLD DISPATCHSCADA  AEMO               PUBLIC 2017/05/24  \
0    I       DISPATCH    UNIT_SCADA   1.0       SETTLEMENTDATE       DUID   
1    D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:10:00    BARCSF1   
2    D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:10:00   BUTLERSG   
..  ..            ...           ...   ...                  ...        ...   
263  D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:10:00      YWPS3   
264  D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:10:00      YWPS4   
265  C  END OF REPORT           267   NaN                  NaN        NaN   

       00:05:08  0000000283857084  DISPATCHSCADA.1  0000000283857078  
0    SCADAVALUE               NaN              NaN               NaN  
1             0               NaN              NaN               NaN  
2      8.299998               NaN              NaN               NaN  
..          ...               ...              ...               ...  
263  388.745570               NaN              NaN               NaN  
264  391.568360               NaN              NaN               NaN  
265         NaN               NaN              NaN               NaN  

[266 rows x 10 columns]

 PUBLIC_DISPATCHSCADA_201705240015_0000000283857169.CSV 

     C     NEMP.WORLD DISPATCHSCADA  AEMO               PUBLIC 2017/05/24  \
0    I       DISPATCH    UNIT_SCADA   1.0       SETTLEMENTDATE       DUID   
1    D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:15:00    BARCSF1   
2    D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:15:00   BUTLERSG   
..  ..            ...           ...   ...                  ...        ...   
263  D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:15:00      YWPS3   
264  D       DISPATCH    UNIT_SCADA   1.0  2017/05/24 00:15:00      YWPS4   
265  C  END OF REPORT           267   NaN                  NaN        NaN   

       00:10:08  0000000283857169  DISPATCHSCADA.1  0000000283857163  
0    SCADAVALUE               NaN              NaN               NaN  
1             0               NaN              NaN               NaN  
2      8.299998               NaN              NaN               NaN  
..          ...               ...              ...               ...  
263  386.205080               NaN              NaN               NaN  
264  389.592410               NaN              NaN               NaN  
265         NaN               NaN              NaN               NaN  

[266 rows x 10 columns]
票数 3
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/50991084

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档