首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >在python中使用pandas构建嵌套结构之外的数据框

在python中使用pandas构建嵌套结构之外的数据框
EN

Stack Overflow用户
提问于 2018-07-13 22:52:23
回答 2查看 404关注 0票数 1

我想用一个有点复杂的数据集来实现机器学习。我想和熊猫一起工作,然后使用短剧学习中的一些内置模型。

数据外观是在JSON文件中给出的,示例如下:

代码语言:javascript
复制
{
  "demo_Profile": {
    "sex": "male",
    "age": 98,
    "height": 160,
    "weight": 139,
    "bmi": 5,
    "someinfo1": [
      "some_more_info1"
    ],
    "someinfo2": [
      "some_more_inf2"
    ],
    "someinfo3": [
      "some_more_info3"
    ],
  },
  "event": {
    "info_personal": {
      "info1": 219.59,
      "info2": 129.18,
      "info3": 41.15,
      "info4": 94.19,
    },
    "symptoms": [
      {
        "name": "name1",
        "socrates": {
          "associations": [
            "associations1"
          ],
          "onsetType": "onsetType1",
          "timeCourse": "timeCourse1"
        }
      },
      {
        "name": "name2",
        "socrates": {
          "timeCourse": "timeCourse2"
        }
      },
      {
        "name": "name3",
        "socrates": {
          "onsetType": "onsetType2"
        }
      },
      {
        "name": "name4",
        "socrates": {
          "onsetType": "onsetType3"
        }
      },
      {
        "name": "name5",
        "socrates": {
          "associations": [
            "associations2"
          ]
        }
      }
    ],
    "labs": [
      {
        "name": "name1 ",
        "value": "valuelab"
      }
    ]
  }
}

我想创建一个考虑这种“嵌套数据”的pandas数据框架,但我不知道如何构建一个除了"singles参数“之外还要考虑”嵌套参数“的数据框架。

例如,我不知道如何将包含“单个参数”的"demo_Profile“与症状合并,症状是字典的列表,在相同的情况下是单值,在其他情况下是列表。

有人知道解决这个问题的方法吗?

编辑*

上面显示的JSON只是一个示例,在其他情况下,列表中的值的数量以及症状的数量将是不同的。因此,上面显示的示例并不适用于所有情况。

EN

回答 2

Stack Overflow用户

回答已采纳

发布于 2018-07-14 01:24:04

平面化json数据的一种快速而简单的方法是使用flatten_json包,该包可以通过pip安装。

代码语言:javascript
复制
pip install flatten_json

我希望您有许多条目的列表,这些条目看起来与您提供的条目相似。因此,下面的代码将为您提供所需的结果:

代码语言:javascript
复制
import pandas as pd
from flatten_json import flatten

json_data = [{...patient1...}, {patient2...}, ...]

flattened = (flatten(entry) for entry in json_data)
df = pd.DataFrame(flattened)

在扁平化的数据中,列表条目以数字为后缀(我在"labs“列表中添加了另一个患者条目):

代码语言:javascript
复制
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| index   demo_Profile_age  demo_Profile_bmi  demo_Profile_height demo_Profile_sex demo_Profile_someinfo1_0 demo_Profile_someinfo2_0 demo_Profile_someinfo3_0  demo_Profile_weight  event_info_personal_info1  event_info_personal_info2  event_info_personal_info3  event_info_personal_info4 event_labs_0_name event_labs_0_value event_labs_1_name event_labs_1_value event_symptoms_0_name event_symptoms_0_socrates_associations_0 event_symptoms_0_socrates_onsetType event_symptoms_0_socrates_timeCourse event_symptoms_1_name event_symptoms_1_socrates_timeCourse event_symptoms_2_name event_symptoms_2_socrates_onsetType event_symptoms_3_name event_symptoms_3_socrates_onsetType event_symptoms_4_name event_symptoms_4_socrates_associations_0 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 0                98                 5                  160             male          some_more_info1           some_more_inf2          some_more_info3                  139                     219.59                     129.18                      41.15                      94.19            name1            valuelab               NaN                NaN                 name1                            associations1                          onsetType1                          timeCourse1                 name2                          timeCourse2                 name3                          onsetType2                 name4                          onsetType3                 name5                            associations2      |
| 1                98                 5                  160             male          some_more_info1           some_more_inf2          some_more_info3                  139                     219.59                     129.18                      41.15                      94.19            name1            valuelab            name2          valuelabr2                 name1                            associations1                          onsetType1                          timeCourse1                 name2                          timeCourse2                 name3                          onsetType2                 name4                          onsetType3                 name5                            associations2      |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

flatten方法包含用于删除不需要的列或前缀的附加参数。

注意:虽然这种方法提供了所需的扁平化DataFrame,但我预计在将数据集提供给机器学习算法时,您将遇到其他问题,这取决于您的预测目标是什么,以及您希望如何将数据编码为特征。

票数 1
EN

Stack Overflow用户

发布于 2018-07-14 01:53:30

考虑一下熊猫的json_normalize。但是,因为还有更深的嵌套,所以可以考虑单独处理数据,然后在“规范化”列上连接起来并向前填充。

代码语言:javascript
复制
import json
import pandas as pd
from pandas.io.json import json_normalize

with open('myfile.json', 'r') as f:
    data = json.loads(f.read()) 

final_df = pd.concat([json_normalize(data['demo_Profile']), 
                      json_normalize(data['event']['symptoms']), 
                      json_normalize(data['event']['info_personal']), 
                      json_normalize(data['event']['labs'])], axis=1)

# FLATTEN NESTED LISTS
n_list = ['someinfo1', 'someinfo2', 'someinfo3', 'socrates.associations']

final_df[n_list] = final_df[n_list].apply(lambda col: 
                     col.apply(lambda x: x  if pd.isnull(x) else x[0]))

# FILLING FORWARD
norm_list = ['age', 'bmi', 'height', 'weight', 'sex', 'someinfo1', 'someinfo2', 'someinfo3', 
             'info1', 'info2', 'info3', 'info4', 'name', 'value']

final_df[norm_list] = final_df[norm_list].ffill()  

输出

代码语言:javascript
复制
print(final_df)

#     age  bmi  height   sex        someinfo1       someinfo2        someinfo3  weight   name socrates.associations socrates.onsetType socrates.timeCourse   info1   info2  info3  info4    name     value
# 0  98.0  5.0   160.0  male  some_more_info1  some_more_inf2  some_more_info3   139.0  name1         associations1         onsetType1         timeCourse1  219.59  129.18  41.15  94.19  name1   valuelab
# 1  98.0  5.0   160.0  male  some_more_info1  some_more_inf2  some_more_info3   139.0  name2                   NaN                NaN         timeCourse2  219.59  129.18  41.15  94.19  name1   valuelab
# 2  98.0  5.0   160.0  male  some_more_info1  some_more_inf2  some_more_info3   139.0  name3                   NaN         onsetType2                 NaN  219.59  129.18  41.15  94.19  name1   valuelab
# 3  98.0  5.0   160.0  male  some_more_info1  some_more_inf2  some_more_info3   139.0  name4                   NaN         onsetType3                 NaN  219.59  129.18  41.15  94.19  name1   valuelab
# 4  98.0  5.0   160.0  male  some_more_info1  some_more_inf2  some_more_info3   139.0  name5         associations2                NaN                 NaN  219.59  129.18  41.15  94.19  name1   valuelab
票数 2
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/51327847

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档