我的记录如下所示,我需要将其写入csv文件:
my_data={"data":[{"id":"xyz","type":"book","attributes":{"doc_type":"article","action":"cut"}}]}
它看起来像json,但是下一条记录是以"data"
而不是"data1"
开始的,这迫使我分别读取每条记录。然后,我使用eval()
将其转换为字典,以遍历特定路径的键和值,以获得所需的值。然后,我根据需要的键生成一个键和值的列表。然后,pd.dataframe()
将该列表转换为我知道如何转换为csv的数据帧。我的代码如下所示。但我相信有更好的方法可以做到这一点。我的规模很小。谢谢。
counter=1
k=[]
v=[]
res=[]
m=0
for line in f2:
jline=eval(line)
counter +=1
for items in jline:
k.append(jline[u'data'][0].keys())
v.append(jline[u'data'][0].values())
print 'keys are:', k
i=0
j=0
while i <3 :
while j <3:
if k[i][j]==u'id':
res.append(v[i][j])
j += 1
i += 1
#res is my result set
del k[:]
del v[:]
发布于 2018-06-30 03:20:21
将my_data更改为:
my_data = [{"id":"xyz","type":"book","attributes":{"doc_type":"article","action":"cut"}}, # Data One
{"id":"xyz2","type":"book","attributes":{"doc_type":"article","action":"cut"}}, # Data Two
{"id":"xyz3","type":"book","attributes":{"doc_type":"article","action":"cut"}}] # Data Three
您可以将其直接转储到数据帧中,如下所示:
mydf = pd.DataFrame(my_data)
不清楚您的数据路径是什么,但如果您正在查找id
、type
等的特定组合,您可以显式搜索
def find_my_way(data, pattern):
# pattern = {'id':'someid', 'type':'sometype'...}
res = []
for row in data:
if row.get('id') == pattern.get('id'):
res.append(row)
return row
mydf = pd.DataFrame(find_my_way(mydata, pattern))
编辑:
在不深入api如何工作的情况下,在伪代码中,您将希望执行类似以下的操作:
my_objects = []
calls = 0
while calls < maximum:
my_data = call_the_api(params)
data = my_data.get('data')
if not data:
calls+=1
continue
# Api calls to single objects usually return a dictionary, to group objects they return lists. This handles both cases
if isinstance(data, list):
my_objects = [*data, *my_objects]
elif isinstance(data, {}):
my_objects = [{**data}, *my_objects]
# This will unpack the data response into a list that you can then load into a DataFrame with the attributes from the api as the columns
df = pd.DataFrame(my_objects)
假设来自api的数据如下所示:
"""
{
"links": {},
"meta": {},
"data": {
"type": "FactivaOrganizationsProfile",
"id": "Goog",
"attributes": {
"key_executives": {
"source_provider": [
{
"code": "FACSET",
"descriptor": "FactSet Research Systems Inc.",
"primary": true
}
]
}
},
"relationships": {
"people": {
"data": {
"type": "people",
"id": "39961704"
}
}
}
},
"included": {}
}
"""
根据文档,这就是我使用my_data.get('data')
的原因。
这应该会将所有数据(未过滤)放入DataFrame中
将DataFrame
保存到最后一位对内存更加友好
https://stackoverflow.com/questions/51107993
复制相似问题