我有一个演示应用程序,它使用crossfilter.js维度和组来通过交互式过滤器为图形提供动力(非常类似于航空公司的准时演示http://square.github.io/crossfilter/)。我的真实数据集太大了,无法使用crossfilter.js。不过,我已经成功地使用pandas对数据进行了类似的过滤。
然而,我在理解如何在pandas中建模/表示crossfilter的group()行为时遇到了困难--即
分组与交叉筛选器的当前筛选器相交,关联维度的筛选器除外。https://github.com/square/crossfilter/wiki/API-Reference#group-map-reduce
例如,使用车辆数据:
Make Year Color
-------------------
Ford 2000 Red
Honda 2001 Blue
Ford 2001 Green
如果我应用筛选器Make: Ford
并获得每个维度/组的计数,我会期望:
Make:
Ford: 2
Honda: 1
Year:
2000: 1
2001: 1
Color:
Red: 1
Blue: 0
Green: 1
因此,对于Make
维度,将删除Make: Ford
过滤器以获取计数。对于Year
和Color
的尺寸,它是适用的,所以2001年的蓝色本田没有贡献的计数。
发布于 2018-10-10 06:55:42
在没有一个好答案的情况下,这就是我拼凑出来的。它需要将过滤器编码到树中,以便可以遍历该树,并在每次传递中将适当的系列过滤器置零。不过,我仍然对更好的解决方案感兴趣。
基于上述问题的示例调用,其中df
是熊猫数据帧:
crossfilter(df, ('eq', 'Make', 'Ford'), ['Make', 'Year', 'Color'])
代码:
# filter operators of the form (operator, filter1, filter2)
group_ops = {
'and': operator.and_,
'or': operator.or_,
}
# hokie way of forcing all-pass or all-fail filters
nan = float('nan')
# recursive function that turns a tree of python-dict encoded filters into
# bitwise operators for pandas
def build_filter(df, payload, nullify_series=None, nullify_value=True):
if not payload:
# no filters, but we have to return something
# so grab the first series and filter out all NaN values
return operator.ne(df.ix[:,0], nan)
op = payload[0]
if op in value_ops:
# format: (operator, series, val)
series = payload[1]
value = payload[2]
if series == nullify_series:
# nullify filter
if nullify_value:
# push toward True (e.g. nested in an 'and' operator)
return operator.ne(df[series], nan)
else:
# push toward False (e.g. nested in an 'or' operator)
return operator.eq(df[series], nan)
return value_ops[op](df[series], value)
elif op == 'not':
# format: ('not', nested_filter)
value = payload[1]
return operator.inv(build_filter(df, value, nullify_series, False))
else:
# format: (operator, nested_filter_1, nested_filter_2)
group1 = payload[1]
group2 = payload[2]
return group_ops[op](build_filter(df, group1, nullify_series, True),
build_filter(df, group2, nullify_series, True))
# returns value counts for all series in `gather`, applying filters in `filters` in all other series
def crossfilter(df, filters, gather):
df_scoped = df[gather]
results = { series: df_filtered[series].value_counts().to_dict()
for series in gather
for df_filtered in [ df_scoped[build_filter(df_scoped, filters, series)]
if filters else df_scoped ]}
return results
https://stackoverflow.com/questions/52721306
复制相似问题