如何将数组转换为spark中的列？
如何将数组转换为spark中的列？
提问于 2022-01-02 15:37:25
我有一个数据文件，如下所示
+-----------------------------+
| Item |
+-----------------------------+
|[[a,b,c], [d,e,f], [g,h,i]] |
+--------------------+--------+如何将其转换为下表？
a b c
d e f
g h i我尝试过使用explode和withColumn函数
a b c
a e c
a h c
d b c
d e c
d h c
... (many other combinations)回答 2
Stack Overflow用户
发布于 2022-01-02 15:44:55
只需引爆第一级数组，就可以选择数组元素作为列：
import pyspark.sql.functions as F
df = spark.createDataFrame(
[([["a","b","c"], ["d","e","f"], ["g","h","i"]],)],
["Item"]
)
df.withColumn("Item", F.explode("Item")).select(
*[F.col("Item")[i].alias(f"col_{i}") for i in range(3)]
).show()
#+-----+-----+-----+
#|col_0|col_1|col_2|
#+-----+-----+-----+
#| a| b| c|
#| d| e| f|
#| g| h| i|
#+-----+-----+-----+Stack Overflow用户
发布于 2022-01-02 21:26:24
@黑主教改善你的答案..。
import pyspark.sql.functions as F
df = spark.createDataFrame(
[([["a","b","c"], ["d","e","f"], ["g","h","i", "j"]],)],
["data"]
)
df.show(20, False)
df = df.withColumn("data1", F.explode("data"))
df.select('data1').show()
# Row(max(size(data1))=4) ---> 4
max_size = df.select(F.max(F.size('data1'))).collect()[0][0]
df.select(
*[F.col("data1")[i].alias(f"col_{i}") for i in range(max_size)]
).show()
+------------------------------------+
|data |
+------------------------------------+
|[[a, b, c], [d, e, f], [g, h, i, j]]|
+------------------------------------+
+------------+
| data1|
+------------+
| [a, b, c]|
| [d, e, f]|
|[g, h, i, j]|
+------------+
+-----+-----+-----+-----+
|col_0|col_1|col_2|col_3|
+-----+-----+-----+-----+
| a| b| c| null|
| d| e| f| null|
| g| h| i| j|
+-----+-----+-----+-----+页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/70557310
复制相关文章
相似问题
腾讯云开发者
