如何使用Spark scala从字符串格式的复杂JSON创建数据帧

使用Spark scala从字符串格式的复杂JSON创建数据帧的步骤如下：

导入必要的Spark库和类：

import org.apache.spark.sql.{SparkSession, DataFrame}
import org.apache.spark.sql.functions._

创建SparkSession对象：

val spark = SparkSession.builder()
  .appName("JSON to DataFrame")
  .getOrCreate()

定义JSON字符串：

val jsonString = """
{
  "name": "John",
  "age": 30,
  "address": {
    "street": "123 Main St",
    "city": "New York",
    "state": "NY"
  },
  "hobbies": ["reading", "traveling"],
  "education": [
    {
      "degree": "Bachelor",
      "major": "Computer Science"
    },
    {
      "degree": "Master",
      "major": "Data Science"
    }
  ]
}
"""

将JSON字符串转换为数据帧：

val df = spark.read.json(Seq(jsonString).toDS())

查看数据帧的结构和内容：

df.printSchema()
df.show()

对于复杂的JSON结构，可以使用Spark的内置函数来处理嵌套字段：

val flattenedDF = df.select(
  col("name"),
  col("age"),
  col("address.street").alias("street"),
  col("address.city").alias("city"),
  col("address.state").alias("state"),
  explode(col("hobbies")).alias("hobby"),
  col("education.degree").alias("degree"),
  col("education.major").alias("major")
)