如何使用scala读取spark中的json文件？
如何使用scala读取spark中的json文件？
提问于 2017-07-26 16:49:44
我想以以下格式读取JSON文件：
{
"titlename": "periodic",
"atom": [
{
"usage": "neutron",
"dailydata": [
{
"utcacquisitiontime": "2017-03-27T22:00:00Z",
"datatimezone": "+02:00",
"intervalvalue": 28128,
"intervaltime": 15
},
{
"utcacquisitiontime": "2017-03-27T22:15:00Z",
"datatimezone": "+02:00",
"intervalvalue": 25687,
"intervaltime": 15
}
]
}
]
}我把我的读取行写成：
sqlContext.read.json("user/files_fold/testing-data.json").printSchema但我没有得到想要的结果-
root
|-- _corrupt_record: string (nullable = true)在这方面请帮帮我
回答 4
Stack Overflow用户
回答已采纳
发布于 2017-07-26 17:03:14
我建议使用wholeTextFiles读取该文件，并应用一些函数将其转换为单行JSON格式。
val json = sc.wholeTextFiles("/user/files_fold/testing-data.json").
map(tuple => tuple._2.replace("\n", "").trim)
val df = sqlContext.read.json(json)最终的有效dataframe应该是
+--------------------------------------------------------------------------------------------------------+---------+
|atom |titlename|
+--------------------------------------------------------------------------------------------------------+---------+
|[[WrappedArray([+02:00,15,28128,2017-03-27T22:00:00Z], [+02:00,15,25687,2017-03-27T22:15:00Z]),neutron]]|periodic |
+--------------------------------------------------------------------------------------------------------+---------+和有效的schema为
root
|-- atom: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- dailydata: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- datatimezone: string (nullable = true)
| | | | |-- intervaltime: long (nullable = true)
| | | | |-- intervalvalue: long (nullable = true)
| | | | |-- utcacquisitiontime: string (nullable = true)
| | |-- usage: string (nullable = true)
|-- titlename: string (nullable = true)Stack Overflow用户
发布于 2018-11-14 00:58:44
Spark 2.2引入了multiLine选项，可用于加载JSON (非JSON)文件：
spark.read
.option("multiLine", true).option("mode", "PERMISSIVE")
.json("/path/to/user.json")Stack Overflow用户
发布于 2017-07-26 16:57:44
它可能与存储在您的文件中的JSON对象有关，您是否可以打印它或确保它是您在问题中提供的对象？我问这个是因为我用了那个，它运行得很好：
val json =
"""
|{
| "titlename": "periodic",
| "atom": [
| {
| "usage": "neutron",
| "dailydata": [
| {
| "utcacquisitiontime": "2017-03-27T22:00:00Z",
| "datatimezone": "+02:00",
| "intervalvalue": 28128,
| "intervaltime": 15
| },
| {
| "utcacquisitiontime": "2017-03-27T22:15:00Z",
| "datatimezone": "+02:00",
| "intervalvalue": 25687,
| "intervaltime": 15
| }
| ]
| }
| ]
|}
""".stripMargin
val spark = SparkSession.builder().master("local[*]").getOrCreate()
spark.read
.json(spark.sparkContext.parallelize(Seq(json)))
.printSchema()页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/45321924
复制相关文章
相似问题