0507-Hive查询json格式表执行MapReduce任务错误问题分析和解决

Fayson

发布于 2019-11-28 23:36:25

9260

发布于 2019-11-28 23:36:25

文章被收录于专栏：Hadoop实操Hadoop实操

作者：冉南阳

文档编写目的

本篇文章主要介绍hive里创建的json格式的表，全表查询时成功，当查询时需要提交mapreduce任务时失败问题描述和解决。

内容概述

1.问题描述

2.问题重新

3.问题解决

测试环境

1.CM和CDH版本为5.13.3

2.操作系统版本为RedHat7.2

3.集群未启用Kerberos

问题描述

1、问题描述

查询hive里创建的json格式的表全表时成功，当查询时需要提交mapreduce任务时失败。

当查询全表信息是成功：

需要提交任务时失败，Hive命令行错误如下，无法看出有效信息：

查看Yarn的日志文件，错误如下：

2、问题原因

对json格式的表执行查询时，若需要对该表的json文件进行解析，则需要依赖类org.apache.hive.hcatalog.data.JsonSerDe；查询全表时，不需要对进行解析，所以不会失败；当查询语句需要对json内容解析时，会提交任务到yarn，而yarn的环境缺少该类，所以造成任务失败。

问题重现

1、创建json表

创建测试文件

{"student":{"name":"king","age":11,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}
{"student":{"name":"wang","age":12,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}
{"student":{"name":"test","age":13,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}
{"student":{"name":"test2","age":14,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}
{"student":{"name":"test3","age":15,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}
{"student":{"name":"test4","age":16,"sex":"M"},"class":{"book":"语文","level":2,"score":80},"teacher":{"name":"t1","class":"语文"}}

2、创建hive表

create external table if not exists student(
student map<string,string> comment "学生信息",
class map<string,string> comment "课程信息",
teacher map<string,string> comment "授课老师信息"
)
comment "学生课程信息"
row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
stored as textfile;

3、上传数据到表的hdfs路径中

hdfs dfs -put json.txt /user/hive/warehouse/student

4、查询全表

查询全表成功

select  *  from student

5、执行需要提交任务的查询

执行失败

select
    student['name'] as stuName,
    class['book'] as cls_book, 
    class['score'] as cls_score,
    teacher['name'] as tech_name 
from student;

问题解决

1、方法一：在每个节点创建软链接

ln -s  /opt/cloudera/parcels/CDH/jars/hive-hcatalog-core-1.1.0-cdh5.13.1.jar /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/hive-hcatalog-core-1.1.0-cdh5.13.1.jar

此方法需要在每个nodemanager节点执行，不需要重启集群。

2、方法二：在yarn界面上配置

将该jar包路径添加到配置MR 应用程序 Classpath中，然后重启集群。