使用CDP遇到的问题1

soundhearer

发布于 2020-10-15 11:44:09

1.5K0

发布于 2020-10-15 11:44:09

文章被收录于专栏：数据湖数据湖

CDP比较新，在使用的过程中会不断地遇到新的问题，在此我们会用一系列的文章把问题和解决方案记录下来，供读者们参考，也方便我们复盘之前遇到的问题。

问题场景

提交hive on Tez的作业后，出现以下错误

Error: Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1594429058117_0003_1_00, diagnostics=[Task failed, taskId=task_1594429058117_0003_1_00_000001, diagnostics=[TaskAttempt 0 failed, info=[Container container_1594429058117_0003_02_000003 finished with diagnostics set to [Container failed, exitCode=-1000. [2020-07-11 07:13:44.159]Not able to initialize app directories in any of the configured local directories for app application_1594429058117_0003
]], TaskAttempt 1 failed, info=[Container container_1594429058117_0003_02_000006 finished with diagnostics set to [Container failed, exitCode=-1000. [2020-07-11 07:13:45.559]Not able to initialize app directories in any of the configured local directories for app application_1594429058117_0003

集群之前开启了 Kerberos，为了使用方便我又禁用了 kerberos，以上错误便是在禁用了 kerberos后出现的

问题分析

经过分析后发现这实际上是权限问题，CDP集群未启用 kerberos之前， /yarn/nm/usercache/test的目录权限为 yarn:yarn，启用后为权限变成了 test:yarn，禁用kerberos后，目录权限没有更新，日志无法生成，导致作业失败。

解决方案

查找NodeManager所在节点

查找 `yarn.nodemanager.local-dirs`所在目录

删除所有NodeManager节点的 `usercache`目录

重启YARN

以上错误解决，但是又出现下面的错误

Caused by: org.apache.orc.FileFormatException: Malformed ORC file hdfs://cdp1.hadoop.com:8020/warehouse/tablespace/managed/hive/student_tb_txt/22.txt. Invalid postscript.
  at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:274)
  at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:580)
  at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:369)
  at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:61)
  at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:111)
  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createOrcReaderForSplit(OrcInputFormat.java:2229)
  at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:143)
  at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:135)
  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2044)
  at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
  ... 28 more