问Apache Spark 2.3.1与蜂巢转移3.1.0

EN

我们将HDP集群升级到3.1.1.3.0.1.0-187，并发现：

1. 蜂巢有一个新的亚稳态位置
2. 星星之火看不到蜂巢数据库

事实上，我们看到：

org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ... not found

你能帮我了解一下发生了什么以及如何解决这个问题吗？

更新：

配置：

(spark.sql.warehouse.dir，/spark.sql.warehouse.dir/tablespace/external/hive/) (spark.admin.acls，) (spark.yarn.dist.files，spark.yarn.dist.files，spark.history.kerberos.keytab (spark.history.kerberos.keytab，/etc/security/keytab/spapk.service.keytab)(scapk.io.压缩.lz4.块大小，128 etc) (spark.executor.extraJavaOptions，( (spark.sql.autoBroadcastJoinThreshold,26214400) ) (spark.io.encryption.keygen.algorithm，HmacSHA1) (spark.eventLog.enabled，true) (spark.shuffle.service.enabled，true) (spark.driver.extraLibraryPath，/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64) (spark.ssl.keyStore，/etc/security/serverKeys/server-keystore.jks) (spark.yarn.queue，默认) (spark.jars，file:/opt/folder/component-assembly-0.1.0-SNAPSHOT.jar) (spark.ssl.enabled，true) ) (spark.sql.orc.filterPushdown，(spark.shuffle.unsafe.file.output.buffer,5m) (spark.yarn.historyServer.address，master2.env.project:18481) (spark.ssl.trustStore，/etc/security/clientKeys/all.jks) (spark.app.name，com.company.env.component.MyClass) (spark.sql.hive.metastore.jars，spark.io.encryption.keySizeBits，128) (spark.driver.memory，(spark.executor.instances,10) (spark.history.kerberos.principal /edge.env.project@enV.PROJECT)spark.executor.instances，10 (spark.ssl.keyPassword，*(编校)) (spark.ssl.keyStorePassword，*(编校)) (spark.history.fs.cleaner.enabled，真) (spark.shuffle.io.serverThreads，(spark.sql.hive.convertMetastoreOrc，true) (spark.submit.deployMode，client) (spark.sql.orc.char.enabled，true) (spark.master，true) (spark.authenticate.enableSaslEncryption，true) (spark.authenticate.enableSaslEncryption，true) (spark.authenticate，true) (spark.history.fs.cleaner.maxAge,90d) (spark.history.ui.acls.enable，true) (spark.acls.enable，true) (spark.history.provider，true)org.apache.spark.deploy.history.FsHistoryProvider) (spark.executor.extraLibraryPath，/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64) (spark.executor.memory,2g) (spark.io.encryption.enabled，true) (spark.shuffle.file.buffer,1m) ) (spark.eventLog.dir，hdfs://hdfs 2-历史/) (spark.ssl.protocol，TLS) (spark.dynamicAllocation.enabled，(真) (spark.executor.cores,3) (spark.history.ui.port,18081) (spark.sql.statistics.fallBackToHdfs，真) (spark.repl.local.jars，spark.ssl.trustStorePassword (spark.ssl.trustStorePassword，*(编校)) (spark.history.ui.admin.acls，) (spark.history.kerberos.enabled，(spark.shuffle.io.backLog,8192) (spark.sql.orc.impl，原生) (spark.ssl.enabledAlgorithms，TLS_RSA_WITH_AES_128_CBC_SHA，TLS_RSA_WITH_AES_256_CBC_SHA) (spark.sql.orc.enabled，true) (spark.yarn.dist.jars，TLS_RSA_WITH_AES_256_CBC_SHA)file:///opt/folder/ojdbc6.jar) (Spark.sql.hive.asiore.version，3.0)

来自hive-site.xml：

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/warehouse/tablespace/managed/hive</value>
</property>

代码看起来如下：

val spark = SparkSession
  .builder()
  .appName(getClass.getSimpleName)
  .enableHiveSupport()
  .getOrCreate()
...
dataFrame.write
  .format("orc")
  .options(Map("spark.sql.hive.convertMetastoreOrc" -> true.toString))
  .mode(SaveMode.Append)
  .saveAsTable("name")

火花-提交：

    --master yarn \
    --deploy-mode client \
    --driver-memory 2g \
    --driver-cores 4 \
    --executor-memory 2g \
    --num-executors 10 \
    --executor-cores 3 \
    --conf "spark.dynamicAllocation.enabled=true" \
    --conf "spark.shuffle.service.enabled=true" \
    --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=cacerts" \
    --conf "spark.sql.warehouse.dir=/warehouse/tablespace/external/hive/" \
    --jars postgresql-42.2.2.jar,ojdbc6.jar \
    --files config.yml,/opt/jdk1.8.0_172/jre/lib/security/cacerts \
    --verbose \
    component-assembly-0.1.0-SNAPSHOT.jar \