spark踩坑——dataframe写入hbase连接异常

最近测试环境基于shc[https://github.com/hortonworks-spark/shc]的hbase-connector总是异常连接不到zookeeper,看下报错日志:

18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5175ab05 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x6399f976 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/20 10:45:02 INFO ZooKeeper: Client environment:host.name=hnode8
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.version=1.8.0_66
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_66/jre
...
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003/tmp
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.name=Linux
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.arch=amd64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.name=yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.home=/var/lib/hadoop-yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.dir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5175ab050x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6399f9760x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

可以观察到hbase-connector莫名其妙连接的是localhost:2181,检查所有的配置文件都没有错。同样的代码线上就正常运行,对比之下发现是缺少了hbase-site.xml的配置文件。

查找shc的issue发现已经有人提出这种问题了: https://github.com/hortonworks-spark/shc/issues/227

大意是说,默认会连接localhost:2181,如果需要连接远程的hbase,只需要拷贝hbase-site.xml到类目录下即可。由于我这边还用了nameservice,因此hdfs-site.xml等配置文件也需要一同打包到Jar中。

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏三杯水

Ambari安装部署Hadoop

Apache Ambari是一种基于Web的工具,支持Apache Hadoop集群的供应、管理和监控。Ambari已支持大多数Hadoop组件,包括HDFS、...

1203
来自专栏Kubernetes

kube-proxy源码分析

##kube-proxy介绍 请参考我的另一篇博文:kube-proxy工作原理 ##源码目录结构分析 cmd/kube-proxy //负责kub...

4525
来自专栏大数据学习笔记

Hadoop基础教程-第9章 HA高可用(9.3 HDFS 高可用运行)(草稿)

第9章 HA高可用 9.3 HDFS 高可用运行 9.3.1 HA节点规划 节点 IP Zookeeper NameNode JournalNode Da...

2355
来自专栏JetpropelledSnake

SNMP学习笔记之SNMP TRAP简介、流程以及使用Python实现接受Trap信息

SNMP(Simple Network Management Protocol) trap是一种很有用,但是也容易让人难以理解的协议。

672
来自专栏跟着阿笨一起玩NET

WebService基于SoapHeader实现安全认证

      本文仅提供通过设置SoapHeader来控制非法用户对WebService的调用,如果是WebService建议使用WSE3.0来保护Web服务,如...

882
来自专栏张高兴的博客

张高兴的 Windows 10 IoT 开发笔记:使用 ADS1115 读取模拟信号

3586
来自专栏Hadoop实操

如何通过Livy的RESTful API接口向非Kerberos环境的CDH集群提交作业

在前面的文章Fayson介绍了《Livy,基于Apache Spark的开源REST服务,加入Cloudera Labs》和《如何编译Livy并在非Kerber...

4877
来自专栏别先生

kafka.common.KafkaException: Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instan

3972
来自专栏Hadoop实操

如何开发HBase Endpoint类型的Coprocessor以及部署使用

1422
来自专栏Java学习123

IBM WebSphere MQ 系列(四) 使用MQ命令

4216

扫码关注云+社区