spark踩坑——dataframe写入hbase连接异常

最近测试环境基于shc[https://github.com/hortonworks-spark/shc]的hbase-connector总是异常连接不到zookeeper,看下报错日志:

18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5175ab05 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x6399f976 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/20 10:45:02 INFO ZooKeeper: Client environment:host.name=hnode8
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.version=1.8.0_66
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_66/jre
...
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003/tmp
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.name=Linux
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.arch=amd64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.name=yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.home=/var/lib/hadoop-yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.dir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5175ab050x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6399f9760x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

可以观察到hbase-connector莫名其妙连接的是localhost:2181,检查所有的配置文件都没有错。同样的代码线上就正常运行,对比之下发现是缺少了hbase-site.xml的配置文件。

查找shc的issue发现已经有人提出这种问题了: https://github.com/hortonworks-spark/shc/issues/227

大意是说,默认会连接localhost:2181,如果需要连接远程的hbase,只需要拷贝hbase-site.xml到类目录下即可。由于我这边还用了nameservice,因此hdfs-site.xml等配置文件也需要一同打包到Jar中。

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏一个爱瞎折腾的程序猿

sqlserver使用存储过程跟踪SQL

USE [master] GO /****** Object: StoredProcedure [dbo].[sp_perfworkload_trace_s...

2880
来自专栏张善友的专栏

Silverlight + Model-View-ViewModel (MVVM)

     早在2005年,John Gossman写了一篇关于Model-View-ViewModel模式的博文,这种模式被他所在的微软的项目组用来创建Expr...

3278
来自专栏闻道于事

js登录滑动验证,不滑动无法登陆

js的判断这里是根据滑块的位置进行判断,应该是用一个flag判断 <%@ page language="java" contentType="text/html...

8698
来自专栏码匠的流水账

聊聊spring cloud netflix的HystrixCommands

本文主要研究一下spring cloud netflix的HystrixCommands。

1042
来自专栏hbbliyong

WPF Trigger for IsSelected in a DataTemplate for ListBox items

<DataTemplate DataType="{x:Type vm:HeaderSlugViewModel}"> <vw:HeaderSlug...

4224
来自专栏C#

DotNet加密方式解析--非对称加密

    新年新气象,也希望新年可以挣大钱。不管今年年底会不会跟去年一样,满怀抱负却又壮志未酬。(不过没事,我已为各位卜上一卦,卦象显示各位都能挣钱...)...

5968
来自专栏转载gongluck的CSDN博客

cocos2dx 打灰机

#include "GamePlane.h" #include "PlaneSprite.h" #include "BulletNode.h" #include...

7226
来自专栏跟着阿笨一起玩NET

c#实现打印功能

3732
来自专栏一个会写诗的程序员的博客

Spring Reactor 项目核心库Reactor Core

Non-Blocking Reactive Streams Foundation for the JVM both implementing a Reactiv...

2802
来自专栏魂祭心

原 canvas绘制clock

5144

扫码关注云+社区