spark踩坑——dataframe写入hbase连接异常

最近测试环境基于shc[https://github.com/hortonworks-spark/shc]的hbase-connector总是异常连接不到zookeeper,看下报错日志:

18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5175ab05 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x6399f976 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/20 10:45:02 INFO ZooKeeper: Client environment:host.name=hnode8
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.version=1.8.0_66
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_66/jre
...
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003/tmp
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.name=Linux
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.arch=amd64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.name=yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.home=/var/lib/hadoop-yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.dir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5175ab050x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6399f9760x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

可以观察到hbase-connector莫名其妙连接的是localhost:2181,检查所有的配置文件都没有错。同样的代码线上就正常运行,对比之下发现是缺少了hbase-site.xml的配置文件。

查找shc的issue发现已经有人提出这种问题了: https://github.com/hortonworks-spark/shc/issues/227

大意是说,默认会连接localhost:2181,如果需要连接远程的hbase,只需要拷贝hbase-site.xml到类目录下即可。由于我这边还用了nameservice,因此hdfs-site.xml等配置文件也需要一同打包到Jar中。

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏逸鹏说道

C#通过WMI的wind32 的API函数实现msinfo32的本地和远程计算机的系统日志查看功能

先不说如何实现,先来看看效果图: ? 读取远程的需要提供下远程的计算用户名和密码即可。 如何实现这个代码功能,请看如下代码部分: #region//获取日志文件...

35050
来自专栏大内老A

我的WCF之旅(13):创建基于MSMQ的Responsive Service

一、One-way MEP V.S. Responsible Service 我们知道MSMQ天生就具有异步的特性,它只能以One-way的MEP(Messag...

22960
来自专栏跟着阿笨一起玩NET

DataGridView减少闪烁的解决办法

http://www.codeproject.com/Tips/390496/Reducing-flicker-blinking-in-DataGridView

17110
来自专栏一个会写诗的程序员的博客

Spring Boot + JPA + Freemarker 实现后端分页 完整示例Spring Boot + JPA + Freemarker 实现后端分页 完整示例

https://github.com/EasyKotlin/chatper15_net_io_img_crawler

11620
来自专栏张善友的专栏

Quartz.NET 1.0.1发布

这个版本包含1.0版本发布以来发现的bug修复,也包括使用AdoJobStore时的性能优化,增加了一个新特性是对SQL Server Compact Edit...

19660
来自专栏木宛城主

SharePoint 2013自定义Providers在基于表单的身份验证(Forms-Based-Authentication)中的应用

由于项目的需要,登录SharePoint Application的用户将从一个统一平台中获取,而不是从Domain中获取,所以需要对SharePoint Ap...

24390
来自专栏函数式编程语言及工具

Akka(29): Http:Server-Side-Api,Low-Level-Api

 Akka-http针对Connection的两头都提供了方便编程的Api,分别是Server-Side-Api和Client-Side-Api。通过这两个Ap...

21880
来自专栏乐沙弥的世界

不可或缺的 sendEmail

    还在为Linux下没有便捷的邮件程序苦恼,还在为复杂的邮件服务器架设Google N多网页? 对于小型,便捷的Linux下命令行邮件程序,sendEma...

9220
来自专栏蘑菇先生的技术笔记

Fd.Service 轻量级WebApi框架

23840
来自专栏hbbliyong

WCF发布多个服务

using System; using System.Collections.Generic; using System.Linq; using System....

39750

扫码关注云+社区

领取腾讯云代金券