如何使用R连接Hive与Impala

Fayson

修改于 2018-04-01 19:21:47

2.5K0

修改于 2018-04-01 19:21:47

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

1.文档编写目的

继之前的文章讲述如何在Redhat中配置R环境和如何在Redhat中安装R的包及搭建R的私有源后，那我们如何使用R连接CDH集群中的Hive和Impala进行数据分析呢？本文档主要讲述如何使用R连接Hive和Impala，并分别通过命令行和CDSW进行演示。

内容概述

1.驱动包准备

2.命令行测试

3.CDSW测试

4.常见问题

测试环境

1.CM和CDH版本5.11.2

2.采用sudo权限的ec2-user用户操作

3.R版本3.4.2

4.CDSW版本1.1.1

前置条件

1.R环境已配置

2.CDSW正常运行

3.集群未启用认证服务

2.驱动包准备

1.准备Hive的JDBC驱动和Impala的JDBC驱动

Hive下载地址：

[ec2-user@ip-172-31-21-45 hive-test]$ wget https://downloads.cloudera.com/connectors/hive_jdbc_2.5.19.1053.zip

Impala下载地址：

[ec2-user@ip-172-31-21-45 hive-test]$ wget https://downloads.cloudera.com/connectors/impala_jdbc_2.5.41.1061.zip

2.解压（以Hive JDBC为例）

[ec2-user@ip-172-31-21-45 hive-test]$ unzip hive_jdbc_2.5.19.1053.zip

[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$ cd 2.5.19.1053\ GA/
[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$ ll
total 17112
-rw-rw-r-- 1 ec2-user ec2-user 8414253 Jul  7 16:19 Cloudera_HiveJDBC41.zip
-rw-rw-r-- 1 ec2-user ec2-user 8398497 Jul  7 16:20 Cloudera_HiveJDBC4.zip
-r--r--r-- 1 ec2-user ec2-user  559084 Jul  4 11:30 Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf
-r--r--r-- 1 ec2-user ec2-user  117144 Jul  4 11:30 Cloudera-JDBC-Driver-for-Apache-Hive-Release-Notes.pdf
-rw-rw-r-- 1 ec2-user ec2-user   22065 Aug 10 06:26 Cloudera-JDBC-Driver-for-Apache-Hive-Release-Notes.txt
[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$

文件说明：

Cloudera_HiveJDBC41.zip支持JDK7或更高版本

Cloudera_HiveJDBC4.zip支持JDK6或更高版本

Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf说明文档

[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$ unzip Cloudera_HiveJDBC41.zip 
…
  inflating: commons-codec-1.3.jar   
  inflating: commons-logging-1.1.1.jar  
  inflating: HiveJDBC41.jar          
  inflating: hive_metastore.jar      
  inflating: hive_service.jar        
  inflating: httpclient-4.1.3.jar    
  inflating: httpcore-4.1.3.jar      
  inflating: libfb303-0.9.0.jar      
  inflating: libthrift-0.9.0.jar     
  inflating: log4j-1.2.14.jar        
  inflating: ql.jar                  
  inflating: slf4j-api-1.5.11.jar    
  inflating: slf4j-log4j12-1.5.11.jar  
  inflating: TCLIServiceClient.jar   
  inflating: zookeeper-3.4.6.jar     
[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$

将解压的所有jar包移动到/home/ec2-user/hive-test/hivedriver目录

[ec2-user@ip-172-31-21-45 2.5.19.1053 GA]$ mv *.jar /home/ec2-user/hivedriver

3.命令行测试

1.运行R控制台，安装依赖包

> install.packages("DBI")
> install.packages("RJDBC")
> install.packages("rJava")

2.编写R连接Hive测试代码

[ec2-user@ip-172-31-21-45 hive-test]$ vim TestHive.R 
library("DBI")
library("rJava")
library("RJDBC")

hive.class.path = list.files(path=c("/opt/cloudera/parcels/CDH/lib/hive/lib"),pattern="jar", full.names=T);

cp = c(hive.class.path)
.jinit(classpath=cp,parameters="-Djavax.security.auth.useSubjectCredsOnly=false")

drv <- JDBC("org.apache.hive.jdbc.HiveDriver", "/home/ec2-user/hive-test/hivedriver/HiveJDBC41.jar",identifier.quote="`")
conn <- dbConnect(drv, "jdbc:hive2://ip-172-31-21-45.ap-southeast-1.compute.internal:10000")
show_databases <- dbGetQuery(conn, "show databases")
show_databases

tableCount <- dbGetQuery(conn, "select count(*) from tpcds_parquet.call_center")
tableCount

运行测试代码

[ec2-user@ip-172-31-21-45 hive-test]$ Rscript TestHive.R
Loading required package: methods
…
  database_name
1       default
2 tpcds_parquet
3    tpcds_text
  _c0
1  24
[ec2-user@ip-172-31-21-45 hive-test]$

3.编写R连接Impala测试代码

[ec2-user@ip-172-31-21-45 hive-test]$ vim TestImpala.R 
library("DBI")
library("rJava")
library("RJDBC")

impala.class.path = list.files(path=c("/home/ec2-user/hive-test/impaladriver"),pattern="jar", full.names=T)

cp = c(impala.class.path)
.jinit(classpath=cp,parameters="-Djavax.security.auth.useSubjectCredsOnly=false")

drv <- JDBC("com.cloudera.impala.jdbc41.Driver", "/home/ec2-user/hive-test/impaladriver/ImpalaJDBC41.jar",identifier.quote="`")
conn <- dbConnect(drv, "jdbc:impala://ip-172-31-26-102.ap-southeast-1.compute.internal:21050")
show_databases <- dbGetQuery(conn, "show databases")
show_databases

tableCount <- dbGetQuery(conn, "select count(*) from tpcds_parquet.call_center")
tableCount

运行测试代码

[ec2-user@ip-172-31-21-45 hive-test]$ Rscript TestImpala.R 
Loading required package: methods
              name                                      comment
1 _impala_builtins System database for Impala builtin functions
2          default                        Default Hive database
3    tpcds_parquet                                             
4       tpcds_text                                             
  expr_0
1     24
[ec2-user@ip-172-31-21-45 hive-test]$

4.CDSW测试

1.登录CDSW，挂载主机的目录到CDSW会话Session中

CDSW服务所在服务器的目录

进入Admin->Engines，将driver目录挂载到CDSW上

2.登录CDSW创建一个R的工程

3.创建TestHive.r和TestImpala.r

4.启动会话

这个时候我们通过终端就能看到我们挂载的driver目录

5.在CDSW中安装依赖包

install.packages("DBI")
install.packages("RJDBC")
install.packages("rJava")

6.将命令行的测试代码复制到CDSW中的TestHive.r文件中，做相应修改

library("DBI")
library("rJava")
library("RJDBC")

hive.class.path = list.files(path=c("/opt/cloudera/parcels/CDH/lib/hive/lib"),pattern="jar", full.names=T)

cp = c(hive.class.path)
.jinit(classpath=cp,parameters="-Djavax.security.auth.useSubjectCredsOnly=false")

drv <- JDBC("org.apache.hive.jdbc.HiveDriver", "/driver/hivedriver/HiveJDBC41.jar",identifier.quote="`")
conn <- dbConnect(drv, "jdbc:hive2://ip-172-31-21-45.ap-southeast-1.compute.internal:10000")
show_databases <- dbGetQuery(conn, "show databases")
show_databases

tableCount <- dbGetQuery(conn, "select count(*) from tpcds_parquet.call_center")
tableCount

7.运行TestHive.r

8.创建TestImpala.r，将命令行代码拷贝至CDSW中，并做相应修改

library("DBI")
library("rJava")
library("RJDBC")

impala.class.path = list.files(path=c("/home/ec2-user/hive-test/impaladriver"),pattern="jar", full.names=T)

cp = c(impala.class.path)
.jinit(classpath=cp,parameters="-Djavax.security.auth.useSubjectCredsOnly=false")

drv <- JDBC("com.cloudera.impala.jdbc41.Driver", "/driver/impaladriver/ImpalaJDBC41.jar",identifier.quote="`")
conn <- dbConnect(drv, "jdbc:impala://ip-172-31-26-102.ap-southeast-1.compute.internal:21050")
show_databases <- dbGetQuery(conn, "show databases")
show_databases

tableCount <- dbGetQuery(conn, "select count(*) from tpcds_parquet.call_center")
tableCount

9.运行TestImpala.r

5.常见问题

1.安装rJava包的时候报错

cpp flags   : '-I/usr/java/jdk1.6.0_31/jre/../include -I/usr/java/jdk1.6.0_31/jre/../include/linux'
java libs   : '-L/usr/java/jdk1.6.0_31/jre/lib/amd64/server -ljvm'
configure: error: One or more Java configuration variables are not set.
Make sure R is configured with full Java support (including JDK). Run
R CMD javareconf
as root to add Java support to R.

If you don't have root privileges, run
R CMD javareconf -e
to set all Java-related variables and then install rJava.

ERROR: configuration failed for package ‘rJava’
* removing ‘/usr/local/R-3.4.2/lib64/R/library/rJava’

解决方法：

使用root用户执行命令

[root@ip-172-31-21-45 ec2-user]# source /etc/profile
[root@ip-172-31-21-45 ec2-user]# R CMD javareconf   
…
JAVA_HOME        : /usr/java/jdk1.7.0_67-cloudera/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
Updating Java configuration in /usr/local/R-3.4.2/lib64/R
Done.
[root@ip-172-31-21-45 ec2-user]#

醉酒鞭名马，少年多浮夸！岭南浣溪沙，呕吐酒肆下！挚友不肯放，数据玩的花！温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看

推荐关注Hadoop实操，第一时间，分享更多Hadoop干货，欢迎转发和分享。

原创文章，欢迎转载，转载请注明：转载自微信公众号Hadoop实操

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2017-10-12，如有侵权请联系 cloudcommunity@tencent.com 删除

其他

本文分享自 Hadoop实操微信公众号，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

其他

登录后参与评论

0 条评论

热度

如何使用R连接Hive与Impala

如何使用R连接Hive与Impala

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐