本文的安装参照了官方的文档:GettingStarted,将Hive 0.12.0安装在Hadoop 2.4.0上。本文将Hive配置成Server模式,并且使用MySQL作为元数据数据库,远程连接MySQL。
关于Hadoop 2.4.0的安装,请参见《Hadoop-2.4.0分布式安装手册》一文。
本文约定Hadoop被安装在/data/hadoop/current,将Hive 0.12.0的安装到目录/data/hadoop/hive(实际是指向/data/hadoop/hive-0.12.0-bin的软链接)。在实际安装部署时,可以指定为其它目录。
10000 | hive.server2.thrift.port,执行hiveserver2时会启动它 |
---|---|
9083 | hive.metastore.uris,执行hive --service metastore时会启动它 |
本文MySQL被安装在172.25.39.166机器上,Hive用MySQL来存储元数据,因此需要先安装好MySQL。这里安装的是最新的MySQL 5.6.17,下载网址为:http://dev.mysql.com/downloads/mysql/,本文选择是的“Linux - Generic”下的“Linux - Generic (glibc 2.5) (x86, 64-bit), Compressed TAR Archive”,它的二进制安装包名为mysql-5.6.17-linux-glibc2.5-x86_64.tar.gz。
将二进制安装包解压后,可看到名为INSTALL-BINARY的文件,该文件有说明如何安装MySQL,本文基本参照它进行的,MySQL安装目录为/data/hadoop/mysql,具体步骤如下(未使用mysql用户及mysql用户组,而是直接使用了当前登录用户hadoop,hadoop隶属用户组users):
# 以下均以当前用户hadoop执行
cd /data/hadoop
tar xzf mysql-5.6.17-linux-glibc2.5-x86_64.tar.gz
ln -s mysql-5.6.17-linux-glibc2.5-x86_64 mysql
cd mysql
scripts/mysql_install_db --user=hadoop
bin/mysqld_safe --user=hadoop &
cp support-files/mysql.server /etc/init.d/mysql.server # 这一条需求以root用户运行
创建数据库hive:
create database if not exists hive;
创建数据库用户hive:
create user hive identified by 'hive2014';
授权可以访问数据库hive的IP和用户,其中localhost的实际IP为172.25.39.166:
grant all on hive.* to 'hive'@'localhost' identified by 'hive2014';
grant all on hive.* to 'hive'@'172.25.39.166' identified by 'hive2014';
grant all on hive.* to 'hive'@'172.25.40.171' identified by 'hive2014';
进入hive数据库:
1) 本机进入:mysql -uhive -phive2014
2) 非本南进入:mysql -uhive -h172.25.39.166 -phive2014
下载网址:http://hive.apache.org/downloads.html,下载后的包名为:hive-0.12.0-bin.tar.gz,然后将hive-0.12.0-bin.tar.gz上传到/data目录下。
1) 切换到/data目录:cd /data
2) 解压二进制安装包:tar xf hive-0.12.0-bin.tar.gz
3) 建立软链接:ln -s hive-0.12.0-bin hive
MySQL-Connector下载网址:http://dev.mysql.com/downloads/connector/。
选择“Connector/J”,接着选择“Platform Independent”,本文下载的是“mysql-connector-java-5.1.30.tar.gz”。
压缩包“mysql-connector-java-5.1.30.tar.gz”中有个mysql-connector-java-5.1.30-bin.jar,解压后将mysql-connector-java-5.1.30-bin.jar上传到Hive的lib目录下,这个是MySQL的JDBC驱动程序。
设置环境变量HIVE_HOME,并将Hive加入到PATH中:
export HIVE_HOME=/data/hadoop/hive
export PATH=$HIVE_HOME/bin:$PATH
进入/data/hadoop/hive/conf目录,可以看到如下:
hadoop@VM-40-171-sles10-64:~/hive/conf> ls
hive-default.xml.template hive-exec-log4j.properties.template
hive-env.sh.template hive-log4j.properties.template
可以看到4个模板文件,复制并重命名成配置文件:
cp hive-env.sh.template hive-env.sh
cp hive-default.xml.template hive-site.xml
cp hive-log4j.properties.template hive-log4j.properties
cp hive-exec-log4j.properties.template hive-exec-log4j.properties
如果之前没有设置好HADOOP_HOME环境变量,则可在hive-env.sh中,进行设置:
HADOOP_HOME=/data/hadoop/current
1) 修复BUG
该文件有个语法BUG,需要修改,进入到hive-site.xml的第2000行,该行内容为:auth,明显的语法错误,需要将“auth”改成“value”。
2) 修改javax.jdo.option.ConnectionURL
将值设置为:jdbc:mysql://172.25.39.166:3306/hive?characterEncoding=UTF-8。
3) 修改javax.jdo.option.ConnectionDriverName
将值设置为:com.mysql.jdbc.Driver。
4) 修改javax.jdo.option.ConnectionUserName
将值设置为访问hive数据库的用户名hive:hive。
5) 修改javax.jdo.option.ConnectionPassword
将值设置为访问hive数据库的密码:hive2014。
6) 修改hive.metastore.schema.verification
该值试情况进行修改。
7) 修改hive.zookeeper.quorum
将值设置为:10.12.154.77,10.12.154.78,10.12.154.79,ZooKeeper被安装在这三台机器上。
8) 修改hive.metastore.uris
将值设置为:thrift://172.25.40.171:9083,9083为Hive元数据的RPC服务端口。
9) 修改hive.metastore.warehouse.dir
将值设置为:/data/hadoop/hive/warehouse,注意启动前,需要创建好该目录(mkdir /data/hadoop/hive/warehouse)。
10) 修改hive.server2.thrift.bind.host
该值默认为localhost,如果需要在其它机器远程访问Hive,则需要将它改成IP地址,本文将它改成172.25.40.171。
修改日志文件存放目录,将日志目录由/tmp/${user.name}改为/data/hadoop/hive/logs:
hive.log.dir=/data/hadoop/hive/logs
然后创建好目录/data/hadoop/hive/logs。
修改日志文件存放目录,将日志目录由默认的/tmp/${user.name}改为/data/hadoop/hive/logs/exec:
hive.log.dir=/data/hadoop/hive/logs/exec
然后创建好目录/data/hadoop/hive/logs/exec。
1) 初始化metastore
安装配置好后,在启动Hive服务端之前,需要在服务端执行一次“schematool -dbType mysql -initSchema”,以完成对metastore的初始化。
2) 启动metastore
执行命令:hive --service metastore &
3) 启动Hive服务
执行:hiveserver2 &。
4) 进入Hive命令行操作界面(类似于mysql)
执行:hive
除了使用hive命令行操作界面之外,hiveserver2还提供了beeline(hive是用户名,hive2014是密码,可以从HiveServer2+Clients获得更多信息):
hadoop@VM-40-171-sles10-64:~/hive/bin> ./beeline
Beeline version 0.12.0 by Apache Hive
beeline> !connect jdbc:hive2://172.25.40.171:10000 hive hive2014 org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://172.25.40.171:10000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/hadoop/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/hadoop/hive-0.12.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Hive (version 0.12.0)
Driver: Hive (version 0.12.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://172.25.40.171:10000> select * from invites limit 2;
+------+----------+-------+
| foo | bar | ds |
+------+----------+-------+
| 474 | val_475 | 2014 |
| 281 | val_282 | 2014 |
+------+----------+-------+
2 rows selected (1.779 seconds)
0: jdbc:hive2://172.25.40.171:10000>
将hive/bin、hive/lib、hive/conf和hive/examples打包,如:tar czf hive-bin.tar.gz hive/bin hive/lib hive/conf hive/examples。
然后将hive-bin.tar.gz上传到其它机器,借助beeline即可远程执行HSQL(用hive可能会遇到问题,本文在操作时,使用hive,在执行HSQL时总会卡住,日志也没有记录特别原因,暂未去定位)。
以下内容来自官网(GettingStarted),注意命令不区分大小写:
CREATE TABLE pokes (foo INT, bar STRING);
CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING);
SHOW TABLES;
SHOW TABLES '.*s';
DESCRIBE invites;
DROP TABLE pokes;
Hive的安装目录下有个examples子目录,存储了示例用到的数据文件等。测试往表invites中加载数据,将文件../examples/files/kv2.txt加载到表invites中:
LOAD DATA LOCAL INPATH '../examples/files/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2014');
可以通过“select * from invites;”来检验加载情况,或者执行“select count(1) from invites;”。
1) Failed to connect to the MetaStore Server
如果运行hiveserver2,遇到下列错误后,推荐打开DEBUG日志级别,以更查看更详细的信息,将日志配置文件hive-log4j.properties中的“hive.root.logger=WARN,DRFA”改成“hive.root.logger=DEBUG,WARN,DRFA”即可。
2014-04-23 06:00:04,169 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2014-04-23 06:00:05,173 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2014-04-23 06:00:06,177 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2014-04-23 06:00:07,181 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2014-04-23 06:00:08,185 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2014-04-23 06:00:09,194 ERROR service.CompositeService (CompositeService.java:start(74)) - Error starting services HiveServer2
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
修改后,再次运行hiveserver2,日志变详细了,猜测是metastore没有起来,可以通过执行“hive --service metastore”来启动metastore。
2014-04-23 06:04:27,053 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://172.25.40.171:9083
2014-04-23 06:04:27,085 WARN hive.metastore (HiveMetaStoreClient.java:open(288)) - Failed to connect to the MetaStore Server...
org.apache.thrift.transport.TTransportException: java.net.ConnectException: 拒绝连接
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:283)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:164)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:104)
at org.apache.hive.service.cli.CLIService.start(CLIService.java:82)
at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2) Version information not found in metastore
执行“./hive --service metastore”报下面这样的错误原因是未对metastore进行初始化,需要执行一次“schematool -dbType mysql -initSchema”。
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5638)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:5622)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
at com.sun.proxy.$Proxy2.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:403)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:286)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:4263)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:4197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)