HBase实战（1）：使用Python连接Hbase数据库【未测试】

双面人

发布于 2019-04-10 16:21:00

2.1K0

发布于 2019-04-10 16:21:00

文章被收录于专栏：热爱IT热爱IT

使用Python连接Hbase数据库

1，Hbase下载。

下载地址：http://hbase.apache.org/downloads.html

2，本地Hbase安装

root@master:/usr/local/setup_tools# tar -zxvf hbase-2.0.0-bin.tar.gz

root@master:/usr/local/setup_tools# mv hbase-2.0.0 /usr/local/ root@master:/usr/local/setup_tools# cd /usr/local root@master:/usr/local# ls | grep hbase hbase-2.0.0 root@master:/usr/local#

root@master:/usr/local/hbase-2.0.0# vi /etc/profile export HBASE_HOME=/usr/local/hbase-2.0.0 export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$HIVE_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin:$IDEA_HOME/bin:$eclipse_HOME:$MAVEN_HOME/bin:$ALLUXIO_HOME/bin:$HBASE_HOME/bin

root@master:/usr/local/hbase-2.0.0# source /etc/profile

3，配置

修改hbase-site.xml，设置存储数据的根目录。

root@master:/usr/local/hbase-2.0.0/conf# vi hbase-site.xml

<configuration> <property> <name>hbase.rootdir</name> <value>file:///usr/local/hbase-2.0.0/data</value> </property> </configuration>

4，启动hbase

root@master:/usr/local/hbase-2.0.0# cd bin root@master:/usr/local/hbase-2.0.0/bin# ls considerAsDead.sh hbase hbase-config.cmd hbase-jruby master-backup.sh replication start-hbase.sh zookeepers.sh draining_servers.rb hbase-cleanup.sh hbase-config.sh hirb.rb region_mover.rb rolling-restart.sh stop-hbase.cmd get-active-master.rb hbase.cmd hbase-daemon.sh local-master-backup.sh regionservers.sh shutdown_regionserver.rb stop-hbase.sh graceful_stop.sh hbase-common.sh hbase-daemons.sh local-regionservers.sh region_status.rb start-hbase.cmd test root@master:/usr/local/hbase-2.0.0/bin# start-hbase.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] running master, logging to /usr/local/hbase-2.0.0/logs/hbase-root-master-master.out root@master:/usr/local/hbase-2.0.0/bin# jps 2757 Jps 2685 HMaster

5，使用hbase shell

root@master:/usr/local/hbase-2.0.0/bin# hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell Use "help" to get list of supported commands. Use "exit" to quit this interactive shell. Version 2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018 Took 0.0044 seconds hbase(main):001:0>

hbase(main):003:0> version 2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018 Took 0.0054 seconds hbase(main):004:0>

启动hbase thrift服务。

root@master:/usr/local/hbase-2.0.0/bin# hbase-daemon.sh start thrift running thrift, logging to /usr/local/hbase-2.0.0/logs/hbase-root-thrift-master.out SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. root@master:/usr/local/hbase-2.0.0/bin# jps 3332 Jps 3254 ThriftServer 2685 HMaster root@master:/usr/local/hbase-2.0.0/bin#

二，使用Python连接Hbase。

1，建立Python开发环境 ---虚拟环境。

先进行检查。

root@master:/usr/local/hbase-2.0.0/bin# pip The program 'pip' is currently not installed. You can install it by typing: apt-get install python-pip root@master:/usr/local/hbase-2.0.0/bin# python Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> exit() root@master:/usr/local/hbase-2.0.0/bin#

安装pip。

root@master:/usr/local/hbase-2.0.0/bin# apt-get install python-pip Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: diffstat hardening-includes libapt-pkg-perl libarchive-zip-perl libautodie-perl libclass-accessor-perl libemail-valid-perl libio-pty-perl libio-string-perl libipc-run-perl libipc-system-simple-perl liblist-moreutils-perl libnet-dns-perl libnet-domain-tld-perl libnet-ip-perl libparse-debianchangelog-perl libperlio-gzip-perl libsub-identify-perl libsub-name-perl libtext-levenshtein-perl patchutils t1utils Use 'apt-get autoremove' to remove them. The following extra packages will be installed: python-chardet-whl python-colorama python-colorama-whl python-distlib python-distlib-whl python-html5lib python-html5lib-whl python-pip-whl python-pkg-resources python-requests-whl python-setuptools python-setuptools-whl python-six-whl python-urllib3-whl python-wheel Suggested packages: python-genshi python-distribute python-distribute-doc Recommended packages: python-dev-all The following NEW packages will be installed: python-chardet-whl python-colorama python-colorama-whl python-distlib python-distlib-whl python-html5lib python-html5lib-whl python-pip python-pip-whl python-requests-whl python-setuptools python-setuptools-whl python-six-whl python-urllib3-whl python-wheel The following packages will be upgraded: python-pkg-resources 1 upgraded, 15 newly installed, 0 to remove and 637 not upgraded. Need to get 1,744 kB of archives. After this operation, 4,184 kB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB] Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB] 1% [2 python-chardet-whl 14.0 kB/170 kB 8%] Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB] Get:4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB] 2% [4 python-chardet-whl 38.1 kB/170 kB 22%] 6% [4 python-chardet-whl 99.9 kB/170 kB 59%] Get:5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama all 0.2.5-0.1ubuntu2 [18.4 kB] Get:6 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama-whl all 0.2.5-0.1ubuntu2 [18.2 kB] Get:7 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib all 0.1.8-1ubuntu1 [113 kB] Get:8 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib-whl all 0.1.8-1ubuntu1 [140 kB] Get:9 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib all 0.999-3~ubuntu1 [83.5 kB] Get:10 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib-whl all 0.999-3~ubuntu1 [109 kB] Get:11 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-six-whl all 1.5.2-1ubuntu1 [10.5 kB] Get:12 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-urllib3-whl all 1.7.1-1ubuntu4 [64.0 kB] Get:13 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-requests-whl all 2.2.1-1ubuntu0.3 [227 kB] Get:14 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools-whl all 3.3-1ubuntu2 [244 kB] Get:15 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip-whl all 1.5.4-1ubuntu4 [111 kB] Get:16 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-pkg-resources all 3.3-1ubuntu2 [61.9 kB] Get:17 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools all 3.3-1ubuntu2 [230 kB] Get:18 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip all 1.5.4-1ubuntu4 [97.3 kB] Get:19 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-wheel all 0.24.0-1~ubuntu1 [44.7 kB] Fetched 1,723 kB in 9min 30s (3,019 B/s) Selecting previously unselected package python-chardet-whl. (Reading database ... 156494 files and directories currently installed.) Preparing to unpack .../python-chardet-whl_2.2.1-2~ubuntu1_all.deb ... Unpacking python-chardet-whl (2.2.1-2~ubuntu1) ... Selecting previously unselected package python-colorama. Preparing to unpack .../python-colorama_0.2.5-0.1ubuntu2_all.deb ... Unpacking python-colorama (0.2.5-0.1ubuntu2) ... Selecting previously unselected package python-colorama-whl. Preparing to unpack .../python-colorama-whl_0.2.5-0.1ubuntu2_all.deb ... Unpacking python-colorama-whl (0.2.5-0.1ubuntu2) ... Selecting previously unselected package python-distlib. Preparing to unpack .../python-distlib_0.1.8-1ubuntu1_all.deb ... Unpacking python-distlib (0.1.8-1ubuntu1) ... Selecting previously unselected package python-distlib-whl. Preparing to unpack .../python-distlib-whl_0.1.8-1ubuntu1_all.deb ... Unpacking python-distlib-whl (0.1.8-1ubuntu1) ... Selecting previously unselected package python-html5lib. Preparing to unpack .../python-html5lib_0.999-3~ubuntu1_all.deb ... Unpacking python-html5lib (0.999-3~ubuntu1) ... Selecting previously unselected package python-html5lib-whl. Preparing to unpack .../python-html5lib-whl_0.999-3~ubuntu1_all.deb ... Unpacking python-html5lib-whl (0.999-3~ubuntu1) ... Selecting previously unselected package python-six-whl. Preparing to unpack .../python-six-whl_1.5.2-1ubuntu1_all.deb ... Unpacking python-six-whl (1.5.2-1ubuntu1) ... Selecting previously unselected package python-urllib3-whl. Preparing to unpack .../python-urllib3-whl_1.7.1-1ubuntu4_all.deb ... Unpacking python-urllib3-whl (1.7.1-1ubuntu4) ... Selecting previously unselected package python-requests-whl. Preparing to unpack .../python-requests-whl_2.2.1-1ubuntu0.3_all.deb ... Unpacking python-requests-whl (2.2.1-1ubuntu0.3) ... Selecting previously unselected package python-setuptools-whl. Preparing to unpack .../python-setuptools-whl_3.3-1ubuntu2_all.deb ... Unpacking python-setuptools-whl (3.3-1ubuntu2) ... Selecting previously unselected package python-pip-whl. Preparing to unpack .../python-pip-whl_1.5.4-1ubuntu4_all.deb ... Unpacking python-pip-whl (1.5.4-1ubuntu4) ... Preparing to unpack .../python-pkg-resources_3.3-1ubuntu2_all.deb ... Unpacking python-pkg-resources (3.3-1ubuntu2) over (3.3-1ubuntu1) ... Selecting previously unselected package python-setuptools. Preparing to unpack .../python-setuptools_3.3-1ubuntu2_all.deb ... Unpacking python-setuptools (3.3-1ubuntu2) ... Selecting previously unselected package python-pip. Preparing to unpack .../python-pip_1.5.4-1ubuntu4_all.deb ... Unpacking python-pip (1.5.4-1ubuntu4) ... Selecting previously unselected package python-wheel. Preparing to unpack .../python-wheel_0.24.0-1~ubuntu1_all.deb ... Unpacking python-wheel (0.24.0-1~ubuntu1) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ... Setting up python-chardet-whl (2.2.1-2~ubuntu1) ... Setting up python-colorama (0.2.5-0.1ubuntu2) ... Setting up python-colorama-whl (0.2.5-0.1ubuntu2) ... Setting up python-distlib (0.1.8-1ubuntu1) ... Setting up python-distlib-whl (0.1.8-1ubuntu1) ... Setting up python-html5lib (0.999-3~ubuntu1) ... Setting up python-html5lib-whl (0.999-3~ubuntu1) ... Setting up python-six-whl (1.5.2-1ubuntu1) ... Setting up python-urllib3-whl (1.7.1-1ubuntu4) ... Setting up python-requests-whl (2.2.1-1ubuntu0.3) ... Setting up python-setuptools-whl (3.3-1ubuntu2) ... Setting up python-pip-whl (1.5.4-1ubuntu4) ... Setting up python-pkg-resources (3.3-1ubuntu2) ... Setting up python-setuptools (3.3-1ubuntu2) ... Setting up python-pip (1.5.4-1ubuntu4) ... Setting up python-wheel (0.24.0-1~ubuntu1) ... root@master:/usr/local/hbase-2.0.0/bin#

Pip安装好之后，安装virtualenv软件，让操作系统支持虚拟环境，可以得到命令virtualenv或者pyvenv命令。

root@master:/usr/local/hbase-2.0.0/bin# pip install virtualenv Downloading/unpacking virtualenv Downloading virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB): 1.9MB downloaded Installing collected packages: virtualenv Successfully installed virtualenv Cleaning up... root@master:/usr/local/hbase-2.0.0/bin#

创建虚拟环境

root@master:/usr/local# mkdir myPythonEnv root@master:/usr/local# cd myPythonEnv root@master:/usr/local/myPythonEnv# pwd /usr/local/myPythonEnv root@master:/usr/local/myPythonEnv# Pyvenv project-env Pyvenv: command not found root@master:/usr/local/myPythonEnv# virtualenv project-env New python executable in /usr/local/myPythonEnv/project-env/bin/python Installing setuptools, pip, wheel...done. root@master:/usr/local/myPythonEnv#

进入虚拟环境：

root@master:/usr/local/myPythonEnv/project-env/bin# source /usr/local/myPythonEnv/project-env/bin/activate (project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

安装Thrift依赖包

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install thrift Collecting thrift /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings SNIMissingWarning /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecurePlatformWarning /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecurePlatformWarning Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB) 100% |████████████████████████████████| 61kB 22kB/s Collecting six>=1.7.2 (from thrift) Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Building wheels for collected packages: thrift Running setup.py bdist_wheel for thrift ... done Stored in directory: /root/.cache/pip/wheels/be/36/81/0f93ba89a1cb7887c91937948519840a72c0ffdd57cac0ae8f Successfully built thrift Installing collected packages: six, thrift Successfully installed six-1.11.0 thrift-0.11.0

安装hbase-thrift依赖包

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install hbase-thrift Collecting hbase-thrift /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings SNIMissingWarning /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecurePlatformWarning /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecurePlatformWarning Downloading https://files.pythonhosted.org/packages/89/f7/dbb6c764bb909ed361c255828701228d8c9867d541cfef84127e6f3704cc/hbase-thrift-0.20.4.tar.gz Requirement already satisfied: Thrift in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from hbase-thrift) (0.11.0) Requirement already satisfied: six>=1.7.2 in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from Thrift->hbase-thrift) (1.11.0) Building wheels for collected packages: hbase-thrift Running setup.py bdist_wheel for hbase-thrift ... done Stored in directory: /root/.cache/pip/wheels/fe/51/f2/afb7b010cd97910aa0b651d492735a38ed69a93a817444904e Successfully built hbase-thrift Installing collected packages: hbase-thrift Successfully installed hbase-thrift-0.20.4 (project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

Python连接HBASE的测试程序

(project-env) root@master:/usr/local/noc_python_201806# vi queryTest.py from thrift import Thrift from thrift.transport import TSocket from thrift.transport import TTransport from thrift.protocol import TBinaryProtocol from hbase import Hbase from hbase.ttypes import * transport = TSocket.TSocket('localhost', 9090) transport = TTransport.TBufferedTransport(transport) protocol = TBinaryProtocol.TBinaryProtocol(transport) client = Hbase.Client(protocol) transport.open() print client.getTableNames()

运行结果为，查询出表：

(project-env) root@master:/usr/local/noc_python_201806# python queryTest.py ['test', 'testHbase'] (project-env) root@master:/usr/local/noc_python_201806#

Hbase中的表

hbase(main):025:0> list TABLE test testHbase 2 row(s) Took 0.0188 seconds => ["test", "testHbase"] hbase(main):026:0>

(adsbygoogle = window.adsbygoogle || []).push({});

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018/11/06 ，如有侵权请联系 cloudcommunity@tencent.com 删除

TDSQL MySQL 版