如何使用JDBC将Impala表直接加载到Spark？
我正在尝试用Python编写一个spark作业，它将打开与Impala的jdbc连接，并将视图直接从Impala加载到Dataframe中。这个问题非常接近，但在scala中：Calling JDBC to impala/hive from within a spark job and creating a table
我该怎么做呢？有很多其他数据源的示例，比如MySQL、PostgreSQL等，但我还没有看到一个用于Impala + Python + Kerberos的示例。举个例子会有很大帮助。谢谢!
用网络上的信息试过了，但不起作用。
SPARK笔记本
#!/bin/bash
export PYSPARK_PYTHON=/home/anave/anaconda2/bin/python
export HADOOP_CONF_DIR=/etc/hive/conf
export PYSPARK_DRIVER_PYTHON=/home/anave/anaconda2/bin/ipython
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --ip=* --no-browser'
# use Java8
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin:$PATH
# JDBC Drivers for Impala
export CLASSPATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30/*.jar:$CLASSPATH
export JDBC_PATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30
# --jars $SRCDIR/spark-csv-assembly-1.4.0-SNAPSHOT.jar \
# --conf spark.sql.parquet.binaryAsString=true \
# --conf spark.sql.hive.convertMetastoreParquet=false
pyspark --master yarn-client \
--driver-memory 4G \
--executor-memory 2G \
# --num-executors 10 \
--jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar
--driver-class-path $JDBC_PATH/*.jarPython代码
properties = {
"driver": "com.cloudera.impala.jdbc41.Driver",
"AuthMech": "1",
# "KrbRealm": "EXAMPLE.COM",
# "KrbHostFQDN": "impala.example.com",
"KrbServiceName": "impala"
}
# imp_env is the hostname of the db, works with other impala queries ran inside python
url = "jdbc:impala:imp_env;auth=noSasl"
db_df = sqlContext.read.jdbc(url=url, table='summary', properties=properties)我收到这个错误消息(Full Error Log)：
Py4JJavaError:调用o42.jdbc时出错。：java.lang.ClassNotFoundException: com.cloudera.impala.jdbc41.Driver
回答 4
Stack Overflow用户
发布于 2016-09-22 01:48:21
您可以使用
--jars $(echo /dir/of/jars/*.jar | tr ' ' ',') 而不是
--jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar或者有关其他方法，请参阅我的answer
Stack Overflow用户
发布于 2020-02-04 22:32:01
的第一种方法是在下面的impala_jdbc_connection.py脚本上使用spark-，比如spark-submit --driver-class-path /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/ImpalaJDBC41.jar --jars /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/ImpalaJDBC41.jar --class com.cloudera.impala.jdbc41.Driver impala_jdbc_connection.py
impala_jdbc_connection.py
properties = {
"drivers": "com.cloudera.impala.jdbc41.Driver"
}
#initalize the spark session
spark = (
SparkSession.builder
.config("spark.jars.packages", "jar-packages-list")
.config("spark.sql.warehouse.dir","hdfs://dwh-hdp-node01.dev.ergo.liferunoffinsuranceplatform.com:8020/user/hive/warehouse")
.enableHiveSupport()
.getOrCreate()
)
db_df = spark.read.jdbc(url= 'jdbc:impala://host_ip_address:21050/database_name', table ='table_name', properties = properties)
db_df.show()spark 不是从impala到spark的直接导入，而是结果到spark数据帧的转换
pip install impyla来源：https://github.com/cloudera/impyla
连接到impala并从impala数据库获取结果，并将结果转换为spark dataframe
from impala.dbapi import connect
conn = connect(host = 'IP_ADDRESS_OF_HOST', port=21050)
cursor = conn.cursor()
cursor.execute('select * from database.table')
res= cursor.fetchall() # convert res to spark dataframe
for data in res:
print(data)Stack Overflow用户
发布于 2021-02-18 05:33:34
在集群库中设置jar后，在Azure Databricks notebook中执行此操作。通常遵循上一篇文章，除了d是驱动程序配置的大写字母。效果很好。
properties = {
"Driver": "com.cloudera.impala.jdbc41.Driver"
}
db_df = spark.read.jdbc(url= 'jdbc:impala://hostname.domain.net:21050/dbname;AuthMech=3;UID=xxxx;PWD=xxxx', table ='product', properties = properties)
db_df.show()https://stackoverflow.com/questions/39400101
复制相似问题
腾讯云开发者
