Supported Engine Types
Standard Spark Engine
Preparing the Environment
Dependency: JDK 1.8
JDBC Download: Click to download hive-jdbc-3.1.2-standalone.jar.
Connecting to a Standard Spark Engine
Creating a Service Access Link
Go to the Data Engine page, click the gateway details button, and then go to the gateway details page:
Click Create Private Link, select the VPC and subnet where the submission server resides, and then click Create. Two access links are generated, one for the Hive2 protocol and the other for the Presto protocol. Use the Hive2 protocol to connect to the Standard Spark Engine, as shown in the following figure.
Note:
Creating a Private Link establishes network connectivity between the engine network and the selected VPC. The submission server can be any accessible server within the selected VPC and is used solely for task submission. If there is no submission server in the selected VPC, you can create a new server to serve as the submission server.
jdbc:hive2://{endpoint}:10009/?spark.engine={DataEngineName};spark.resourcegroup={ResourceGroupName};secretkey={SecretKey};secretid={SecretId};region={Region};kyuubi.engine.type=SPARK_SQL;kyuubi.engine.share.level=ENGINE
Loading the JDBC Driver
Class.forName("org.apache.hive.jdbc.HiveDriver");
Creating a Connection via DriverManager
jdbc:hive2://{endpoint}:10009/?spark.engine={DataEngineName};spark.resourcegroup={ResourceGroupName};secretkey={SecretKey};secretid={SecretId};region={Region};kyuubi.engine.type=SPARK_SQL;kyuubi.engine.share.level=ENGINEProperties properties = new Properties();properties.setProperty("user", {AppId});Connection cnct = DriverManager.getConnection(url, properties);
JDBC Connection String Parameters
Parameter | Required | Description |
spark.engine | Yes | Standard Spark Engine Name |
spark.resourcegroup | No | Standard Spark engine resource group name. If not specified, temporary resources are created. |
secretkey | Yes | The SecretKey in TencentCloud API Key Management |
secretid | Yes | The SecretId in TencentCloud API Key Management |
region | Yes | Region. Currently, the DLC service supports ap-nanjing, ap-beijing, ap-beijing-fsi, ap-guangzhou, and ap-shanghai. ap-chengdu,ap-chongqing, na-siliconvalley, ap-singapore, ap-hongkong, na-ashburn, eu-frankfurt, ap-shanghai-fsi |
kyuubi.engine.type | Yes | Fixed value: SparkSQLTask |
kyuubi.engine.share.level | Yes | Fixed value: ENGINE |
user | Yes | User AppID. |
Complete Data Query Example
import org.apache.hive.jdbc.HiveStatement;import java.sql.*;import java.util.Properties;public class TestStandardSpark {public static void main(String[] args) throws SQLException {try {Class.forName("org.apache.hive.jdbc.HiveDriver");} catch (ClassNotFoundException e) {e.printStackTrace();return;}String url = "jdbc:hive2://{endpoint}:10009/?spark.engine={DataEngineName};spark.resourcegroup={ResourceGroupName};secretkey={SecretKey};secretid={SecretId};region={Region};kyuubi.engine.type=SPARK_SQL;kyuubi.engine.share.level=ENGINE";Properties properties = new Properties();properties.setProperty("user", {AppId});Connection connection = DriverManager.getConnection(url, properties);HiveStatement statement = (HiveStatement) connection.createStatement();String sql = "SELECT * FROM dlc_test LIMIT 100";statement.execute(sql);ResultSet rs = statement.getResultSet();while (rs.next()) {System.out.println(rs.getInt(1) + ":" + rs.getString(2));}rs.close();statement.close();connection.close();}}
After compilation is complete, you can submit the jar package to the submission server for execution.