The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
Help & Documentation>Data Lake Compute

Submitting Tasks Using Apache Airflow to Schedule the DLC Engine

Last updated: 2026-05-20 17:47:04
This document introduces DLC's support for the Apache Airflow scheduling tool and provides examples to demonstrate how to run various types of DLC engine tasks using Apache Airflow.

Background Information

Apache Airflow is a scheduling tool open-sourced by Airbnb. It is written in Python and uses Directed Acyclic Graphs (DAGs) to define and schedule a set of dependent jobs. It supports sub-jobs written in Python and provides various Operators to execute tasks, such as Bash commands, Python functions, SQL queries, and Spark jobs, offering high flexibility and scalability. Apache Airflow is widely used in fields such as data engineering, data processing, and workflow automation. With the rich features and visual interface provided by Apache Airflow, users can easily monitor and manage the status and execution of workflows. For more information about Apache Airflow, see Apache Airflow.

Prerequisites

1. Apache Airflow Environment Preparation.
1.1 Install and start Apache Airflow. For more operations on installing and starting Apache Airflow, see Apache Airflow Quick Start.
1.2 Install the jaydebeapi dependency package: pip install jaydebeapi.
2. Prepare the DLC environment.
2.1 Activate the DLC engine service.
2.2 If you use the standard Spark engine, prepare the Hive JDBC driver and click to download hive-jdbc-3.1.2-standalone.jar.
2.3 If you use the standard Presto engine, prepare the Presto JDBC driver and click to download presto-jdbc-0.284.jar.
2.4 If you use the SuperSQL engine, prepare the DLC JDBC driver and click to download the JDBC driver.
Note:
The Presto engine is deprecated and only available for existing users.

Key Steps

Creating a Connection and Scheduling a Task

In the Apache Airflow working directory, create a dags directory. Then, within the dags directory, create a scheduling script and save it as a .py file. For example, this document creates the scheduling script /root/airflow/dags/airflow-dlc-test.py as shown below:
import time from datetime import datetime, timedelta import jaydebeapi from airflow import DAG from airflow.operators.python_operator import PythonOperator jdbc_url='jdbc:dlc:dlc.tencentcloudapi.com?task_type=SparkSQLTask&database_name={dataBaseName}&datasource_connection_name={dataSourceName}&region={region}&data_engine_name={engineName}' user = 'xxx' pwd = 'xxx' dirver = 'com.tencent.cloud.dlc.jdbc.DlcDriver' jar_file = '/root/airflow/jars/dlc-jdbc-2.5.3-jar-with-dependencies.jar' def createTable(): sqlStr = 'create table if not exists db.tb1 (c1 int, c2 string)' conn = jaydebeapi.connect(dirver, jdbc_url, [user, pwd], jar_file) curs = conn.cursor() curs.execute(sqlStr) rows = curs.rowcount.real if rows != 0: result = curs.fetchall() print(result) curs.close() conn.close() def insertValues(): sqlStr = "insert into db.tb1 values (111, 'this is test')" conn = jaydebeapi.connect(dirver,jdbc_url, [user, pwd], jar_file) curs = conn.cursor() curs.execute(sqlStr) rows = curs.rowcount.real if rows != 0: result = curs.fetchall() print(result) curs.close() conn.close() def selectColums(): sqlStr = 'select * from db.tb1' conn = jaydebeapi.connect(dirver, jdbc_url, [user, pwd], jar_file) curs = conn.cursor() curs.execute(sqlStr) rows = curs.rowcount.real if rows != 0: result = curs.fetchall() print(result) curs.close() conn.close() def get_time(): print('Current time is:', datetime.now().strftime('%Y-%m-%d %H:%M:%S')) return time.time() default_args = { 'owner': 'tencent', # Owner name 'start_date': datetime(2024, 11, 1), # The first execution start time, in UTC 'retries': 2, # Number of retries upon failure 'retry_delay': timedelta(minutes=1), # Retry interval upon failure } dag = DAG( dag_id='airflow_dlc_test', # DAG id, must consist entirely of letters, numbers, and underscores default_args=default_args, # Externally defined parameters in dictionary format schedule_interval=timedelta(minutes=1), # Defines the DAG execution frequency, configurable in days, weeks, hours, minutes, seconds, or milliseconds catchup=False # When executing the DAG, all tasks that should have been executed from the start time to the present are executed. The default is True. ) t1 = PythonOperator( task_id='create_table', python_callable=createTable, dag=dag) t2 = PythonOperator( task_id='insert_values', python_callable=insertValues, dag=dag) t3 = PythonOperator( task_id='select_values', python_callable=selectColums, dag=dag) t4 = PythonOperator( task_id='print_time', python_callable=get_time, dag=dag) t1 >> t2 >> [t3, t4]
Parameter description:
Parameter
Description
jdbc_url
The JDBC connection URL and configuration parameters. For details, see Hive JDBC Access, Presto JDBC Access, and DLC JDBC Access.
user
SecretId
pwd
SecretKey
driver
Load the JDBC driver. For details, see Hive JDBC Access, Presto JDBC Access, and DLC JDBC Access.
jar_file
The storage path for the driver jar file. Replace it with the absolute path where the corresponding engine's JDBC driver jar file is stored. For details, see Hive JDBC Access, Presto JDBC Access, and DLC JDBC Access.

Running a Scheduled Task

You can go to the Web page, locate the submitted scheduling workflow under the DAGs tab, and start the scheduling.




Viewing Task Execution Results