The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
This document introduces DLC's support for the Apache Airflow scheduling tool and provides examples to demonstrate how to run various types of DLC engine tasks using Apache Airflow.
Background Information
Apache Airflow is a scheduling tool open-sourced by Airbnb. It is written in Python and uses Directed Acyclic Graphs (DAGs) to define and schedule a set of dependent jobs. It supports sub-jobs written in Python and provides various Operators to execute tasks, such as Bash commands, Python functions, SQL queries, and Spark jobs, offering high flexibility and scalability. Apache Airflow is widely used in fields such as data engineering, data processing, and workflow automation. With the rich features and visual interface provided by Apache Airflow, users can easily monitor and manage the status and execution of workflows. For more information about Apache Airflow, see Apache Airflow.
Prerequisites
1. Apache Airflow Environment Preparation.
1.1 Install and start Apache Airflow. For more operations on installing and starting Apache Airflow, see Apache Airflow Quick Start.
1.2 Install the jaydebeapi dependency package: pip install jaydebeapi.
The Presto engine is deprecated and only available for existing users.
Key Steps
Creating a Connection and Scheduling a Task
In the Apache Airflow working directory, create a dags directory. Then, within the dags directory, create a scheduling script and save it as a .py file. For example, this document creates the scheduling script /root/airflow/dags/airflow-dlc-test.py as shown below:
import time
from datetime import datetime, timedelta
import jaydebeapi
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
jdbc_url='jdbc:dlc:dlc.tencentcloudapi.com?task_type=SparkSQLTask&database_name={dataBaseName}&datasource_connection_name={dataSourceName}®ion={region}&data_engine_name={engineName}'
user ='xxx'
pwd ='xxx'
dirver ='com.tencent.cloud.dlc.jdbc.DlcDriver'
jar_file ='/root/airflow/jars/dlc-jdbc-2.5.3-jar-with-dependencies.jar'defcreateTable():
sqlStr ='create table if not exists db.tb1 (c1 int, c2 string)'
conn = jaydebeapi.connect(dirver, jdbc_url,[user, pwd], jar_file)
curs = conn.cursor()
curs.execute(sqlStr)
rows = curs.rowcount.real
if rows !=0:
result = curs.fetchall()print(result)
curs.close()
conn.close()definsertValues():
sqlStr ="insert into db.tb1 values (111, 'this is test')"
conn = jaydebeapi.connect(dirver,jdbc_url,[user, pwd], jar_file)
curs = conn.cursor()
curs.execute(sqlStr)
rows = curs.rowcount.real
if rows !=0:
result = curs.fetchall()print(result)
curs.close()
conn.close()defselectColums():
sqlStr ='select * from db.tb1'
conn = jaydebeapi.connect(dirver, jdbc_url,[user, pwd], jar_file)
curs = conn.cursor()
curs.execute(sqlStr)
rows = curs.rowcount.real
if rows !=0:
result = curs.fetchall()print(result)
curs.close()
conn.close()defget_time():print('Current time is:', datetime.now().strftime('%Y-%m-%d %H:%M:%S'))return time.time()
default_args ={'owner':'tencent',# Owner name'start_date': datetime(2024,11,1),# The first execution start time, in UTC'retries':2,# Number of retries upon failure'retry_delay': timedelta(minutes=1),# Retry interval upon failure}
dag = DAG(
dag_id='airflow_dlc_test',# DAG id, must consist entirely of letters, numbers, and underscores
default_args=default_args,# Externally defined parameters in dictionary format
schedule_interval=timedelta(minutes=1),# Defines the DAG execution frequency, configurable in days, weeks, hours, minutes, seconds, or milliseconds
catchup=False# When executing the DAG, all tasks that should have been executed from the start time to the present are executed. The default is True.)
t1 = PythonOperator(
task_id='create_table',
python_callable=createTable,
dag=dag)
t2 = PythonOperator(
task_id='insert_values',
python_callable=insertValues,
dag=dag)
t3 = PythonOperator(
task_id='select_values',
python_callable=selectColums,
dag=dag)
t4 = PythonOperator(
task_id='print_time',
python_callable=get_time,
dag=dag)
t1 >> t2 >>[t3, t4]
The storage path for the driver jar file. Replace it with the absolute path where the corresponding engine's JDBC driver jar file is stored. For details, see Hive JDBC Access, Presto JDBC Access, and DLC JDBC Access.
Running a Scheduled Task
You can go to the Web page, locate the submitted scheduling workflow under the DAGs tab, and start the scheduling.