Airflow Scheduler为schedule_interval为None的DAG抛出错误
Airflow Scheduler为schedule_interval为None的DAG抛出错误
提问于 2020-04-08 16:09:36
我对气流有问题。有一个客户生成器脚本，它接受来自yaml文件的输入并加载DAG。当所有DAG yaml文件的调度间隔都不是"None“时，它工作得很好。有许多DAG的schedule_interval为None，很少有DAG有@once。
YAML文件示例如下：
cluster:
nodes: 10
subnet: "subnet-A"
instance: "m4.2xlarge"
configbucket: "bucketabc"
jar: "s3://xxxxx.jar"
conf: "app.conf"
schedule:
state: "unpause"
concurrency: 10
startdate: "2050-08-05 00:00"
cron: "None"生成器脚本具有以下内容-
if "schedule" in project_settings:
schedule_settings = project_settings["schedule"]
concurrency = schedule_settings["concurrency"]
cron = schedule_settings["cron"]
startdate = datetime.strptime(schedule_settings["startdate"], "%Y-%m-%d %H:%M")
#print "my projectname is: " + project
dag = DAG(
dag_id = project,
default_args=args,
user_defined_macros=user_macros,
schedule_interval=cron,
concurrency=concurrency,
start_date=startdate
)当schedule_interval=None中有多个DAG时，我收到的错误
INFO - [2020-04-08 12:30:45,529] {dagbag.py:302} ERROR - Failed to bag_dag: /home/deploy/airflow/dags/genertor.py
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 296, in process_file
croniter(dag._schedule_interval)
File "/usr/local/lib/python3.6/site-packages/croniter/croniter.py", line 91, in __init__
self.expanded, self.nth_weekday_of_month = self.expand(expr_format)
File "/usr/local/lib/python3.6/site-packages/croniter/croniter.py", line 468, in expand
raise CroniterBadCronError(cls.bad_length)
croniter.croniter.CroniterBadCronError: Exactly 5 or 6 columns has to be specified for iteratorexpression.有人遇到过这个问题吗？
回答 1
Stack Overflow用户
回答已采纳
发布于 2020-04-08 16:37:44
Airflow DAG schedule_interval可以是作为string的cron表达式，也可以是None (NB而不是string "None")。
在您的设置中，您具有：
cron: "None"这是Python中的字符串。如果您不能将YAML文件更改为：
cron: None您仍然可以在DAG本身中检查该字符串：
schedule_interval = None if cron == "None" else cron页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/61096115
复制相关文章
相似问题
腾讯云开发者
