blocks|key|2622273|text|我们以前遇到过这个问题，花了相当多的时间来解决它。我们将其打包为docker文件并发布到github+https://github.com/michaelchanwahyan/datalab上。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2622274|它是通过修改开放源码包nbparameterize并集成传递参数(如execution_date.+)来完成的动态生成的图形也可以在笔记本中更新和保存。|style|BOLD|2622275|当它执行时|2622276|2622277|+notebook将被读取并注入参数|ordered-list-item|2622278|+notebook将被执行，并且输出将覆盖原始路径|2622279|2622280|此外，还安装和配置了spark、keras、tensorflow等常用工具。|2622281|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/michaelchanwahyan/datalab^0|1F|18|0|0|B|E|Y|E|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@$A|11|B|12|1|13]]|C|$]]|$1|D|3|E|5|6|7|14|8|@$A|15|B|16|F|G]|$A|17|B|18|F|G]]|9|@]|C|$]]|$1|H|3|I|5|6|7|19|8|@]|9|@]|C|$]]|$1|J|3|-4|5|6|7|1A|8|@]|9|@]|C|$]]|$1|K|3|L|5|M|7|1B|8|@]|9|@]|C|$]]|$1|N|3|O|5|M|7|1C|8|@]|9|@]|C|$]]|$1|P|3|-4|5|6|7|1D|8|@]|9|@]|C|$]]|$1|Q|3|R|5|6|7|1E|8|@]|9|@]|C|$]]|$1|S|3|-4|5|6|7|1F|8|@]|9|@]|C|$]]]|T|$U|$5|V|W|X|C|$Y|Z]]]]

We encountered this problem before and spent quite a couple of days to solve it.
We packaged it as a docker file and published on github <a href="https://github.com/michaelchanwahyan/datalab" rel="nofollow noreferrer">https://github.com/michaelchanwahyan/datalab</a>.

It is done by modifing an open source package nbparameterize and integrating the passing arguments such as execution_date. Graph generated on the fly can also be updated and saved within inside the notebook.

When it is executed

<ol>
<li>the notebook will be read and inject the parameters </li>
<li>the notebook is executed and the output will overwrite the original path</li>
</ol>

Besides, it also installed and configured common tools such as spark, keras, tensorflow, etc.

blocks|key|2622550|text|您也可以组合使用airflow+%2B+papermill。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2622551|造纸厂|2622552|Papermill是一个运行jupyter+notebook的工具，带参数：https://github.com/nteract/papermill运行jupyter+notebook非常简单，你可以从python脚本中运行：|2622553|import+papermill+as+pm

pm.execute_notebook(
+++'path/to/input.ipynb',
+++'path/to/output.ipynb',
+++parameters+=+dict(alpha=0.6,+ratio=0.1)
)|code-block|syntax|javascript|2622554|或从CLI：|2622555|$+papermill+local/input.ipynb+s3://bkt/output.ipynb+-p+alpha+0.6+-p+l1_ratio+0.1|2622556|它将从输入路径运行一个notebook，在输出路径中创建一个副本，并在每个单元格运行后更新此副本。|2622557|气流集成|2622558|为了将其与Airflow集成，有一个专门的造纸厂操作员用于运行参数化笔记本：https://airflow.readthedocs.io/en/latest/howto/operator/papermill.html您可以直接在DAG定义中设置相同的输入/输出/参数参数，并对aifrlow变量使用模板：|2622559|run_this+=+PapermillOperator(
++++task_id="run_example_notebook",
++++dag=dag,
++++input_nb="/tmp/hello_world.ipynb",
++++output_nb="/tmp/out-{{+execution_date+}}.ipynb",
++++parameters={"msgs":+"Ran+from+Airflow+at+{{+execution_date+}}!"}
)|2622560|entityMap|0|LINK|mutability|MUTABLE|url|https://airflow.apache.org/docs/apache-airflow/1.10.10/howto/operator/papermill.html|1|https://github.com/nteract/papermill|2|https://airflow.readthedocs.io/en/latest/howto/operator/papermill.html^0|8|J|0|0|0|12|10|1|0|0|0|0|0|0|12|1Y|2|0|0^^$0|@$1|2|3|4|5|6|7|1A|8|@]|9|@$A|1B|B|1C|1|1D]]|C|$]]|$1|D|3|E|5|6|7|1E|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|1F|8|@]|9|@$A|1G|B|1H|1|1I]]|C|$]]|$1|H|3|I|5|J|7|1J|8|@]|9|@]|C|$K|L]]|$1|M|3|N|5|6|7|1K|8|@]|9|@]|C|$]]|$1|O|3|P|5|J|7|1L|8|@]|9|@]|C|$K|L]]|$1|Q|3|R|5|6|7|1M|8|@]|9|@]|C|$]]|$1|S|3|T|5|6|7|1N|8|@]|9|@]|C|$]]|$1|U|3|V|5|6|7|1O|8|@]|9|@$A|1P|B|1Q|1|1R]]|C|$]]|$1|W|3|X|5|J|7|1S|8|@]|9|@]|C|$K|L]]|$1|Y|3|-4|5|6|7|1T|8|@]|9|@]|C|$]]]|Z|$10|$5|11|12|13|C|$14|15]]|16|$5|11|12|13|C|$14|17]]|18|$5|11|12|13|C|$14|19]]]]

You can also use combination of <a href="https://airflow.apache.org/docs/apache-airflow/1.10.10/howto/operator/papermill.html" rel="nofollow noreferrer">airflow + papermill</a>.
<h3>Papermill</h3>
Papermill is a tool for running jupyter notebooks with parameters: <a href="https://github.com/nteract/papermill" rel="nofollow noreferrer">https://github.com/nteract/papermill</a>
Running a jupyter notebook is very easy, you can do it from python script:
<pre class="lang-py prettyprint-override"><code>import papermill as pm

pm.execute_notebook(
 'path/to/input.ipynb',
 'path/to/output.ipynb',
 parameters = dict(alpha=0.6, ratio=0.1)
)
</code></pre>
or from CLI:
<pre class="lang-sh prettyprint-override"><code>$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
</code></pre>
and it will run a notebook from the input path, create a copy in the output path and update this copy after each cell run.
<h3>Airflow Integration</h3>
To integrate it with Airflow, there is a dedicated papermill operator for running parametrized notebooks: <a href="https://airflow.readthedocs.io/en/latest/howto/operator/papermill.html" rel="nofollow noreferrer">https://airflow.readthedocs.io/en/latest/howto/operator/papermill.html</a>
You can setup the same input/output/paramters arguments directly in the DAG definition and use the templating for the aifrlow variables:
<pre class="lang-py prettyprint-override"><code>run_this = PapermillOperator(
 task_id=&quot;run_example_notebook&quot;,
 dag=dag,
 input_nb=&quot;/tmp/hello_world.ipynb&quot;,
 output_nb=&quot;/tmp/out-{{ execution_date }}.ipynb&quot;,
 parameters={&quot;msgs&quot;: &quot;Ran from Airflow at {{ execution_date }}!&quot;}
)
</code></pre>

blocks|key|2622611|text|另一种选择是使用Ploomner+(免责声明:我是作者)。它使用引擎盖下的造纸厂来建造多级管道。任务可以是笔记本、脚本、函数或它们的任意组合。您可以在本地运行、Airflow或Kubernetes+(使用Argo工作流)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2622612|这是管道声明的样子：|2622613|tasks:
++-+source:+notebook.ipynb
++++product:
++++++nb:+output.html
++++++data:+output.csv

++-+source:+another.ipynb
++++product:
++++++nb:+another.html
++++++data:+another.csv|code-block|syntax|javascript|2622614|2622615|Repository|unordered-list-item|offset|length|2622616|Exporting+to+Airflow|2622617|Exporting+to+Kubernetes|2622618|Sample+pipelines|2622619|2622620|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/ploomber/ploomber|1|https://soopervisor.readthedocs.io/en/stable/airflow.html|2|https://soopervisor.readthedocs.io/en/stable/kubernetes.html|3|https://github.com/ploomber/projects^0|0|0|0|0|0|A|0|0|0|K|1|0|0|N|2|0|0|G|3|0|0^^$0|@$1|2|3|4|5|6|7|19|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1A|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|1B|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|1C|8|@]|9|@]|A|$]]|$1|J|3|K|5|L|7|1D|8|@]|9|@$M|1E|N|1F|1|1G]]|A|$]]|$1|O|3|P|5|L|7|1H|8|@]|9|@$M|1I|N|1J|1|1K]]|A|$]]|$1|Q|3|R|5|L|7|1L|8|@]|9|@$M|1M|N|1N|1|1O]]|A|$]]|$1|S|3|T|5|L|7|1P|8|@]|9|@$M|1Q|N|1R|1|1S]]|A|$]]|$1|U|3|-4|5|6|7|1T|8|@]|9|@]|A|$]]|$1|V|3|-4|5|6|7|1U|8|@]|9|@]|A|$]]]|W|$X|$5|Y|Z|10|A|$11|12]]|13|$5|Y|Z|10|A|$11|14]]|15|$5|Y|Z|10|A|$11|16]]|17|$5|Y|Z|10|A|$11|18]]]]

Another alternative is to use Ploomner (disclaimer: I'm the author). It uses papermill under the hood to build multi-stage pipelines. Tasks can be notebooks, scripts, functions, or any combination of them. You can run locally, Airflow, or Kubernetes (using Argo workflows).
This is how a pipeline declaration looks like:
<pre class="lang-yaml prettyprint-override"><code>tasks:
 - source: notebook.ipynb
 product:
 nb: output.html
 data: output.csv

 - source: another.ipynb
 product:
 nb: another.html
 data: another.csv
</code></pre>
<ul>
<li><a href="https://github.com/ploomber/ploomber" rel="nofollow noreferrer">Repository</a></li>
<li><a href="https://soopervisor.readthedocs.io/en/stable/airflow.html" rel="nofollow noreferrer">Exporting to Airflow</a></li>
<li><a href="https://soopervisor.readthedocs.io/en/stable/kubernetes.html" rel="nofollow noreferrer">Exporting to Kubernetes</a></li>
<li><a href="https://github.com/ploomber/projects" rel="nofollow noreferrer">Sample pipelines</a></li>
</ul>

My code is written in jupyter and saved as .ipynb format.

We want to use airflow to schedule the execution and define the dependencies.

How can the notebooks be executed in airflow?

I know I can convert them to python files first but the graphs generated on the fly will be difficult to handle. 

Is there are any easier solution? Thanks

How to run jupyter notebook in airflow

Kubernetes

Docker

我的代码是用jupyter编写的，并保存为.ipynb格式。我们希望使用airflow来调度执行并定义依赖项。如何在气流中执行笔记本？我知道我可以先将它们转换成python文件，但在运行时生成的图形将很难处理。有没有更简单的解决方案？谢谢

问如何在气流中运行jupyter笔记本
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在气流中运行jupyter笔记本EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在气流中运行jupyter笔记本
EN