blocks|key|1499551|text|您可以通过调用project.locations.jobs.get或通过运行以下命令通过gcloud访问优化的图和融合阶段：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1499552|gcloud+dataflow+jobs+describe+--full+$JOB_ID+--format+json|code-block|syntax|javascript|1499553|根据响应的输出，融合阶段将在ExecutionStageSummary数组中的ComponentTransform对象下描述。下面是云Pub/Sub到BigQuery+Google提供的模板的示例输出。在这种情况下，我们可以看到图被融合成3个步骤，主要由Reshuffle步骤在BigQueryIO接收器中描述：|style|CODE|1499554|S03+-在WriteSuccessfulRecords和WriteFailedRecords中，在+Reshuffle之前所有的转换|ordered-list-item|1499555|S02+-在+Reshuffle+in+WriteSuccessfulRecords之后所有的转换|1499556|S01+-在+Reshuffle+in+WriteFailedRecords之后所有的转换|1499557|全输出|1499558|​|1499559|📷|atomic|1499560|1499561|由于任务描述非常冗长，您可以考虑将输出传递到jq，以便轻松提取单行命令中的相关位，如下所示：|1499562|gcloud+dataflow+jobs+describe+--full+$JOB_ID+--format+json+%7C+jq+'.pipelineDescription.executionPipelineStage[]+%7C+{"stage_id":+.id,+"stage_name":+.name,+"fused_steps":+.componentTransform+}'|1499563|entityMap|0|LINK|mutability|MUTABLE|url|https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get|1|https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#executionstagesummary|2|https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.ComponentTransform|3|https://cloud.google.com/dataflow/docs/templates/provided-templates#cloudpubsubtobigquery|4|https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html|5|https://gist.github.com/ryanmcdowell/34cd1670e0b1c386b555d6ac488e14aa|6|IMAGE|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/037159f2db687d827e8f0c7fb4d8f1f4.png|imageAlt^0|7|Q|0|0|0|3J|9|E|L|1|13|I|2|1U|H|3|3V|A|4|0|6|M|T|I|1F|9|0|7|9|K|M|0|7|9|K|I|0|0|3|5|0|0|0|1|6|0|0|M|2|0|0^^$0|@$1|2|3|4|5|6|7|1T|8|@]|9|@$A|1U|B|1V|1|1W]]|C|$]]|$1|D|3|E|5|F|7|1X|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|1Y|8|@$A|1Z|B|20|K|L]]|9|@$A|21|B|22|1|23]|$A|24|B|25|1|26]|$A|27|B|28|1|29]|$A|2A|B|2B|1|2C]]|C|$]]|$1|M|3|N|5|O|7|2D|8|@$A|2E|B|2F|K|L]|$A|2G|B|2H|K|L]|$A|2I|B|2J|K|L]]|9|@]|C|$]]|$1|P|3|Q|5|O|7|2K|8|@$A|2L|B|2M|K|L]|$A|2N|B|2O|K|L]]|9|@]|C|$]]|$1|R|3|S|5|O|7|2P|8|@$A|2Q|B|2R|K|L]|$A|2S|B|2T|K|L]]|9|@]|C|$]]|$1|T|3|U|5|6|7|2U|8|@]|9|@$A|2V|B|2W|1|2X]]|C|$]]|$1|V|3|W|5|6|7|2Y|8|@]|9|@]|C|$]]|$1|X|3|Y|5|Z|7|2Z|8|@]|9|@$A|30|B|31|1|32]]|C|$]]|$1|10|3|W|5|6|7|33|8|@]|9|@]|C|$]]|$1|11|3|12|5|6|7|34|8|@$A|35|B|36|K|L]]|9|@]|C|$]]|$1|13|3|14|5|F|7|37|8|@]|9|@]|C|$G|H]]|$1|15|3|-4|5|6|7|38|8|@]|9|@]|C|$]]]|16|$17|$5|18|19|1A|C|$1B|1C]]|1D|$5|18|19|1A|C|$1B|1E]]|1F|$5|18|19|1A|C|$1B|1G]]|1H|$5|18|19|1A|C|$1B|1I]]|1J|$5|18|19|1A|C|$1B|1K]]|1L|$5|18|19|1A|C|$1B|1M]]|1N|$5|1O|19|1P|C|$1Q|1R|1S|-4]]]]

You can access your optimized graph and fused stages by either calling <a href="https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get" rel="noreferrer">project.locations.jobs.get</a> or via gcloud by running the following command:

<pre class="lang-sh prettyprint-override"><code>gcloud dataflow jobs describe --full $JOB_ID --format json
</code></pre>

From the output of the response, the fused stages will be described under the <a href="https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#executionstagesummary" rel="noreferrer">ExecutionStageSummary</a> object within the <a href="https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.ComponentTransform" rel="noreferrer">ComponentTransform</a> array. Below is an example output from the <a href="https://cloud.google.com/dataflow/docs/templates/provided-templates#cloudpubsubtobigquery" rel="noreferrer">Cloud Pub/Sub to BigQuery</a> Google-provided template. In this case, we can see the graph was fused into 3 steps largely delineated by the <code>Reshuffle</code> step within the <a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html" rel="noreferrer">BigQueryIO</a> sink:

<ol>
<li>S03 - All transforms before the <code>Reshuffle</code> in <code>WriteSuccessfulRecords</code> and <code>WriteFailedRecords</code></li>
<li>S02 - All transforms after the <code>Reshuffle</code> in <code>WriteSuccessfulRecords</code></li>
<li>S01 - All transforms after the <code>Reshuffle</code> in <code>WriteFailedRecords</code></li>
</ol>

<a href="https://gist.github.com/ryanmcdowell/34cd1670e0b1c386b555d6ac488e14aa" rel="noreferrer">Full Output</a>

<a href="https://i.stack.imgur.com/BTzvw.png" rel="noreferrer"><img src="https://i.stack.imgur.com/BTzvw.png" alt="Job Description"></a>

Since the job description is pretty verbose, you may consider piping the output to <code>jq</code> to easily extract the relevant bits in a one-line command such as the below:

<pre class="lang-sh prettyprint-override"><code>gcloud dataflow jobs describe --full $JOB_ID --format json | jq '.pipelineDescription.executionPipelineStage[] | {"stage_id": .id, "stage_name": .name, "fused_steps": .componentTransform }'
</code></pre>

I'm trying to optimize a pipeline that pulls messages from PubSubIO and sends those messages to a 3rd party API. One interesting observation I have is that if I put a <code>GroupBy</code> and a "Degroup" transform after the <code>PubSubIO.read</code>, the throughput of pipeline increased significantly. I added the <code>GroupBy</code> just to prevent fusion optimization, and now I wonder how exactly the transform being merged in a given pipeline.

What is the best way to find out how a pipeline looks like after the fusion?

What is the best way to see how Dataflow is doing fusion optimization

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我正在尝试优化一个管道，它从PubSubIO中提取消息并将这些消息发送给第三方API。我有一个有趣的观察，如果我在GroupBy之后放置一个PubSubIO.read和一个"Degroup“转换，管道的吞吐量就会显著增加。我添加GroupBy只是为了防止融合优化，现在我想知道如何将转换合并到给定的管道中。在融合之后，找...

查看Dataflow如何进行融合优化的最佳方法是什么？-腾讯云开发者社区-腾讯云

问查看Dataflow如何进行融合优化的最佳方法是什么？
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问查看Dataflow如何进行融合优化的最佳方法是什么？EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问查看Dataflow如何进行融合优化的最佳方法是什么？
EN