blocks|key|472058|text|根据您的MPI实现，应该可以在同一个MPI作业中运行几个不同的程序。例如，使用OpenMPI，您可以运行|type|unstyled|depth|inlineStyleRanges|entityRanges|data|472059|+mpirun+-n+1+big_program+:+-n+20+little_program|code-block|syntax|javascript|472060|而且您应该能够使用MPI_COMM_WORLD访问这两个程序。然后，您就可以使用常用的MPI函数将数据从大程序传递到小程序。|472061|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

It should be possible, depending on your MPI implementation, to run several different programs in the same MPI job. For instance, using OpenMPI you can run

<pre><code> mpirun -n 1 big_program : -n 20 little_program
</code></pre>

and you should be able to access both programs using MPI_COMM_WORLD. From there you'd then be able to use the usual MPI functions to pass your data from the big program to the little ones.

blocks|key|830531|text|一种解决方案可能是让这两个程序驻留在不同的通信器中；一个可执行文件可以通过利用MPI-2的动态进程管理来启动这两组应用程序，而“生产者”程序通过MPI_COMM_WORLD与“消费者”应用程序通信。随后，消费者应用程序的所有IPC都必须在排除生产者部分的子通信器中运行。然而，这将意味着重写以避免直接调用MPI_COMM_WORLD。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|830532|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

One answer might be to have the two programs reside in separate communicators; a single executable could launch both sets of apps by utilizing MPI-2's dynamic process management, and the "producer" program communicate through MPI_COMM_WORLD to the "consumer" application. Subsequently, all IPC for the consumer app would have to run inside a subcommunicator that excluded the producer portion. This would mean rewriting to avoid direct calls to MPI_COMM_WORLD, however.

blocks|key|2787871|text|根据您的描述，"Program+1“不是MPI应用程序，而"Program+2”是MPI应用程序。通向解决方案的最短路径可能是在两个程序之间打开一个套接字，并以这种方式发送数据。这不需要将“程序1”修改为MPI程序。我会从"Program+1“和”Program+2:+Rank+0“之间的一个套接字开始，Rank+0将数据分配给其余的Rank。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2787872|到目前为止，有几个建议涉及启动一组异构的可执行文件作为一种可能的解决方案。不要求单个MPI作业中的所有级别都是相同的可执行文件。这要求两个可执行文件都是"MPI程序“(例如，至少包括MPI_Init和MPI_Finalize调用)。“程序1”所需的修改级别，以及无法在MPI环境之外运行它，可能会使此选项失去吸引力。|2787873|我建议您避免使用“动态过程”方法，除非您使用的是提供支持的商业实现。在MPI的开源实现中，对连接/接受的支持往往参差不齐。它可能“只是工作”，但如果它不能获得技术帮助，可能是一个开放的问题。|2787874|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

Based on your description "Program 1" is not an MPI application, and "Program 2" is an MPI application. The shortest path to a solution is likely to open a socket between the two programs and send the data that way. This does not require that "Program 1" be modified to be an MPI Program. I would begin with a socket between "Program 1" and " Program 2 : Rank 0", with Rank 0 distributing the data to the remaining ranks. 

Several suggestions so far have involved launching a heterogeneous set of executables as one possible solution. There is no requirement that all the ranks in a single MPI job be the same executable. This requires that both executables be "MPI Programs" (e.g. include at least MPI_Init, and MPI_Finalize calls). The level of modification required to "Program 1", and the inability to run it outside of the MPI environment, may make this option unattractive. 

I would recommend that you avoid the "dynamic process" approach, unless you are using a commercial implementation that offers support. Support for connect/accept tends to be spotty in the open source implementations of MPI. It may "just work", but getting technical help if it does not can be an open ended problem.

blocks|key|836619|text|混合套接字和MPI不是一个好主意。实现这一点的最简单方法是将进程1和进程2都移入到单个MPI应用程序中。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|836620|实现这一点的最好方法是使用称为-+MPMD或多程序多数据的编程模型。顾名思义，您的MPI应用程序将有多个程序对多个数据进行操作。即使程序1不是MPI应用程序，也不需要做太多更改。只需调用MPI_Init并添加用于发送/接收数据的例程。您可以将其视为一种主从模型，其中Prg1是主服务器，而rest是从主服务器获取要处理的数据片段。|836621|另一种方法是通过使程序1与程序2相同来实现一个工作者池，每个人都读取数据文件的一部分并开始工作。但是您已经排除了文件IO，所以我假设proc2-n在运行时没有访问文件的权限。Master-Slave将最好地满足您的需求。|836622|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

It is not a good idea to mix sockets &amp; MPI. The easiest way to achieve this is moving both process 1 &amp; process 2 in to a single MPI application.

The best way to implement this is to use a programming model called - MPMD or Multi-Program Multi-Data. As the name implies, your MPI application will have multiple programs operating on multiples of data. Even if Program 1 is not an MPI application, you need not make too many changes. Just call MPI_Init &amp; add the routines for send/recv data. You can think of this as a sort of a Master-Slave model where Prg1 is master and rest are slaves getting pieces of data to work on from the master. 

Another method could be to implement a pool of workers by making program 1 same as program 2 and everyone reads a part of the data file and starts working. But you ruled out file IO, so I assume prog2-n do not have access to the file at run time. Master-Slave will work best for your needs.

I have the following problem:

Program 1 has a huge amount of data, say 10GB. The data in question consists of large integer- and double-arrays. 
Program 2 has 1..n MPI processes that use tiles of this data to compute results.

How can I send the data from program 1 to the MPI Processes?

Using File I/O is out of question. The compute node has sufficient RAM.

Communicate between separate MPI-Programs

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有以下问题：程序1有大量的数据，比如10 of。有问题的数据由大型整数数组和双精度数组组成。程序2有1..n个MPI进程，这些进程使用这些数据的平铺来计算结果。如何将程序1中的数据发送到MPI进程？使用文件I/O是毫无疑问的。计算节点具有足够的RAM。

问在独立的MPI程序之间进行通信
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在独立的MPI程序之间进行通信EN