blocks|key|2458087|text|Tensorflow不自动使用所有GPU，它只使用一个gpu，特别是第一个gpu+/gpu:0
您必须编写多个gpus代码来利用所有可用的gpus。cifar+mutli-gpu示例|type|ordered-list-item|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2458088|每0.1秒检查一次使用情况
watch+-n0.1+nvidia-smi|2458089|unstyled|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/petewarden/tensorflow_makefile/blob/master/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py^0|15|6|22|H|0|0|E|M|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]]|D|@$9|T|A|U|1|V]]|E|$]]|$1|F|3|G|5|6|7|W|8|@$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|H|3|-4|5|I|7|Z|8|@]|D|@]|E|$]]]|J|$K|$5|L|M|N|E|$O|P]]]]

<ol>
<li>Tensorflow automatically doesn't utilize all GPUs, it will use only one GPU, specifically first gpu <code>/gpu:0</code>

You have to write multi gpus code to utilize all gpus available. <a href="https://github.com/petewarden/tensorflow_makefile/blob/master/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">cifar mutli-gpu example</a></li>
<li>to check usage every 0.1 seconds

<code>watch -n0.1 nvidia-smi</code></li>
</ol>

blocks|key|1166413|text|如果没有其他指示，启用GPU的TensorFlow安装将默认使用第一个可用的GPU+(只要您安装了Nvidia驱动程序和CUDA8.0，并且GPU有必要的计算能力，根据医生的说法是3.0)。如果您想使用更多的GPU，则需要在图形中使用tf.device指令(更多关于它的这里)。|type|ordered-list-item|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1166414|检查GPU使用情况的最简单方法是控制台工具nvidia-smi。然而，与top或其他类似的程序不同，它只显示当前的使用和完成。正如注释中所建议的那样，您可以使用类似于watch+-n1+nvidia-smi的东西来连续地重新运行程序(在这种情况下，每秒钟运行一次)。|1166415|unstyled|entityMap|0|LINK|mutability|MUTABLE|url|https://developer.nvidia.com/cuda-gpus|1|https://www.tensorflow.org/install/install_linux|2|https://www.tensorflow.org/api_docs/python/tf/device|3|https://www.tensorflow.org/tutorials/using_gpu|4|http://manpages.ubuntu.com/manpages/precise/man1/alt-nvidia-current-smi.1.html^0|39|9|25|4|0|2A|7|1|39|9|2|3R|2|3|0|L|A|10|3|2B|K|L|A|4|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]|$9|14|A|15|1|16]|$9|17|A|18|1|19]|$9|1A|A|1B|1|1C]]|E|$]]|$1|F|3|G|5|6|7|1D|8|@$9|1E|A|1F|B|C]|$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]]|D|@$9|1K|A|1L|1|1M]]|E|$]]|$1|H|3|-4|5|I|7|1N|8|@]|D|@]|E|$]]]|J|$K|$5|L|M|N|E|$O|P]]|Q|$5|L|M|N|E|$O|R]]|S|$5|L|M|N|E|$O|T]]|U|$5|L|M|N|E|$O|V]]|W|$5|L|M|N|E|$O|X]]]]

<ol>
<li>If no other indication is given, a GPU-enabled TensorFlow installation will default to use the first available GPU (as long as you have the Nvidia driver and CUDA 8.0 installed and the GPU has the necessary <a href="https://developer.nvidia.com/cuda-gpus" rel="nofollow noreferrer">compute capability</a>, which, <a href="https://www.tensorflow.org/install/install_linux" rel="nofollow noreferrer">according to the docs</a> is 3.0). If you want to use more GPUs, you need to use <a href="https://www.tensorflow.org/api_docs/python/tf/device" rel="nofollow noreferrer"><code>tf.device</code></a> directives in your graph (more about it <a href="https://www.tensorflow.org/tutorials/using_gpu" rel="nofollow noreferrer">here</a>).</li>
<li>The easiest way to check the GPU usage is the console tool <a href="http://manpages.ubuntu.com/manpages/precise/man1/alt-nvidia-current-smi.1.html" rel="nofollow noreferrer"><code>nvidia-smi</code></a>. However, unlike <code>top</code> or other similar programs, it only shows the current usage and finishes. As suggested in the comments, you can use something like <code>watch -n1 nvidia-smi</code> to re-run the program continuously (in this case every second).</li>
</ol>

blocks|key|2458276|text|我建议nvtop，它显示实时状态，比nvidia更容易观看。它也显示在一个图表中。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|2458277|$+sudo+apt+install+nvtop
$+nvtop|code-block|syntax|javascript|2458278|​|2458279|📷|atomic|2458280|2458281|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/5247cd9bc9cc361598c1d150720174a8.png|imageAlt^0|3|5|0|0|0|0|1|0|0|0^^$0|@$1|2|3|4|5|6|7|Z|8|@$9|10|A|11|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|12|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|13|8|@]|D|@]|E|$]]|$1|M|3|N|5|O|7|14|8|@]|D|@$9|15|A|16|1|17]]|E|$]]|$1|P|3|L|5|6|7|18|8|@]|D|@]|E|$]]|$1|Q|3|-4|5|6|7|19|8|@]|D|@]|E|$]]]|R|$S|$5|T|U|V|E|$W|X|Y|-4]]]]

I would suggest nvtop, it shows real-time status and easier to watch than nvidia-smi. It also shows in a graph.
<pre><code>$ sudo apt install nvtop
$ nvtop
</code></pre>
<a href="https://i.stack.imgur.com/1kDWX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1kDWX.png" alt="enter image description here" /></a>

blocks|key|2496043|text|尝试以下命令：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2496044|nvidia-smi+--query-gpu=utilization.gpu+--format=csv+--loop=1|code-block|syntax|javascript|2496045|下面是一个演示：|2496046|​|2496047|📷|atomic|offset|length|2496048|2496049|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/81b97fc0d59bf3e463c025bc39c03e1e.gif|imageAlt^0|0|0|0|0|0|1|0|0|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|10|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|11|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|12|8|@]|9|@]|A|$]]|$1|K|3|L|5|M|7|13|8|@]|9|@$N|14|O|15|1|16]]|A|$]]|$1|P|3|J|5|6|7|17|8|@]|9|@]|A|$]]|$1|Q|3|-4|5|6|7|18|8|@]|9|@]|A|$]]]|R|$S|$5|T|U|V|A|$W|X|Y|-4]]]]

Try this command: 

<pre><code>nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1
</code></pre>

Here is a demo: 

<a href="https://i.stack.imgur.com/71LAt.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/71LAt.gif" alt="enter image description here"></a>

blocks|key|2495982|text|上面的所有命令都使用watch，通过使用builin：nvidia-smi+-l+1来保持上下文的活动效率要高得多。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2495983|如果您想同时看到类似htop和nvidia-smi的东西，可以尝试一瞥+(pip+install+glances)。|2495984|entityMap|0|LINK|mutability|MUTABLE|url|https://nicolargo.github.io/glances/^0|A|5|R|F|0|A|4|F|A|11|J|X|2|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@$9|Q|A|R|B|C]|$9|S|A|T|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]]|E|$]]|$1|H|3|-4|5|6|7|14|8|@]|D|@]|E|$]]]|I|$J|$5|K|L|M|E|$N|O]]]]

All the above commands use <code>watch</code>, it's much more efficient to keep the context alive by using the builin looper:
<code>nvidia-smi -l 1</code>.

If you want to see something like <code>htop</code> and <code>nvidia-smi</code> at the same time, you can try <a href="https://nicolargo.github.io/glances/" rel="nofollow noreferrer">glances</a> (<code>pip install glances</code>).

blocks|key|1166491|text|如果您正在使用GCP，请查看此脚本，它允许您在StackDriver中监视GPU的利用率，您还可以使用它来使用nvidia-smi+-l+5命令收集nvidia数据，并报告这些统计数据以供您跟踪。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1166492|https://github.com/GoogleCloudPlatform/ml-on-gcp/tree/master/dlvm/gcp-gpu-utilization-metrics|1166493|​|1166494|📷|atomic|1166495|1166496|entityMap|0|LINK|mutability|MUTABLE|url|1|IMAGE|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/b9806b31d3ca8fa9ce5369508d2cb347.png|imageAlt^0|1J|F|0|0|2L|0|0|0|0|1|1|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|13|8|@]|D|@$9|14|A|15|1|16]]|E|$]]|$1|H|3|I|5|6|7|17|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|18|8|@]|D|@$9|19|A|1A|1|1B]]|E|$]]|$1|M|3|I|5|6|7|1C|8|@]|D|@]|E|$]]|$1|N|3|-4|5|6|7|1D|8|@]|D|@]|E|$]]]|O|$P|$5|Q|R|S|E|$T|G]]|U|$5|V|R|W|E|$X|Y|Z|-4]]]]

If you are using GCP, please take a look at this script which allows you to monitor GPU utilization in StackDriver, you can also use it to collect nvidia-smi data using <code>nvidia-smi -l 5</code> command and reporting those statistics for you to track.

<a href="https://github.com/GoogleCloudPlatform/ml-on-gcp/tree/master/dlvm/gcp-gpu-utilization-metrics" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/ml-on-gcp/tree/master/dlvm/gcp-gpu-utilization-metrics</a>

<a href="https://i.stack.imgur.com/4IMm0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4IMm0.png" alt="enter image description here"></a>

blocks|key|1166517|text|您应该使用nvidia-smi。请记住，根据工作负载的不同，如果任务在两个采样事件之间完成，那么负载可能不会发生任何变化。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1166518|还请记住，最大采样间隔为1/6秒，如下所示：http://manpages.org/nvidia-smi|1166519|利用率报告每个GPU随着时间的推移有多忙，并且可以用来确定应用程序在系统中使用GPU的程度。注意:在驱动程序初始化期间，当ECC启用时，可以看到较高的GPU和内存利用率读数。这是由在驱动程序初始化期间执行的ECC内存清理机制造成的。
在过去的示例期间，GPU在GPU上执行一个或多个内核的时间百分比。根据产品的不同，样品周期可能介于1秒至1/6秒之间。
过去读取或写入全局(设备)内存的示例期间的内存百分比。根据产品的不同，样品周期可能介于1秒至1/6秒之间。|blockquote|1166520|entityMap|0|LINK|mutability|MUTABLE|url|http://manpages.org/nvidia-smi^0|5|A|0|M|U|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|V|8|@]|D|@$9|W|A|X|1|Y]]|E|$]]|$1|H|3|I|5|J|7|Z|8|@]|D|@]|E|$]]|$1|K|3|-4|5|6|7|10|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]]]

You should use <code>nvidia-smi</code>. Just keep in mind that depending on your workload you might not see any change in the load if the task completes between 2 sampling events.

Also keep in mind that the maximum sampling interval is 1/6 second as per: <a href="http://manpages.org/nvidia-smi" rel="nofollow noreferrer">http://manpages.org/nvidia-smi</a>

<blockquote>
 Utilization rates report how busy each GPU is over time, and can be used to determine how much an application is using the GPUs in the system.
 Note: During driver initialization when ECC is enabled one can see high GPU and Memory Utilization readings. This is caused by ECC Memory Scrubbing mechanism that is performed during driver initialization.
 
 GPU
 Percent of time over the past sample period during which one or more kernels was executing on the GPU. The sample period may be between 1 second and 1/6 second depending on the product.
 
 Memory
 Percent of time over the past sample period during which global (device) memory was being read or written. The sample period may be between 1 second and 1/6 second depending on the product.
</blockquote>

I am new to Ubuntu and GPUs and have recently been using a new PC with Ubuntu 16.04 and 4 NVIDIA 1080ti GPUs in our lab. The machine also has an i7 16 core processor. 

I have some basic questions:

<ol>
<li>Tensorflow is installed for GPU. I presume then, that it automatically prioritises GPU usage? If so, does it use all 4 together or does it use 1 and then recruit another if needed?</li>
<li>Can I monitor in real-time, the GPU use/activity during training of a model?</li>
</ol>

I fully understand this is basic hardware stuff but clear definitive answers to these specific questions would be great. 

EDIT:

Based on this output - it this really saying that nearly all the memory on each one of my GPUs is being used? 

<a href="https://i.stack.imgur.com/11zI5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/11zI5.png" alt="enter image description here"></a>

Tensorflow: How do you monitor GPU performance during model training in real-time?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

我是Ubuntu和GPU的新手，最近在我们的实验室使用了Ubuntu 16.04和4 NVIDIA 1080 to的新PC。这台机器还有一个i7 16核心处理器。我有一些基本问题：Tensorflow是为GPU安装的。那么，我猜想它会自动优先考虑GPU的使用吗？如果是这样的话，它是使用所有4在一起还是使用1，然后在需要时再招募另一个？我能实时监控GPU在模型训练中的使用/活动吗？我完全理解这是基本

问Tensorflow:如何实时监控GPU在模型培训中的性能？
EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Tensorflow:如何实时监控GPU在模型培训中的性能？EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Tensorflow:如何实时监控GPU在模型培训中的性能？
EN