GPU使用率百分比prometheus查询
GPU使用率百分比prometheus查询
提问于 2021-09-03 16:08:37
我能否从Prometheus的下面提到的指标中找出GPU利用率百分比？我不知道如何查询它。我没有用于PPC64lE环境的dcgm-exporter映像。您还可以共享用于制作ppc64le环境的dcgm-exporter的docker映像的链接
HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.17"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.499048e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.499048e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 4593
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 761
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.368032e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.499048e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.13696e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.760128e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 5731
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.13696e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.897088e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 6492
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 153600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 163840
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 58752
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65536
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.473924e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.037183e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.4027792e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 9
# HELP nvidia_gpu_duty_cycle Percent of time over the past sample period during which one or more kernels were executing on the GPU device
# TYPE nvidia_gpu_duty_cycle gauge
nvidia_gpu_duty_cycle{minor_number="0",name="Tesla V100-SXM2-32GB",uuid="GPU-5481fdc1-1b2c-381d-90d9-2df35fc8cecf"} 0
nvidia_gpu_duty_cycle{minor_number="1",name="Tesla V100-SXM2-32GB",uuid="GPU-af66b351-1498-c103-f39e-7592b645dc80"} 0
nvidia_gpu_duty_cycle{minor_number="2",name="Tesla V100-SXM2-32GB",uuid="GPU-95887069-482a-9a95-d02a-7c6e79c47893"} 0
nvidia_gpu_duty_cycle{minor_number="3",name="Tesla V100-SXM2-32GB",uuid="GPU-a38af12e-e2f7-ee15-b064-4628cf1fc5da"} 0
# HELP nvidia_gpu_memory_total_bytes Total memory of the GPU device in bytes
# TYPE nvidia_gpu_memory_total_bytes gauge
nvidia_gpu_memory_total_bytes{minor_number="0",name="Tesla V100-SXM2-32GB",uuid="GPU-5481fdc1-1b2c-381d-90d9-2df35fc8cecf"} 3.4089730048e+10
nvidia_gpu_memory_total_bytes{minor_number="1",name="Tesla V100-SXM2-32GB",uuid="GPU-af66b351-1498-c103-f39e-7592b645dc80"} 3.4089730048e+10
nvidia_gpu_memory_total_bytes{minor_number="2",name="Tesla V100-SXM2-32GB",uuid="GPU-95887069-482a-9a95-d02a-7c6e79c47893"} 3.4089730048e+10
nvidia_gpu_memory_total_bytes{minor_number="3",name="Tesla V100-SXM2-32GB",uuid="GPU-a38af12e-e2f7-ee15-b064-4628cf1fc5da"} 3.4089730048e+10
# HELP nvidia_gpu_memory_used_bytes Memory used by the GPU device in bytes
# TYPE nvidia_gpu_memory_used_bytes gauge
nvidia_gpu_memory_used_bytes{minor_number="0",name="Tesla V100-SXM2-32GB",uuid="GPU-5481fdc1-1b2c-381d-90d9-2df35fc8cecf"} 4.470079488e+09
nvidia_gpu_memory_used_bytes{minor_number="1",name="Tesla V100-SXM2-32GB",uuid="GPU-af66b351-1498-c103-f39e-7592b645dc80"} 2.588934144e+09
nvidia_gpu_memory_used_bytes{minor_number="2",name="Tesla V100-SXM2-32GB",uuid="GPU-95887069-482a-9a95-d02a-7c6e79c47893"} 0
nvidia_gpu_memory_used_bytes{minor_number="3",name="Tesla V100-SXM2-32GB",uuid="GPU-a38af12e-e2f7-ee15-b064-4628cf1fc5da"} 5.640290304e+09
# HELP nvidia_gpu_num_devices Number of GPU devices
# TYPE nvidia_gpu_num_devices gauge
nvidia_gpu_num_devices 4
# HELP nvidia_gpu_power_usage_milliwatts Power usage of the GPU device in milliwatts
# TYPE nvidia_gpu_power_usage_milliwatts gauge
nvidia_gpu_power_usage_milliwatts{minor_number="0",name="Tesla V100-SXM2-32GB",uuid="GPU-5481fdc1-1b2c-381d-90d9-2df35fc8cecf"} 68088
nvidia_gpu_power_usage_milliwatts{minor_number="1",name="Tesla V100-SXM2-32GB",uuid="GPU-af66b351-1498-c103-f39e-7592b645dc80"} 56426
nvidia_gpu_power_usage_milliwatts{minor_number="2",name="Tesla V100-SXM2-32GB",uuid="GPU-95887069-482a-9a95-d02a-7c6e79c47893"} 38826
nvidia_gpu_power_usage_milliwatts{minor_number="3",name="Tesla V100-SXM2-32GB",uuid="GPU-a38af12e-e2f7-ee15-b064-4628cf1fc5da"} 71068
# HELP nvidia_gpu_temperature_celsius Temperature of the GPU device in celsius
# TYPE nvidia_gpu_temperature_celsius gauge
nvidia_gpu_temperature_celsius{minor_number="0",name="Tesla V100-SXM2-32GB",uuid="GPU-5481fdc1-1b2c-381d-90d9-2df35fc8cecf"} 45
nvidia_gpu_temperature_celsius{minor_number="1",name="Tesla V100-SXM2-32GB",uuid="GPU-af66b351-1498-c103-f39e-7592b645dc80"} 46
nvidia_gpu_temperature_celsius{minor_number="2",name="Tesla V100-SXM2-32GB",uuid="GPU-95887069-482a-9a95-d02a-7c6e79c47893"} 37
nvidia_gpu_temperature_celsius{minor_number="3",name="Tesla V100-SXM2-32GB",uuid="GPU-a38af12e-e2f7-ee15-b064-4628cf1fc5da"} 51
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.02
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 22
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.6646144e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.63059385687e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.264910336e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0回答 1
Stack Overflow用户
发布于 2021-09-04 09:31:36
根据您共享的指标，以下指标将为您提供有关GPU利用率的信息：
nvidia_gpu_duty_cycle -过去采样期间一个或多个内核在GPU设备上执行的时间百分比
nvidia_gpu_memory_total_bytes -可用于GPU设备的总内存(字节
nvidia_gpu_memory_used_bytes - GPU设备使用的内存，单位为字节
nvidia_gpu_num_devices - GPU设备的数量
nvidia_gpu_power_usage_milliwatts -nvidia_gpu_power_usage_milliwatts设备的功耗，以毫瓦为单位
nvidia_gpu_temperature_celsius - GPU设备的温度，单位为摄氏度
在以Prometheus为数据源的Prometheus UI或Grafana中，可以在查询表达式中使用这些值来检索关联的GPU度量。例如，如果您要执行一个简单的查询，例如nvidia_gpu_memory_total_bytes，它将返回与此度量名称匹配的所有时间序列。
还请注意，您共享的指标包含上述每个值的4个条目，每个可用的GPU设备都有一个条目，编号为0-3。如果你只想查询一个特定设备的指标，比如说#2，你的查询需要看起来像这样：nvidia_gpu_memory_total_bytes{minor_number="2"}。请注意每个指标名称后的{}之间的各种逗号分隔标签，因为它们可用于根据您的喜好更好地过滤查询。更多关于普罗米修斯查询here的信息。
对于DCGM本身，您可以使用官方github repo的源码来构建专门针对PPC64IE的Docker镜像。instructions将首先让您创建一个单独的Docker镜像，该镜像将用于生成DCGM构建。如果为generating the DCGM build，则在执行./build.sh脚本以实现PPC64IE目标时，需要包含--arch ppc选项。
对于dcgm-exporter (github)，NVIDIA在其Docker Hub repo上提供了许多预构建的映像，并提供了here的官方文档。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/69047834
复制相关文章
相似问题
腾讯云开发者
