01 / ARTICLE
GPU Utilization Monitoring with DCGM: Beyond nvidia-smi
nvidia-smi reports GPU utilization as a binary: was a kernel running? DCGM gives you the actual metrics you need for capacity planning.
For most engineers, checking GPU health begins and ends with nvidia-smi. While it provides a quick snapshot of memory allocation and current processes, relying on it for capacity planning or workload optimization is fundamentally flawed. nvidia-smi reports utilization essentially as a binary metric: during the sample period, was a kernel running on the GPU? It does not tell you if that kernel was effectively saturating the hardware or merely utilizing a fraction of the available compute.
To accurately assess GPU workloads, infrastructure teams must deploy the Data Center GPU Manager (DCGM). DCGM bypasses superficial metrics and interfaces directly with the hardware performance counters, providing high-fidelity data on how the silicon is actually being utilized.
The distinction between nvidia-smi utilization and DCGM’s Streaming Multiprocessor (SM) utilization is critical. An inference workload might show 100% utilization in nvidia-smi because it is constantly processing requests, but DCGM might reveal an SM active metric (DCGM_FI_PROF_SM_ACTIVE) of only 20%. This indicates that while the GPU is constantly working, the workload is not parallelized enough to saturate the compute units. This is the exact scenario where Multi-Instance GPU (MIG) partitioning becomes highly effective, allowing you to slice the underutilized GPU and run multiple workloads concurrently.
DCGM also provides crucial visibility into specialized hardware utilization. For modern ML workloads, tensor cores do the heavy lifting. Monitoring DCGM_FI_PROF_PIPE_TENSOR_ACTIVE confirms whether your models are actually leveraging the tensor cores, which requires specific precision formats like FP16 or BF16. If this metric is zero, your highly expensive A100 is effectively functioning as a legacy GPU.
Furthermore, monitoring memory bandwidth and footprint via DCGM_FI_DEV_FB_USED and related PCIe metrics is essential for diagnosing bottlenecks. In many training scenarios, the compute units sit idle waiting for data transfer, a problem that nvidia-smi masks entirely.
Implementing this visibility in a Kubernetes environment is straightforward using the DCGM exporter for Prometheus. By scraping these metrics and visualizing them in Grafana, teams can set precise alerts on actual hardware saturation, moving away from arbitrary provisioning and toward data-driven, right-sized GPU allocations.