GPU Fleet Optimization

Machine learning infrastructure is exceptionally expensive, yet GPU utilization rates in many organizations remain strikingly low. A common failure mode is provisioning entire A100 or H100 instances for inference workloads that only require a fraction of the hardware’s capacity. When streaming multiprocessor (SM) utilization hovers around 30% while memory is fully allocated, the result is significant capital inefficiency.

We approach GPU fleet optimization by moving past basic nvidia-smi checks and relying on high-fidelity metrics. Using Data Center GPU Manager (DCGM), we analyze the actual workloads to determine the precise constraints—whether compute, memory bandwidth, or tensor core utilization. This data dictates the right optimization strategy, allowing us to increase the density of workloads per physical node without violating latency SLAs.

For modern architectures like Ampere and Hopper, we implement Multi-Instance GPU (MIG) partitioning. MIG allows a single physical GPU to be divided into isolated instances, each with guaranteed memory and compute resources. This is particularly effective for inference, where a large model might fit perfectly into a 3g.40gb slice, while smaller supporting models can run simultaneously in 1g.10gb partitions. We evaluate the tradeoffs between MIG, Multi-Process Service (MPS), and simple time-slicing based on your specific isolation and throughput requirements.

Typical findings

• H100s at 30% SM utilization because inference pods request a full GPU for a workload that fits in a 3g.40gb MIG slice.
• CUDA memory fragmentation causing Out-Of-Memory (OOM) errors despite low average utilization.
• Over-provisioning on-demand instances for training jobs that could utilize preemptible or spot instances with checkpointing.
• Inefficient batch scheduling leading to long periods of zero SM utilization between training epochs.

By right-sizing instance selection, implementing inference autoscaling, and optimizing batch scheduling, ML teams can significantly expand their compute capacity within their existing budget. The focus remains on maximizing the useful work extracted from every allocated GPU cycle.