Scheduling Order Unlocks 33% More GPU Utilization Without New Hardware

Same Cluster, Thirty-Three Points: No New Hardware

On identical hardware and with identical workloads, GPU utilization in a controlled benchmark rose by as much as 33 percentage points. The change was not a new accelerator, a faster interconnect, or a different model; it was the sequence of allocation decisions.

The result comes from a constraint-aware GPU allocator built by the Dharma-AI engineering team and benchmarked against a FIFO scheduler across seven scenarios. Utilization improved in six of seven cases and tied in the seventh, while priority-weighted value rose between 24.6% and 105.1%.

What FIFO Scheduling Costs Under Contention

A FIFO scheduler is not unreasonable when the cluster has slack. If capacity is plentiful, allocation order costs nothing because every job fits regardless of sequence.

Under real contention, order becomes a capacity decision. The baseline scheduler reserves GPUs for real-time inference at the day’s peak demand and places every other job in arrival order without regard for priority.

Reserving for the peak leaves GPUs idle for hours that are not the peak. An application that needs six GPUs at midday and two at 4am holds all six for twenty-four hours.

The idle four GPUs are unavailable to batch work for the entire day. In two reservation-dominated scenarios, baseline utilization sat at 51.6% and 53.6%, meaning roughly half the pool was reserved or unused.

Ordering compounds the damage. FIFO places each job as it arrives without checking what still must fit inside the horizon, so high-priority work waits behind whatever asked first.

Across five contended benchmark scenarios, the Dharma-AI allocator moved utilization from a 52–85% band to a 72–88% band. The strongest case was a training-heavy workload on 8 GPUs: utilization climbed from 53.6% to 87.0%, and priority-weighted value more than doubled.

Five Constraints That Define a Legal Allocation

The allocator does not rely on a longer list of heuristic rules. It formalizes scheduling as one combinatorial decision: which GPU runs which job, in which timestep, and at what priority.

Four workload types compete for the grid: training, real-time inference, batch inference, and quantization. Training, batch inference, and quantization are batch-like and need contiguous GPUs held until completion; real-time inference is elastic and follows a demand curve that changes every timestep.

Five constraints make an allocation legal:

• A GPU serves at most one job per timestep.
• Each job respects its demand range, and running work is inherited and held.
• Batch-like jobs use contiguous GPU blocks sized to a power of two.
• Real-time jobs have a hard cap on GPU churn between consecutive timesteps.
• A started job cannot be interrupted.

The objective function has two terms. Allocating a GPU to batch-like work earns a reward equal to priority multiplied by a time-decay weight; missing real-time demand incurs a penalty proportional to the shortfall.

The real-time penalty weight is set 5 to 10 times higher than the allocation weight. That asymmetry makes the elastic treatment of real-time demand safe, because the optimizer is penalized heavily for underserving traffic later.

The heuristic is built directly from the formal model’s structural constraints, so every grid it produces is legal by design. It runs in 1 to 2 milliseconds on the contended scenarios and 15 milliseconds at 64 GPUs with 30 jobs.

Forecasts are the prerequisite. A single generic estimator fails because the four workload types have different cost drivers.

Training estimates condition on 22 features, including a categorical variable for 10 training variants. Quantization gets its own forecast built from calibration tiers and algorithm-specific handling for bitsandbytes, AWQ, and GPTQ.

Real-time inference is not estimated per job. It is forecast as a continuously recalibrated weekly demand profile mapped to GPU counts under the same swap cost the formal model enforces.

The scheduler optimizes a 24-hour horizon but commits only the current timestep. It re-runs every 30 to 60 minutes, inherits running work, and pins it in place, so later plans update instead of thrash.

The scale test is the most instructive. On 64 GPUs and 30 jobs, FIFO and the allocator delivered identical utilization at 44.9% and identical throughput at 27 of 30 jobs completed.

Yet the allocator produced 15.9% more priority-weighted value. Utilization alone is necessary but not sufficient; priority is what turns occupancy into output.

The uniform-priority test addresses the skeptical reading. When every job was assigned identical priority, the allocator still lifted utilization from 76.8% to 87.5% and value by 23.1%.

Where GPU Orchestration Is Heading Next

The same cluster, same workload problem is attracting research attention beyond enterprise infrastructure teams. A preprint on arXiv from Beihang University and collaborators describes ElastiCo, a Kubernetes-native GPU management framework that co-locates deep learning training and offline inference without user-code changes.

ElastiCo reports that its average job completion time normalizes to 0.34x against Volcano on a 64-GPU A100 testbed, with GPU SM utilization rising from roughly 25% to 46%. These are different metrics from allocation utilization, so the figures should not be directly overlaid on the Dharma-AI benchmark.

ElastiCo’s architecture uses resource shape transformation, elastic shadow pricing, interference-aware co-location, and phase-aware disaggregated scheduling. The interference predictor reports a 7.6% mean absolute percentage error and an R2 of 0.89.

A separate study in Future Generation Computer Systems presents STAO, a spatio-temporal adaptive orchestration system for DNN inference. STAO combines offline profiling with online switching between elastic SM-level spatial partitioning and temporal isolation for high-risk workloads.

STAO reports sub-millisecond runtime overhead and up to 1.15x system throughput improvement against a sequential baseline on an NVIDIA RTX 4090. Its current evaluation is limited mainly to dual-task co-location pairs.

The comparison clarifies what has changed. ElastiCo delivers middleware-level scheduling without code changes; STAO targets fine-grained inference isolation; the Dharma-AI allocator formalizes the allocation problem and runs a lightweight heuristic on the hot path.

There is a tension hidden in the numbers. The Dharma-AI benchmark reports allocation utilization as high as 87% on 8-GPU scenarios, while ElastiCo’s GPU SM utilization peaks near 46% under co-located training and inference.

The difference is not a contradiction. It reflects different measurement layers: allocation utilization measures occupancy, while SM utilization measures how busy the processors actually are.

Each system attacks a different layer of the same issue. GPUs are already installed, already committed, and already depreciating, so the recoverable margin is not in buying more capacity but in deciding which job gets which GPU in which timestep.

The Order Beats the Model

The GPUs were already installed and depreciating; the recovery came from order, not hardware. For teams building AI infrastructure analysis that needs to rank, programmatic SEO AI automation is how Andres SEO Expert turns technical depth into search-scale visibility — contact.