Key Takeaways
- Vera Rubin NVL72 delivers up to 30x greater agentic inference throughput per megawatt than GB300 NVL72.
- AgentX benchmark measures real coding sessions with long-context, tool calls, and variable concurrency.
- Efficiency gains stem from a coordinated stack: SGLang/TensorRT-LLM, Dynamo prefill/decode separation, NVLink fabric.
Table of Contents
A Thirtyfold Efficiency Jump Rearranges the Agentic Inference Map
A technical briefing published on NVIDIA’s developer blog this week reports that the Vera Rubin NVL72 platform can deliver as much as 30 times more agentic inference throughput per megawatt than the GB300 NVL72 generation.
The comparison sits inside SemiAnalysis AgentX, an open-source benchmark that replays production-style coding agent sessions with tool calls, long-context prefill, and variable concurrency rather than fixed prompt-response pairs.
The Vera Rubin NVL72 numbers are clearly marked as preview figures: they were measured by NVIDIA using the AgentX workload and remain pending review by SemiAnalysis.
The same workload, detailed in NVIDIA’s developer blog, also shows the GB300 NVL72 stretching its lead over H200 NVL8, with throughput per megawatt reaching roughly 80 times higher on a large mixture-of-experts model such as Kimi K3 2.8T.
Why AgentX Changes the Benchmarking Equation
From static tests to real sessions
Fixed-length inference tests have lost relevance as agents become the dominant serving workload. A coding session does not behave like a single prompt-and-response pair; it chains model calls, pauses for tool execution, and accumulates context that can be reused on later turns.
AgentX addresses that by replaying recorded Claude Code sessions through the AIPerf client. The benchmark keeps each session’s expanding context, turn lengths, reasoning time, and tool-call pauses intact, so every platform faces the same recorded traffic with its full timing pressure.
Because AgentX varies concurrency across operating points, it maps the trade-off between raw throughput and user-visible responsiveness. The result is a per-megawatt throughput figure paired with latency and interactivity data that reveal whether the output is actually usable.
The metrics that separate efficiency from responsiveness
- Tokens per megawatt is the primary measure of how much agentic output an AI factory produces for each provisioned unit of power.
- End-to-end normalized interactivity covers the full request cycle from submission to final token.
- Standard interactivity captures generation speed after the first token arrives.
- End-to-end latency exposes efficiency numbers that would otherwise hide slow request completion.
- Time to first token indicates whether the platform reacts quickly when agents begin long-context turns.
The rack-scale stack behind the throughput
The efficiency gains emerge from a coordinated system stack rather than a single component. Runtimes such as SGLang, TensorRT-LLM, and vLLM distribute expert execution across the NVL72 domain, while DeepGEMM-based kernels and mixed-precision formats like MXFP4 and MXFP8 cut data movement between expert stages.
NVIDIA Dynamo separates prefill and decode into independently scaled worker pools, then uses session IDs and cache-aware routing to reuse context and avoid unnecessary recomputation. The NVLink scale-up fabric ties 72 GPUs into a single high-bandwidth domain for expert-parallel communication and KV-cache movement.
The AI Factory Economics Hiding Inside the Rack
The throughput-per-megawatt numbers translate directly into AI factory unit economics. On the AgentX workload, GB300 NVL72 delivers as much as 10 times lower cost per million tokens than H200 NVL8, meaning a fixed power and infrastructure budget can support substantially more interactive agentic capacity.
That advantage widens with model scale. Against H200 NVL8 on Kimi K3 2.8T, the GB300 NVL72 reaches roughly 80 times the throughput per megawatt at comparable interactivity and pushes the interactivity ceiling to approximately 215 tokens per second per user.
OpenRouter’s State of AI report supplies the demand-side case: across 100 trillion tokens of real-world usage, average prompt tokens per request grew roughly fourfold. A single agentic request can consume about 15 times the token volume of ordinary chat, which is why long-context prefill and KV-cache reuse now dominate infrastructure planning.
The broader Vera Rubin platform extends that logic beyond the GPU. Rubin GPUs handle large-context processing, Vera CPUs target tool execution and KV-cache offload, and Groq 3 LPX focuses on lower-latency interactivity.
Across the AI factory, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X move tokens, context, and tool results between compute, memory, and network resources. The unstated shift is clear: energy-proportional agentic throughput is becoming the governing constraint for AI infrastructure decisions.
From Rack-Scale Throughput to Operational Reality
The clearest takeaway is not a single benchmark result; it is that agentic inference has separated from the static serving assumptions that dominated previous hardware evaluations.
For teams translating AI infrastructure shifts into search-ready technical authority, programmatic SEO AI automation is how Andres SEO Expert approaches it — contact the team.
Frequently Asked Questions
What is the AgentX benchmark and how does it differ from traditional inference tests?
AgentX is an open-source benchmark from SemiAnalysis that replays production-style coding agent sessions, including tool calls, long-context prefill, and variable concurrency. Unlike fixed prompt-response tests, it preserves each session’s expanding context, turn lengths, reasoning time, and tool-call pauses, providing a more realistic measure of agentic inference performance.
How much more efficient is the NVIDIA Vera Rubin NVL72 compared to GB300 NVL72 for agentic inference?
According to NVIDIA’s preview figures measured with the AgentX workload, the Vera Rubin NVL72 platform can deliver up to 30 times more agentic inference throughput per megawatt than the GB300 NVL72 generation. These figures are pending review by SemiAnalysis.
What key metrics does AgentX use to evaluate agentic inference performance?
AgentX uses tokens per megawatt as the primary throughput measure, along with end-to-end normalized interactivity, standard interactivity, end-to-end latency, and time to first token. These metrics reveal both efficiency and user-visible responsiveness under realistic concurrent workloads.
Why do long-context prefill and KV-cache reuse matter for AI factory economics?
Real-world usage shows average prompt tokens per request have grown roughly fourfold, and a single agentic request can consume about 15 times the token volume of ordinary chat. Long-context prefill and KV-cache reuse reduce recomputation, directly lowering cost per million tokens and enabling more interactive capacity per megawatt.
What is the reported cost advantage of GB300 NVL72 over H200 NVL8 on the AgentX workload?
On the AgentX workload, the GB300 NVL72 delivers as much as 10 times lower cost per million tokens than H200 NVL8. For large mixture-of-experts models like Kimi K3 2.8T, the throughput per megawatt advantage reaches roughly 80 times at comparable interactivity.
What system-level technologies contribute to the rack-scale efficiency gains?
The efficiency gains come from a coordinated stack that includes runtimes like SGLang, TensorRT-LLM, and vLLM; DeepGEMM-based kernels; mixed-precision formats like MXFP4 and MXFP8; NVIDIA Dynamo for prefill/decode separation and cache-aware routing; and the NVLink scale-up fabric that enables expert-parallel communication across 72 GPUs.
How does Vera Rubin extend the AI factory platform beyond GPU compute?
The Vera Rubin platform pairs Rubin GPUs for large-context processing with Vera CPUs for tool execution and KV-cache offload, and includes Groq 3 LPX for lower-latency interactivity. NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X handle token, context, and tool-result movement across the entire AI factory.
