Nvidia founder and CEO Jensen Huang speaks during a press conference at the Asia-Pacific Economic Cooperation (APEC) CEO Summit in Gyeongju on October 31, 2025. JUNG YEON-JE/AFP via Getty Images

At the AI Infra Summit in Santa Clara on September 15, NVIDIA and independent semiconductor research firm SemiAnalysis presented AgentX benchmark results showing NVIDIA's Vera Rubin NVL72 system delivering up to 30 times higher throughput per megawatt than NVIDIA's own GB300 NVL72 on agentic-coding inference workloads. The figure, measured on early silicon with software optimization still ongoing, was released at the same moment that power availability — not raw compute — has become the binding constraint on AI factory revenue across the industry. For operators of power-constrained data centers, a 30x improvement in throughput per megawatt means 30 times as many concurrent AI agent sessions from the same grid allocation. That arithmetic is why the number matters — and why the methodology behind it matters just as much.

AgentX: Why the Benchmark Methodology Is the Story

Conventional AI inference benchmarks, including the widely cited MLPerf Inference suite, evaluate hardware by feeding models a fixed-length prompt and measuring output tokens per second. That approach made sense when chatbots answering isolated questions were the dominant production workload. It no longer does.

An agentic AI session looks nothing like a single chat turn. An agent researching a company for an investment decision queries financial databases, searches filings, invokes sub-agents to run peer comparisons, synthesizes multiple sources, and loops through tool calls — all before producing a final answer. At each step, the accumulated context from prior turns becomes the input to the next, compounding the token volume rapidly. Across 100 trillion tokens of real-world traffic analyzed by OpenRouter, single agentic requests consume roughly 15 times the tokens of an ordinary chat interaction. Fixed-prompt benchmarks miss this entirely.

AgentX addresses that gap by replaying recorded real sessions rather than synthesizing uniform prompts. The corpus at the heart of the benchmark consists of 393 anonymized real-world coding-agent traces — actual sessions captured from SemiAnalysis's internal Claude Code usage — preserving the real prompt sizes, tool-call timing, sub-agent fan-out, KV-cache prefix-sharing structure, and inter-turn delays of production agentic traffic. The corpus cost more than $3 million to generate and is published openly on HuggingFace under an Apache 2.0 license, making it independently verifiable. Every benchmark data point on the SemiAnalysis InferenceX benchmark dashboard links back to the specific GitHub Actions workflow run that produced it, with full logs and artifacts publicly visible.

The median session in the corpus sends 142,000 input tokens and 444 output tokens per request, and 44% of sessions spawn sub-agents. That input-to-output ratio — roughly 320 input tokens for every output token — creates a radically different computational profile than the 8K input / 1K output scenarios that have historically anchored MLPerf's fixed-sequence benchmarks.

The key metric AgentX reports is tokens per megawatt at a given interactivity level, measured in output tokens per second per user. Higher interactivity means more responsive sessions; the benchmark maps the trade-off between throughput and interactivity across a range of concurrency levels. This captures something prior benchmarks could not: the performance cliff that appears when a hardware system is asked to serve many concurrent long-context sessions simultaneously.

What Does 30x Throughput Per Megawatt Actually Mean?

On the DeepSeek V4 Pro model — a 1.6-trillion-parameter Mixture-of-Experts architecture — NVIDIA's results show Vera Rubin NVL72 delivering 30 times higher throughput per megawatt than GB300 NVL72 under AgentX conditions. The cost advantage is steeper still: token costs are 45 times lower per million.

To understand the magnitude, it helps to situate GB300 NVL72 as a baseline. GB300 NVL72 is not an aging predecessor — it is NVIDIA's current Blackwell Ultra flagship. On the same DeepSeek V4 Pro AgentX workload, GB300 NVL72 delivers 15 times more throughput per megawatt than the NVIDIA Hopper H200 NVL8. Vera Rubin doubles that advantage relative to GB300. For the computationally demanding Kimi K3 2.8-trillion-parameter model, GB300 NVL72 reaches 80 times higher efficiency per megawatt than H200 NVL8.

The advantage is not uniform across the interactivity curve. At lower interactivity targets — around 110 output tokens per second per user — Vera Rubin holds roughly a 2x advantage over GB300. The gap widens sharply at higher concurrency targets: approximately 10x at 130 tokens per second per user, and 30x at 160 tokens per second per user. This scaling behavior is consistent with Vera Rubin's architecture being optimized specifically for the high-concurrency, long-context operating regime that agentic production workloads occupy.

How Throughput Per Megawatt Became the Number That Matters

For most of AI's recent commercial history, the dominant procurement metric was peak theoretical FLOPs per GPU, followed by tokens per second per chip. Those metrics were appropriate when compute availability was the binding constraint. Increasingly, power is.

New power infrastructure takes years to permit and build. Cooling capacity limits are forcing data center operators to manage thermals more aggressively. In this environment, the question governing AI factory revenue is not how many FLOPs a GPU delivers in a controlled test, but how many tokens it produces per megawatt of available grid power — continuously, at scale, under realistic workloads. NVIDIA VP of Hyperscale and HPC Ian Buck framed the shift at the AI Infra Summit: "The metric for AI infrastructure is fast shifting from peak performance to validated agentic tokens per megawatt."

An independent validation of this dynamic emerged at the same summit. AI cloud provider Lambda released results showing NVIDIA DSX MaxLPS — a software-level dynamic power allocation system — ran 19 nodes within the power budget typically allocated to 16 full-power nodes, increasing cluster-wide token throughput by 24%, from roughly 4 million to 5 million tokens per second, while improving performance per watt by 23%. No new hardware was required. For Vera Rubin NVL72 deployments, NVIDIA says DSX MaxLPS can enable 40% more GPU capacity within the same megawatt budget.

How Does This Architecture Actually Work?

The efficiency claims are the product of a system designed end-to-end for agentic workloads, not retrofitted from a general-purpose compute platform. Several specific mechanisms drive the result.

Disaggregated serving separates context processing (prefill) from response generation (decode), allowing each to scale independently. Prefill is compute-bound; decode is memory-bandwidth-bound. NVIDIA Dynamo implements this separation, routing prefill and decode to different GPU worker pools based on current load.

KV-aware routing directs incoming requests to GPUs already holding the relevant cached context from prior turns of the same session, avoiding redundant recomputation. In a session that has accumulated 400,000 tokens of context, skipping that recomputation is not a minor optimization — it is a first-order throughput driver.

Large-scale expert parallelism becomes viable at 72-GPU scale. DeepSeek V4 Pro is a Mixture-of-Experts model: 1.6 trillion parameters total, but only a subset of "expert" sub-networks activates for each token. Distributing those experts across 72 GPUs connected by NVLink 6 allows many tokens to be routed to their experts in parallel, rather than queuing for a small GPU pool.

NVFP4 quantization compresses model weights to 4-bit floating-point precision, reducing memory footprint and increasing the throughput achievable within a given power envelope. The Rubin GPU's fifth-generation Tensor Cores with third-generation Transformer Engine execute NVFP4 natively.

NVLink 6 is the interconnect that makes the above techniques viable at rack scale. The sixth generation delivers 10 times higher packet rates and 3 times lower latency than off-the-shelf Ethernet alternatives, with 3.6 terabits per second of bidirectional bandwidth per GPU and 260 terabits per second across the full 72-GPU rack. Without this bandwidth, distributing KV-caches and routing MoE experts across 72 GPUs would introduce latency that eliminates the throughput advantage.

MegaMoE fused kernels combine computation and inter-GPU communication into single CUDA execution passes, keeping GPUs active rather than idle while waiting for data movement to complete.

What the Summit Revealed Beyond the Headline Number

The AI Infra Summit — which drew more than 8,000 attendees this year, more than double last year's 3,500 — produced several additional ecosystem signals that extend the Vera Rubin picture.

On grid management: Emerald AI demonstrated automated load reduction at Silicon Valley Power using NVIDIA DSX Flex, successfully responding to hundreds of utility demand signals while protecting active AI workloads. The system is designed to allow AI factories to function as flexible grid resources — throttling power on low-priority jobs during peak demand, then resuming automatically.

On ecosystem integration: Amazon's Annapurna Labs is partnering with NVIDIA on NVHBM, a custom high-bandwidth memory technology. D-Matrix is integrating with NVLink Fusion to combine Vera CPUs with its Raptor XPUs for low-latency inference.

On the Vera CPU specifically: multiple startups reported benchmark results at the summit. Perplexity found 1.9 times faster sandbox starts for its SPACE secure agentic AI platform. Redpanda reported 5.5 times lower latencies and 73% higher throughput than on other CPUs. Starburst recorded 3 times faster query throughput and Kinetica 2.7 times faster analytical query performance. These results do not affect the AgentX throughput-per-megawatt figures, which do not yet include Vera CPU performance for tool calling — a component NVIDIA flags as a significant pending measurement.

What to Watch: Three Questions the Results Leave Open

How broadly does the benchmark generalize? AgentX is currently built from 393 traces drawn entirely from a single agentic workload type — software coding — captured from a single tool (Claude Code). The benchmark is open source and designed to accept additional trace corpora, but whether the 30x throughput-per-megawatt advantage holds for agentic workloads outside coding — retrieval-augmented generation, multi-agent orchestration, scientific research workflows — remains unmeasured. MLCommons added an Agentic Inference benchmark to MLPerf in July 2026, using 490 coding trajectories plus 500 enterprise workflow trajectories from Workato, with AMD, Intel, and NVIDIA participating as co-developers. That broader benchmark will provide the cross-workload comparison AgentX currently cannot.

How much will software optimization shift the numbers? NVIDIA explicitly notes ongoing software improvements for both Vera Rubin and GB300. SemiAnalysis's own pre-summit analysis, published September 14, found that even on pre-release software, real-world testing showed results roughly 7 times higher than Jensen Huang's own GTC 2026 claim of a 3x advantage — a finding consistent with SemiAnalysis's prior history of measuring GB200 at 98x over Hopper vs. Jensen's stated 30x. Software maturation cuts in two directions: Vera Rubin's lead could extend, but GB300 results on the same benchmark may also improve.

Can AgentX achieve the governance independence needed for industry adoption? MLPerf Inference has broad industry recognition precisely because it is developed and governed by a neutral consortium — MLCommons — with participation from NVIDIA, AMD, Intel, Google, and others. AgentX's technical methodology is compelling, and its open-source publication is a genuine step toward auditability. But NVIDIA hosts AgentX's primary implementation documentation, and the first verified results were released in conjunction with a major NVIDIA product announcement. For a benchmark to function as an industry standard, the question of who controls its governance matters as much as whether the methodology is sound. SemiAnalysis operates the open InferenceX results dashboard and designed the traces; NVIDIA documents the implementation. Whether that relationship creates a structural conflict — or whether AgentX's open-source corpus and independent dashboard are sufficient transparency — is a question the industry will work out as AMD, Google, and others begin submitting competing results. The InferenceX dashboard already includes results across Google TPU v7 Ironwood, AMD MI355X, and multiple Blackwell generations, which is a meaningful signal of cross-vendor adoption.

Frequently Asked Questions

What is the AgentX benchmark and how is it different from MLPerf?

MLPerf Inference traditionally measures hardware using fixed-length synthetic prompts — a controlled, reproducible test that tells you how fast a system processes a given number of tokens in isolation. AgentX measures hardware by replaying recorded real-world coding-agent sessions, preserving the actual structure of agentic traffic: long accumulated contexts, sub-agent spawning, tool calls, and highly variable input/output lengths. A single agentic session in the AgentX corpus has a median input of 142,000 tokens — compared to the 8,000-token inputs common in fixed-sequence benchmarks. MLCommons added its own agentic inference track to MLPerf in July 2026 with broader workload coverage, but AgentX remains the only replay-based benchmark currently publishing cross-hardware results under an open-source license.

What does 30x throughput per megawatt mean for someone buying or renting AI infrastructure?

If you are building or operating an AI factory where power availability is the binding constraint — and for most large deployments today, it is — throughput per megawatt directly determines how much revenue you can generate from a fixed grid allocation. A 30x improvement means you can serve 30 times as many concurrent agentic sessions, or serve the same sessions at one-thirtieth the power cost, within the same megawatt budget. The 45x lower cost per million tokens compounds the economic effect: at current GPU rental rates and power costs, the difference between a system at GB300 efficiency and a system at Vera Rubin efficiency, running at scale across hundreds of megawatts, reaches into the hundreds of millions of dollars annually.

Is NVIDIA's role in documenting AgentX a problem for the benchmark's credibility?

It is a legitimate question that the industry has not yet resolved. SemiAnalysis designed the benchmark, owns the trace corpus, and operates the results dashboard independently. The open-source license and public GitHub logs mean any organization can run and verify the tests. However, NVIDIA hosts the primary implementation documentation for AgentX on its own developer infrastructure, and the first Vera Rubin results were published through NVIDIA's own blog network alongside a product announcement. By contrast, MLPerf results are published through MLCommons, a neutral consortium. For a benchmark to be trusted as an industry standard rather than a vendor tool, governance independence from any single vendor matters. Whether AgentX's open-source foundation is sufficient, or whether formal MLCommons-style governance is needed, will become clearer as AMD, Google, and other vendors publish their own AgentX results and independent operators begin replicating the NVIDIA numbers.

What are the specific caveats on the 30x figure that a buyer should understand before acting on it?

Three caveats are explicit in NVIDIA's own materials. First, these are early silicon results — software optimization is ongoing for both Vera Rubin and GB300, and both figures will change. Second, the benchmark corpus covers only software-coding agentic workloads, and the advantage may differ for other agentic workload types such as retrieval-augmented generation or enterprise workflow automation. Third, the Vera CPU's contribution to agentic performance — specifically for tool calling — has not yet been measured and is not included in the current results, which means the complete Vera Rubin platform's efficiency advantage is not yet fully characterized.