Power banks on the left, and active servers are seen at a Digital Realty data center in Ashburn, Virginia on November 12, 2025. ANDREW CABALLERO-REYNOLDS/AFP via Getty Images

NVIDIA published a technical argument on July 14, 2026, that the AI infrastructure cycle has turned on a single question: how many tokens can a fixed megawatt produce? In a power-constrained buildout where transformer lead times stretch five years and switchgear is sold out through 2028, tokens per watt — not GPU count, not teraflops, not server rack density — determines which AI factory generates enough revenue to justify its grid commitment, and which one doesn't. The company backs the argument with benchmark data from SemiAnalysis InferenceX showing its GB300 NVL72 system delivering up to 25 times more tokens per watt than the prior Hopper generation, and with a software finding that matters at least as much: the same hardware delivered a 5x efficiency improvement on a single model in a single month through software changes alone.

The infrastructure decisions that data center operators and hyperscalers are making in the second half of 2026 will lock in their competitive position for years, because the efficiency advantage compounds. A site constrained to 200 megawatts of committed grid power earns materially different revenue depending on how many tokens each watt produces — and as NVIDIA's post demonstrates, that number can move dramatically faster than hardware refresh cycles.

Power Is the Binding Constraint in 2026 AI Infrastructure

Hyperscalers are on track to spend more than $650 billion on AI infrastructure in 2026, but capital is no longer the binding constraint. Transformer delivery times have stretched from the pre-2020 standard of 24 to 30 months to as long as five years. Switchgear is sold out through 2028. Grid interconnection queues for new data center sites in many U.S. and European markets run to five to seven years. According to industry analysis, between 30% and 50% of planned 2026 data center capacity is projected to slip to 2028 or later because of these physical bottlenecks.

The result is that the operators who secured power commitments years ago now hold a structural advantage that money cannot quickly replicate. What they can do is extract more intelligence from the megawatts they already have. NVIDIA's argument, published today, is that this is the primary source of competitive leverage in AI infrastructure right now — and that the efficiency gap between hardware generations is large enough to function as a site-selection criterion.

In a typical AI factory, only about 60% of electricity drawn from the grid converts to useful AI compute. The remaining 40% is dissipated in cooling systems, power distribution losses, and rack-level inefficiencies before a single token is generated. NVIDIA's DSX MaxLPS software platform, announced at GTC Taipei in May, targets this 40% as recoverable compute.

Why the 72-GPU Domain Changes the Efficiency Math for MoE Models

The GB300 NVL72's performance-per-watt gains are not primarily the result of having more powerful individual GPUs. They are the result of having the right architecture for the models that matter most.

Virtually every frontier AI model in production today — DeepSeek V4 Pro, Kimi K2.6, GLM5.1, Qwen3 235B — uses a mixture-of-experts (MoE) architecture. In a sparse MoE model, each input token is routed to only a subset of the model's "expert" sub-networks rather than activating the full model. The efficiency gain is substantial: a 671-billion-parameter model like DeepSeek R1 activates roughly 37 billion parameters per forward pass, keeping compute proportional to what the input actually needs. The problem is communication overhead. When expert sub-networks are distributed across GPUs that can only communicate over slower InfiniBand or PCIe connections, the routing step that makes MoE efficient generates all-to-all network traffic that can eat most of the efficiency gain. The 72-GPU NVLink domain in the GB300 NVL72, interconnected at 130 terabytes per second through the NVLink Switch fabric, eliminates that bottleneck: every expert in a frontier MoE model is reachable within the same high-bandwidth fabric, without crossing a node boundary. The prior Hopper generation's 8-GPU NVLink domain was simply too small to house all active expert weights simultaneously, forcing inter-node communication that the architecture was never designed for. Moving from 8 to 72 GPUs under a single NVLink fabric is therefore not just a scale upgrade — it is the architectural prerequisite that makes MoE inference efficient at all. The efficiency improvement is not incremental; it removes a structural bottleneck.

Across the current generation of leading open MoE models, the GB300 NVL72 delivers up to 25 times more tokens per watt than the Hopper generation on DeepSeek V4 Pro, up to 20 times more on GLM5.1, and up to 10 times more on Kimi K2.6, according to SemiAnalysis InferenceX benchmarks. NVIDIA presents these as Pareto-curve comparisons rather than single-point numbers, acknowledging that the multiple varies significantly with serving configuration, quantization format, and whether latency or throughput is being optimized. The 25x figure represents the upper bound at particular operating conditions; real deployments will land somewhere on the curve depending on their service-level objectives.

The gains are also software-dependent in ways that raise an important caveat and an important opportunity. Software improvements alone delivered a 5x increase in tokens per watt on DeepSeek V4 over a single month on the same hardware. That magnitude of gain from software — without any hardware change — demonstrates how much headroom remains on deployed systems. It also means that benchmark snapshots captured today will not reflect the platform's performance in six months.

SemiAnalysis InferenceX: What the Benchmark Infrastructure Says and What It Doesn't

The performance numbers NVIDIA presents come from SemiAnalysis InferenceX, an independently operated, continuously updated benchmark platform that publishes results across hardware configurations in near-real time. SemiAnalysis is not an NVIDIA property, and AMD's own developer blog cites the same dataset — including to demonstrate that AMD's MI355X delivers lower cost-per-token than GB300 NVL72 at high-concurrency FP8 configurations without multi-token prediction enabled. That finding is material: at 60-plus tokens per second per user interactivity without NVFP4's quantization advantage, the MI355X was showing a lower cost-per-million-tokens figure on the same InferenceX dataset as of March 2026.

The benchmark comparison is therefore configuration-sensitive in ways that matter for procurement decisions. NVIDIA's NVFP4 quantization format — a proprietary 4-bit floating-point standard — reduces model weights to one-quarter the memory footprint of BF16 and one-half the footprint of FP8, allowing larger batch sizes and higher throughput from the same memory bandwidth. AMD's competing FP8 format (one byte per parameter versus NVFP4's half-byte) is competitive on throughput at high concurrency, but the memory efficiency gap means NVFP4 configurations pull ahead as model sizes grow and batch sizes are optimized. The bottom line is that GB300 NVL72's efficiency advantage is most pronounced on large MoE models, at quantization formats that require NVFP4, and at latency targets that favor large-batch throughput — exactly the operating conditions that production frontier-model inference runs in. It is least pronounced on smaller models at high concurrency without multi-token prediction, where the MI355X's memory capacity advantage and software maturity can match or beat it.

GB300 NVL72 delivers AI inference at a claimed $0.123 per million tokens at 116 tokens per second per user interactivity using NVIDIA Dynamo and TensorRT-LLM — positioned as the lowest cost-per-token figure on the SemiAnalysis InferenceX leaderboard as of April 2026. That benchmark reflects a specific operating point and will move as both software stacks improve. AMD's MI450 GPUs, integrated into the Helios rack-scale platform, are targeted for the second half of 2026 and will contest the same efficiency claims with a competing fabric architecture based on the UALink interconnect standard.

Software as Ongoing Infrastructure Investment

NVIDIA emphasizes that hardware purchase decisions in this cycle are inseparable from software optimization trajectories. The inference software stack — combining NVIDIA Dynamo, TensorRT-LLM, and open-source frameworks including SGLang and vLLM — layers multiple techniques that compound: NVFP4 quantization, disaggregated serving, large-scale expert parallelism, KV-aware routing, and KV cache offloading. Each technique operates at a different level of the stack, and they interact in ways that experienced deployment teams learn to tune over months of production traffic.

Disaggregated serving is particularly significant for understanding the efficiency gains. A standard inference pipeline handles both the prefill phase — processing the entire input prompt in parallel, which is compute-bound and benefits from high FP4 throughput — and the decode phase — generating tokens autoregressively one at a time, which is memory-bandwidth-bound and benefits from fast memory access. By separating these phases and routing each to the hardware best suited for it, NVIDIA Dynamo allows the GB300 NVL72's GPU resources to focus on prefill while the decode phase runs on more specialized hardware. This is the architectural logic behind the Groq 3 LPX pairing in the upcoming Vera Rubin configuration.

The production customer list NVIDIA names provides some grounding for the efficiency claims. Anthropic and OpenAI use Blackwell NVL72 systems for inference production workloads. CoreWeave deployed Kimi K2.6 on GB300 NVL72 using NVFP4 quantization combined with EAGLE3 speculative decoding, as detailed in CoreWeave's published benchmark results — a technique that uses a fast draft model to speculatively predict several tokens ahead, with the main model verifying the draft in a single forward pass, achieving 2 to 4 times lower latency at equivalent quality. Perplexity runs Qwen3 235B and a post-trained Qwen3.5-397B-A17B on GB200 NVL72 for its AI agent platform, serving millions of queries daily. Fireworks AI deploys GLM 5.2 on the Blackwell platform for customers including Cursor and Factory AI. Each of these deployments represents a sustained, revenue-generating production workload on the platform — a stronger signal than synthetic benchmarks, though not a substitute for testing the specific model and serving configuration that a prospective buyer would actually run.

DSX MaxLPS: Recovering the 40% of Grid Power That Currently Reaches No GPU

DSX MaxLPS — NVIDIA's power-efficiency software layer, announced at GTC Taipei in May and part of the broader NVIDIA DSX platform — addresses the 40% power loss problem from the facility side rather than the silicon side.

The platform combines 45-degree Celsius liquid cooling with real-time power steering across GPUs and racks. Conventional data center liquid cooling uses chilled water, which requires energy-intensive chillers. Warm-water liquid cooling at 45°C allows the coolant to reject heat directly to ambient-temperature water supplies in many climates, eliminating the chiller entirely or reducing its load substantially. The power recovered from reduced cooling overhead is then redirected toward compute through DSX MaxLPS's power-steering algorithm, which continuously monitors GPU utilization and available thermal headroom and shifts the permitted power budget toward nodes that can make immediate use of it.

The claimed outcome is that operators can run up to 40% more GPUs within the same power budget by operating at the most energy-efficient point on the GPU's power-performance curve, with minimal impact on workload performance. Cloud partners including CoreWeave, Lambda, and Nebius have deployed DSX components; Dell Technologies, HPE, Lenovo, and Supermicro are building DSX-ready systems. The effect at scale is significant: a data center constrained to a 100-megawatt grid connection that recovers 40% of previously wasted power gains the effective compute equivalent of adding 40 megawatts of capacity without requiring a new grid interconnection. In markets where grid interconnections take five to seven years to obtain, that is not a marginal efficiency gain — it is a meaningful competitive advantage.

What Does Vera Rubin Add?

NVIDIA's argument extends forward to the Vera Rubin platform, which entered production in June and is targeting second-half 2026 availability from major cloud providers. In December 2025, NVIDIA entered into a non-exclusive license agreement for the inference technology developed by Groq, the AI inference startup whose language processing units had demonstrated among the fastest token-generation speeds in the industry, and hired key members of Groq's founding team. That licensing deal produced the Groq 3 LPU, announced at GTC 2026 in March.

The Groq 3 LPX rack integrates 256 language processing units that operate on a fundamentally different memory architecture than GPUs. Where a GPU's HBM3e memory delivers approximately 8 terabytes per second of bandwidth, the Groq 3 LPU uses on-chip SRAM that achieves 150 terabytes per second per chip — a nearly 19-fold bandwidth advantage per chip for the memory-bound autoregressive decode phase. The Groq 3 LPX rack is designed to work alongside the Vera Rubin NVL72 rack in a disaggregated serving configuration: Vera Rubin handles the compute-intensive prefill phase while the Groq 3 LPX handles the memory-bandwidth-intensive decode phase, routing each stage of inference to the hardware architecture best suited for it.

According to NVIDIA, the Vera Rubin NVL72 combined with the Groq 3 LPX delivers up to 35 times higher inference throughput per megawatt and approximately 10 times the revenue opportunity for trillion-parameter, high-context workloads compared with the current Blackwell generation. Those figures are NVIDIA's self-reported projections for a system still in production ramp, not yet validated by independent third-party benchmarks at production scale. Buyers should treat them as directional targets rather than confirmed performance numbers — the same caveat that applied to Blackwell's efficiency claims before production deployments provided independent corroboration.

The Vera Rubin NVL72 pairs 72 Rubin GPUs and 36 Vera CPUs over NVLink 6, which doubles the all-to-all fabric bandwidth from 130 terabytes per second in GB300 to 260 terabytes per second, and moves from HBM3e to HBM4 memory, tripling per-GPU bandwidth from 8 to 22 terabytes per second.

How This Changes Site-Selection Economics

The efficiency argument carries direct implications for AI data center planning that extend well beyond hardware procurement.

Next-generation AI campuses are being planned at 100 megawatts to 750 megawatts per site. At that scale, a data center competes directly with municipal power infrastructure for grid capacity, and approval cycles resemble those for industrial facilities. A 40% software-driven efficiency improvement from DSX MaxLPS represents tens of megawatts of effective capacity at a 100-megawatt site — without requiring any additional grid interconnection.

For hyperscalers evaluating which platform to standardize on, the compounding implications matter. A site running GB300 NVL72 hardware with DSX MaxLPS software generates meaningfully different revenue than the same physical site running Hopper-generation hardware at the same grid commitment. If NVIDIA's efficiency roadmap continues delivering generational improvements of this scale, the infrastructure decisions being made today will determine competitive positioning not just for this hardware cycle but for the site capacity that cannot be easily augmented once grid commitments are locked.

NVIDIA's framing of tokens per watt as "a metric that can't be gamed, only earned through real-world results" is self-serving but structurally sound. Unlike peak teraflop ratings measured under synthetic conditions, tokens per watt on production MoE models at real service-level objectives is harder to inflate through selective benchmark configurations — though the sensitivity to quantization format, serving configuration, and model selection means that a single headline multiplier will always obscure meaningful variation across real deployments.

What AMD's Competing Answer Means for Procurement Teams

AMD's rack-scale response to this efficiency argument is the Helios platform with MI450 GPUs, targeted for the second half of 2026. The Helios rack uses a UALink-based interconnect rather than NVLink, which AMD positions as an open-standard alternative designed to avoid vendor lock-in. The MI450 features 432 gigabytes of HBM4 memory per GPU — significantly more than the GB300's 288 gigabytes — and 19.6 terabytes per second of memory bandwidth, compared to 8 terabytes per second for the GB300. The memory capacity advantage is most significant for serving very large models in high-concurrency configurations where model shards and KV cache together fill available HBM.

AMD's own analysis of SemiAnalysis InferenceX data shows that the MI355X — the current deployed Instinct GPU — already outperforms GB300 NVL72 on cost-per-million-tokens at operating points above 60 tokens per second per user in FP8 configurations without multi-token prediction. That finding matters for teams running high-concurrency inference where the decode-phase throughput ceiling is the binding constraint rather than the prefill-phase compute ceiling. AMD's software stack continues to improve under ROCm, and the meaningful performance-per-software-dollar gap that has historically favored NVIDIA is narrowing.

The InferenceX dataset that currently supports NVIDIA's efficiency argument publishes competitor results alongside NVIDIA's. If AMD's Helios rack delivers on its H2 2026 timeline with competitive tokens-per-watt numbers, the same benchmark infrastructure that currently tells the story NVIDIA published today will document the competitive narrowing in real time.

Frequently Asked Questions

What does tokens per watt actually mean, and why does it matter more than GPU count for AI data center decisions?

Tokens per watt measures how much useful AI output — specifically, the text tokens that AI services generate and users pay for — a system produces per unit of power consumed. In a power-constrained buildout where new grid capacity takes years to obtain, the number of tokens a megawatt produces directly determines a data center's revenue potential. GPU count matters only insofar as it translates into tokens per watt at the facility's fixed power budget. A data center that can generate 25 times more tokens from the same megawatt commitment earns 25 times more revenue from the same grid contract — which is why NVIDIA's framing has gained traction with infrastructure buyers who can no longer simply buy more power to grow capacity.

Why do MoE models specifically require a 72-GPU NVLink domain to achieve these efficiency gains?

Mixture-of-experts models route each input token to a subset of specialized sub-networks called experts. These experts are distributed across GPUs, and the routing step requires every GPU to potentially communicate with every other GPU — an all-to-all communication pattern. When the GPUs communicating are on different nodes connected by InfiniBand or PCIe, this inter-node traffic adds substantial latency and bandwidth overhead that erases much of MoE's efficiency advantage. NVIDIA's 72-GPU NVLink domain at 130 terabytes per second keeps all active experts within a single high-bandwidth fabric, eliminating cross-node communication entirely. The Hopper generation's 8-GPU NVLink domain was too small to house all expert weights for frontier-scale MoE models, making the 72-GPU domain a structural prerequisite for efficient MoE inference, not merely an incremental performance improvement.

How much of the 25x efficiency gain comes from hardware versus software, and does software-only improvement change hardware procurement logic?

The 25x figure reflects the combined hardware-plus-software improvement from Hopper to GB300 NVL72. The software contribution within that comparison is substantial — NVIDIA separately reports that software improvements alone delivered a 5x increase in tokens per watt on DeepSeek V4 on the same GB300 hardware within a single month. This means the 25x represents a snapshot that is actively moving upward. For procurement teams, the implication is that buying into the GB300 NVL72 platform is purchasing access to an ongoing software optimization trajectory, not a fixed performance level. The flip side is that benchmark comparisons captured today will understate actual production performance in six months, and that software maturity gaps between vendors will continue to close at different rates.

Does AMD's MI355X offer a competitive alternative to the GB300 NVL72 today?

At certain operating points, yes. AMD's own analysis of SemiAnalysis InferenceX data shows that the MI355X delivers lower cost-per-token than GB300 NVL72 at high-concurrency FP8 configurations above 60 tokens per second per user without multi-token prediction enabled. AMD's MI355X also carries more HBM memory than the comparable NVIDIA B300 GPU. The GB300 NVL72's efficiency advantage is most pronounced on large MoE models using NVFP4 quantization at low-to-moderate interactivity targets — the operating conditions that most frontier-model production deployments use. Teams whose workloads run at high concurrency on smaller models in FP8 should benchmark the MI355X against the GB300 directly on their specific model, context length, and latency target before assuming the NVIDIA platform offers better economics. AMD's Helios rack with MI450 GPUs, targeted for later in 2026, will be the first direct rack-scale competitor with equivalent fabric bandwidth at scale.