Mlcommons.org

The most useful finding in MLCommons' MLPerf Inference v6.1 results — published September 16, 2026 — may not be NVIDIA's next-generation hardware numbers. It is the evidence that software optimization alone, applied to hardware enterprises have already purchased, delivered 8 to 36 percent throughput gains in five months. For a market where multi-million-dollar infrastructure decisions routinely hinge on benchmark scores, that gap between optimized and unoptimized deployments is now measurable from a neutral, consortium-governed source for the first time.

That distinction matters because v6.1 also delivers the first peer-reviewed, independently verified performance data for NVIDIA's Vera Rubin NVL72 platform — numbers that belong to a different category than the AgentX results released the previous day at NVIDIA's AI Infra Summit. Those results were measured on early silicon by a benchmark hosted on NVIDIA's own infrastructure. MLPerf results are submitted to and verified by MLCommons, a neutral consortium with 130 members spanning NVIDIA, AMD, Intel, Google, and others, governed by bylaws that prevent any single vendor from controlling the outcome. For procurement teams deciding between platforms, the governance structure of the data source is as material as the numbers themselves.

The round also marks a structural milestone for the benchmark suite itself. The two new benchmark categories — End-to-End RAG and Edge Agentic Inference — represent an explicit admission by the industry's leading neutral benchmark body that single-model, fixed-prompt tests have been measuring the wrong workloads for over a year. Both new categories reflect what production AI systems increasingly look like: multi-component pipelines running on constrained hardware, not isolated models processing synthetic prompts.

Record Participation Signals Maturing Market

The v6.1 cycle drew submissions from 30 organizations, generating 486 individual datacenter and edge results — the largest participation in MLPerf history. The prior round (v6.0, April 2026) had drawn 24 organizations. Six organizations submitted for the first time: Atlas Inference, Crusoe, Orrick Industries, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis.

"The groundswell of participation from the AI community tells us that our work is important and is making a difference to stakeholders," said David Kanter, Head of MLPerf at MLCommons.

The round also logged the largest system ever submitted to the MLPerf Inference benchmark — a cluster of 512 accelerators, submitted by Crusoe using AMD Instinct MI355X GPUs — and two structurally novel heterogeneous configurations: one combining high-performance networking with accelerators from two different vendors, and another geographically distributed across the Pacific Ocean.

What 'Software Gains Beat New Hardware' Actually Means for Your Infrastructure

Enterprises that deployed Blackwell-generation GPUs six months ago and have not yet invested in software stack optimization are, by the evidence of v6.1 results, leaving a measurable fraction of that hardware's capacity unused. Three independent submitters documented this:

CoreWeave reported a 19.8 percent per-GPU throughput gain on DeepSeek-R1 on identical GB200 NVL72 hardware — derived by comparing its v6.1 72-GPU submission with its v6.0 64-GPU submission, normalized per accelerator. Lambda reported 8.85 percent server-scenario improvements on identical Blackwell Ultra hardware, with no silicon change since the prior round. Intel reported that its Arc Pro B70 four-GPU node improved GPT-OSS-120B performance by 36 percent in server scenario and 27 percent in offline scenario on identical hardware versus v6.0, driven entirely by software optimizations.

Lambda's analysis of its Blackwell Ultra improvements attributed the gains specifically to a set of TensorRT optimizations: torch.compile fusing operations into Triton kernels, piecewise CUDA graph capture covering 39 variants for prefill and decode phases, 14 additional decode-only steady-state graphs, a Mixture-of-Experts AutoTuner selecting optimal computation strategies per expert network, and a KV-cache sizing method that dry-runs a forward pass to maximize use of available high-bandwidth memory. Each of these is a software change. None requires new hardware.

The practical implication is a two-part procurement framework: an organization evaluating whether to buy Vera Rubin hardware now should first determine whether its current Blackwell or Blackwell Ultra hardware is running a v6.0-equivalent software stack. If yes, the accessible gains may partially satisfy the performance need without new capital expenditure.

Vera Rubin NVL72: What Peer-Reviewed Data Shows

NVIDIA submitted its Vera Rubin NVL72 in MLPerf's "Preview" category — a designation reserved for platforms expected to reach commercial availability by the following submission round. Nebius also submitted Vera Rubin results, making the round the first set of independently verified performance data for the architecture.

On Qwen3-VL, a 235-billion-parameter visual language model from Alibaba, Vera Rubin NVL72 delivered up to 3.7 times higher throughput than the current-generation GB300 NVL72 across offline, server, and interactive scenarios, using vLLM with NVIDIA's open-source Dynamo inference framework. On DeepSeek-R1, using the TensorRT-LLM library, Vera Rubin NVL72 reached up to 2.5 times the throughput of GB300 NVL72.

For context: GB300 NVL72 demonstrated 99 percent scaling efficiency when expanded from a single 72-GPU rack to a 288-GPU, four-rack configuration on the DeepSeek-R1 offline test — meaning throughput grew almost linearly as hardware was added, rather than hitting the inter-node communication bottlenecks that have historically limited multi-rack scaling.

The 30x throughput-per-megawatt figure from NVIDIA's own AgentX benchmark, reported the prior day at the AI Infra Summit, is a separate measurement on a separate benchmark and has not been peer-reviewed by MLCommons. The MLPerf v6.1 Preview numbers represent what MLCommons consortium members have verified.

How the Leaderboard Looks Across the Field

Year-over-year and round-over-round gains exceeded what most observers expected from software iteration alone. On DeepSeek-R1, the best per-accelerator server result in v6.1 was 5.7 times better than the equivalent result from v5.1 one year earlier. On Qwen3-VL, the best per-accelerator server result improved 2.99 times over v6.0 from just six months prior.

CoreWeave achieved the highest per-GPU throughput of any v6.1 Datacenter Closed submission on GPT-OSS-120B, on any silicon, in both server and offline scenarios — reaching 16,635 tokens per second per GPU in offline scenario on a single GB300 NVL72 rack, and sustaining over 1.16 million tokens per second aggregate in server scenario and over 1.19 million tokens per second in offline scenario. CoreWeave also delivered the highest server throughput among cloud providers on Qwen3-VL, sustaining 1,196 queries per second on a GB300 NVL72 rack.

AMD expanded its submission to six model families — language, reasoning, text-to-video, and recommendation tasks — deploying Instinct MI355X, MI350X, and the newly introduced MI350P PCIe card. The AMD-based Crusoe cluster — 512 Instinct MI355X GPUs, the round's largest system — led aggregate throughput results on DeepSeek-R1 and GPT-OSS-120B. Intel broadened its Xeon 6 server portfolio from two SKUs in v6.0 to five in v6.1, with CPU-only inference results growing from 24 to 35 configurations — making Intel Xeon the only standalone server CPU submitted to MLPerf Inference.

What Is the End-to-End RAG Benchmark, and Why Does It Exist?

Standard MLPerf benchmarks feed a fixed-length prompt to a single model and measure how fast it responds. That design was accurate when the dominant production deployment was a standalone language model answering questions from its own training data. It no longer reflects how most enterprise AI query systems are built.

Retrieval-Augmented Generation separates knowledge from reasoning. Rather than training facts into model parameters — where they become static and can go stale — a RAG system stores knowledge in an external vector database and retrieves relevant passages at inference time. When a user asks a question, an embedding model converts the query into a vector; a retriever searches the vector database for semantically similar passages; a re-ranker orders the results; and an LLM generates a final answer from the retrieved text. The whole process is repeated, if necessary, up to five times, with a sufficiency checker deciding whether the answer is complete.

The MLPerf E2E-RAG benchmark tests this full pipeline under load, measuring performance across two separate tasks: ingesting a document corpus to build the vector database, and answering live queries against a pre-built database. The reference implementation uses four models concurrently — GPT-OSS-120B handles query decomposition, sufficiency checking, and final answer generation; GPT-OSS-20B grades retrieved documents; and smaller embedding and re-ranking models handle retrieval. Intel's Arc Pro B70 four-GPU node, with 128 gigabytes of combined VRAM, was among the first systems to submit E2E-RAG results in this inaugural round.

"We added the End-to-end RAG test because it's clear that query-answering has evolved beyond simply an LLM trained on a corpus," said Miro Hodak, MLPerf Inference working group co-chair. "Stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today."

Edge Agentic Inference: When the Model Has to Think, Tool-Call, and Loop on a Single Device

The second new category targets a different deployment reality: agentic workloads running on a single user device, where memory, power, and context window are all constrained and each request can take many tool-calling steps to complete. Unlike cloud inference — where requests arrive in parallel batches and throughput is the governing metric — edge agentic inference must serve one session at a time, keeping latency low even as the conversation history grows turn by turn.

The MLPerf Edge Agentic Inference benchmark adapts the forthcoming MLPerf Agentic datacenter benchmark for edge hardware, using a quantized model, a single-stream coding workload, and latency-based metrics rather than throughput targets. It measures two tasks: accuracy under time constraints, and performance on a workload where accuracy verification is intrinsic to each step. NVIDIA submitted Jetson AGX Thor results in the new category.

Lambda submitted an open-division result that demonstrated the benchmark's range: instead of the reference 27-billion-parameter model, Lambda deployed Kimi K2.6, a mixture-of-experts architecture with over a trillion parameters, on its NVIDIA HGX B200 datacenter system — the first trillion-parameter MLPerf deployment and the first agentic workload run on datacenter hardware in MLPerf history. The submission completed 1,007 turns with zero failures, with mean per-turn latency of 770.8 milliseconds, a median of 361.0 milliseconds, and a time-to-first-token average of 179.7 milliseconds — substantially lower latency than the closed-division edge submissions running a model 40 times smaller.

Five New Chips Make Their MLPerf Debut

Version 6.1 marks the first MLPerf appearance of five silicon platforms:

AMD Ryzen AI Max+ 395 is a unified-memory accelerated processing unit targeting local AI workloads on desktop and mobile systems. AMD Instinct MI350P is a PCIe form-factor version of AMD's CDNA 4 accelerator designed for air-cooled servers that cannot accommodate the liquid cooling required by the NVL72-class rack systems. Intel Arc Pro B70 is a 32 GB workstation GPU suited to large-model inference in multi-GPU nodes; its four-GPU configuration submitted the first E2E-RAG results of any system in this round. NVIDIA Rubin — the entry-level GPU of NVIDIA's Rubin generation — and NVIDIA Vera Rubin NVL72 both appeared in the Preview category.

What MLPerf Endpoints Means for Future Procurement Decisions

More than half of all v6.1 submitters used MLPerf's new API-centric test harness rather than the legacy in-process measurement approach. The new harness operates as a true client/server architecture over industry-standard APIs, more accurately reflecting how production inference systems handle requests in deployed environments.

MLCommons confirmed that MLPerf Endpoints — a next-generation benchmark suite built on this API-centric foundation — will eventually replace the current MLPerf Inference suite for datacenter measurements. "Moving forward, MLPerf Endpoints will replace Inference in our family of benchmarks for the datacenter, and the quick uptake of our API-centric harness will contribute to making that transition seamless," said David Kanter, Head of MLPerf.

For enterprise teams evaluating vendors today: results produced on the new API-centric harness are closer to what MLPerf Endpoints will measure than results produced on the legacy harness. When comparing submissions within v6.1, noting which harness each submitter used is now a material detail.

The full v6.1 results, including individual submission entries, are available through the MLCommons datacenter and edge benchmark results pages and the MLPerf interactive visualizer dashboard.

Frequently Asked Questions

What is the difference between the AgentX benchmark results released September 15 and the MLPerf v6.1 results released September 16?

The AgentX results announced at NVIDIA's AI Infra Summit on September 15 were produced on early silicon by a benchmark whose implementation documentation is hosted on NVIDIA's developer infrastructure, with initial results published through NVIDIA's blog network alongside a product announcement. MLPerf v6.1 results are submitted to and verified by MLCommons, an independent consortium with 130 members — including NVIDIA's direct competitors AMD, Intel, and Google — governed by bylaws designed to prevent any single vendor from controlling results. For procurement decisions, the governance structure of a benchmark's verification process is as material as the performance numbers: a result that an independent consortium has verified carries different evidentiary weight than one a vendor has self-reported. Both sets of numbers may be accurate, but they are not equivalent for due-diligence purposes.

My organization already has Blackwell hardware deployed. What does v6.1 actually tell us about whether to upgrade?

MLPerf v6.1 provides the first neutral, multi-submitter evidence of how much performance software optimization alone can add to deployed Blackwell hardware. CoreWeave documented a 19.8 percent per-GPU throughput gain on identical GB200 NVL72 hardware comparing v6.0 to v6.1. Lambda documented 8.85 percent server-scenario gains on identical Blackwell Ultra hardware. Intel documented 36 percent server-scenario gains on identical Arc Pro B70 hardware. If your current serving stack has not been updated since you deployed your hardware, these gains are accessible without new capital expenditure. A rational pre-upgrade checklist now includes: have you implemented piecewise CUDA graph capture? Have you enabled a Mixture-of-Experts expert auto-tuner if you are running mixture-of-experts models? Have you right-sized your KV cache using a dry-run method? If the answer to any of these is no, the software path likely has headroom before a hardware upgrade is the next productive step.

What is the E2E-RAG benchmark, and why should my team care about its results even if we are not building a RAG system?

The MLPerf E2E-RAG benchmark reveals something that applies beyond RAG specifically: the performance of a multi-component AI pipeline is not predictable from the performance of any individual model in that pipeline. A system that delivers fast single-model inference may become a bottleneck when re-ranking and embedding steps are added. The benchmark's introduction acknowledges that production AI deployments have shifted from single-model endpoints to coordinated pipelines — and that a procurement decision made on single-model benchmark scores alone is now systematically understating real-world compute requirements. If your production system uses an embedding step, a retriever, a re-ranker, or multiple models in any sequence, E2E-RAG results are more relevant to your infrastructure decision than a single-model tokens-per-second figure.

How does Speculative Decoding work, and does its inclusion in v6.1 change how I should interpret prior benchmark results?

Speculative Decoding is an inference optimization in which a small "draft" model generates a sequence of candidate tokens and a larger "verifier" model approves or rejects them in a single forward pass. When the draft model is right — which, for well-matched model pairs, it often is — the system produces multiple tokens at the cost of one forward pass of the verifier, increasing effective throughput without changing the output. The technique has been in widespread production use for over two years. Its omission from prior MLPerf rounds meant that official benchmark scores systematically understated the throughput of systems that had already deployed it. Its inclusion in v6.1 for two interactive-scenario benchmarks and the GPT-OSS task closes that gap: scores on these benchmarks now reflect what well-optimized production systems actually deliver. If you are comparing a v6.1 result against a pre-v6.1 result for the same hardware on the same model, the gap may partly reflect Speculative Decoding's inclusion rather than hardware differences.