MLPerf Training v6.1 Adds Post-Training Test; NVIDIA, Google, CoreWeave Results Not Comparable
1 day ago / Read about 27 minute
Source:TechTimes

Gerd Altmann/ Pixabay

MLCommons on Monday published MLPerf Training v6.1, which adds the first post-training benchmark to the suite and the first results from NVIDIA's Rubin-generation GPUs and Google's Ironwood TPU. The headline times for the new Qwen3.5-397B reinforcement learning test, 10.44 minutes on Rubin, 18.02 on Ironwood and 22.78 on a GB200 NVL72 system, came from clusters of different sizes that split their work differently, so they show what each system did rather than which chip is faster. Anyone planning a training cluster, or reading a vendor's "fastest" claim, needs to know what these numbers can and cannot tell them.

Twenty-four organizations submitted 153 systems that produced 250 results, and 57 of those systems (37%) ran in the cloud, up from 29% in the v6.0 round six months ago. Twelve systems used 1,000 or more accelerators, and the largest used 10,240. MLCommons' release does not name winners. Each submitter reports how long its system takes to train a model to a fixed quality target, and the comparisons in this article come from vendor write-ups of the published results, not from MLCommons.

Post-Training Loop Splits Work Between Generation, Training and Sandbox CPUs

The new Qwen3.5-397B benchmark is the first post-training workload in MLPerf Training, and it uses Group Relative Policy Optimization (GRPO) to teach the model to solve software-engineering tasks, including tool calling. GRPO is a variant of Proximal Policy Optimization (PPO) that researchers introduced in a February 2024 paper to optimize memory usage. Working group co-chair Shriya Rishab said "Post-training has added a new level of complexity to the architecture of AI training."

That complexity is concrete. In reinforcement learning, a model generates its own attempts, an automatic check scores them, and the model updates from the scores. The benchmark's reference design has the agent receive a coding problem in a Linux container, produce a fix patch, and earn a binary pass or fail reward, with hidden test files so the model cannot game the reward. Each attempt is limited to 30 agent turns and a 65,536-token context, and the model samples 16 generations per prompt. The model has 397 billion parameters but activates only 17 billion per token, because it routes each token to 10 of 512 specialist subnetworks, called experts, plus one shared expert.

An RL run therefore has three jobs that must keep pace with each other: updating the model, generating attempts, and running tests in sandboxes on conventional processors. The reference implementation is built on NVIDIA's NeMo-RL, with Ray coordinating training and rollout generation. SemiAnalysis, the semiconductor research firm, framed the engineering problem in June as matching trainer and generator throughput: if the generator is slower the trainer starves, and if it is faster the samples go stale. In one case study the trainer sat idle about 74% of the time. The result is that the measured time depends on how a submitter divides its hardware between jobs, not only on how fast one accelerator multiplies matrices.

Rubin, Ironwood and GB200 Reached Target at Three Scales and Three Splits

NVIDIA reports its Vera Rubin NVL72 finished the Qwen3.5 GRPO run in 10.44 minutes using NeMo-RL with TensorRT-LLM, and in 10.94 minutes with the vLLM inference backend, against 13.78 minutes for GB300 NVL72. NVIDIA's post ties that 10.44-minute and 13.78-minute comparison to 1,024 GPUs, while a separate per-GPU table lists a 64-GPU Qwen3.5 comparison drawn from different submissions, so readers should check which entry a figure comes from.

Google Cloud reports that 640 Ironwood TPUs reached 0.765 accuracy on the coding task, against a 0.69 target, in 18.02 minutes, and that code generation and learning ran at the same time so the training chips waited under 3% of the time. CoreWeave reports 22.78 minutes on 256 GB200 GPUs across four racks, with 128 GPUs on training and 128 on generation.

The three results differ in chip generation, chip count and workload split, so no single variable explains the gap. TechArena reports 29 submissions on the Qwen3.5 GRPO test, so the field is not thin. What is thin is the set of comparable pairs that vendors have disclosed.

Read more: MLPerf v6.1 Puts First Peer-Reviewed Numbers on Vera Rubin: Software Gains Beat New Hardware

Vera Rubin NVL72 Claims Up to 2x Per-GPU Gain Over Its Own GB300 Baseline

NVIDIA says Vera Rubin NVL72 posted the fastest time to train on six of the seven benchmarks in its first MLPerf Training appearance, and claims up to 2x the per-GPU performance of an optimized GB300 NVL72 baseline. On DeepSeek-V3 671B at 1,024 GPUs, the times were 4.14 minutes for Rubin and 8.03 minutes for GB300. NVIDIA reports 1.6x on gpt-oss 20B (11.213 against 18.24 minutes at 64 GPUs), 1.6x on Llama 3.1 8B (7.162 against 11.541 minutes at 72 GPUs), and 1.4x on DLRMv4 (2.829 against 3.871 minutes at 128 GPUs). It lists 1.5x on Llama 2 70B LoRA and FLUX.1.

The one benchmark NVIDIA did not win outright was DeepSeek-V3. A 10,240-GPU GB300 NVL72 cluster spread across 160 racks set the round's fastest absolute time there, 1.38 minutes, and a Microsoft Azure GB200 NVL72 submission also reached 10,240 GPUs. NVIDIA says its largest Rubin run spanned 16 racks and scaled with efficiency above 97% from 256 to 1,024 GPUs.

NVIDIA also reports unverified reruns on a newer software stack, including DLRMv4 at 1.628 minutes against the 3.871-minute GB300 submission, or up to 2.4x. MLCommons has not verified those reruns, and they are not a like-for-like hardware comparison. CoreWeave says it was the first cloud provider to deliver multi-rack Vera Rubin results, and reports 17.65 minutes for DeepSeek-V3 on 256 Rubin GPUs across four racks against 48.40 minutes for its own GB200 system at the same GPU count, which is 2.7x. On Llama 3.1 8B, a single 72-GPU Rubin rack took 7.16 minutes against 13.70 for GB200, a 1.9x gap. CoreWeave's footnotes say the speedups are its own analysis of published results, and that MLCommons has not verified them.

Ironwood Reaches DeepSeek-V3 Target in 1.70 Minutes on 8,192 TPUs

Google Cloud submitted 10 results across Ironwood TPUs and NVIDIA Grace Blackwell GPUs. On pretraining, 8,192 Ironwood TPUs reached the DeepSeek-V3 target in 1.70 minutes, and Google says the pretraining run scales to 4,096 chips with 96% efficiency. Google also submitted GB300 and GB200 systems, including a 256-GPU GB300 DeepSeek-V3 run at 31.96 minutes.

The DeepSeek-V3 numbers do not line up as a head-to-head. NVIDIA's 1.38-minute record used 10,240 GPUs and Google's 1.70-minute run used 8,192 TPUs, so the two times describe what each system can do at its own scale, not which chip is better.

AMD Instinct MI355X Trails B300 by Single Digits and Leads B200 on Llama Tests

AMD submitted on five of the seven benchmarks, using Instinct MI355X and MI350X, and its own framing is that the MI355X trails NVIDIA's B300 by 4.7% on Llama 3.1 8B and 7.4% on Llama 2 70B LoRA while running about 12% ahead of the B200 on both. Six partners (Crusoe, Dell, HPE, MiTAC, Oracle and Supermicro) reproduced AMD's results across four benchmarks, averaging about 96% of AMD's reference performance over 20 matched 8-GPU runs.

On DLRMv4, AMD says a 64-GPU run from Crusoe beat a B300 system by 39%, but that figure compares a partner submission against a different vendor's submission. AMD notes that its multi-node scaling code was tuned mostly for single-node runs and may improve, and that an unofficial DLRMv4 improvement missed the submission deadline and is unverified.

DLRMv4-HSTU Replaces Click-Prediction Recommender With Attention-Based Architecture

The second new workload, DLRMv4-HSTU, replaces DLRM-dcnv2. It uses the HSTU architecture, which Meta researchers introduced in 2024 as a sequential transducer for generative recommendation, and trains on the Yambda dataset, which MLCommons describes as one of the largest open-sourced recommendation datasets. Yandex released Yambda from its Yandex Music service, with 4.79 billion user-item interactions. TechArena counts 20 DLRMv4-HSTU submissions.

Which MLPerf Training Comparisons Hold Up Across Vendors?

The comparisons that hold up are the ones MLCommons builds itself. To separate new hardware from new software, MLCommons compares best times at fixed accelerator counts, and compares within a constant accelerator family. By that method, the best FLUX.1 time fell 2.49x in six months, from 17.12 to 6.88 minutes, at 512 accelerators in both rounds. Every result in the closed division's "available" category comes from systems that can be rented or bought now.

MLCommons working group co-chair Pavan Yalamanchili said the round marks an exciting new threshold for the benchmark suite. The group has also added visualization tools at results.mlcommons.org/training that compare systems and rank results among similar-scale systems.

Read more: AgentX Benchmark: Vera Rubin NVL72 Achieves 30x Efficiency Gain Over GB300 for AI Agents

Each vendor claim above uses a different baseline: an older generation of the same vendor's chips, a competitor, a different cluster size, or an unverified software build. Readers who want to compare systems should pull the closed-division results from MLCommons and match them by benchmark and accelerator count. On the Qwen3.5 GRPO test, add one more check: ask how many accelerators ran generation and how many ran training, because that split is part of the result.

Frequently Asked Questions

Can I compare Rubin's 10.44 minutes with Ironwood's 18.02 minutes on the Qwen3.5 GRPO test?

No. The two runs used different chip generations, different chip counts and different splits between generation and training, and NVIDIA's post cites several GPU counts for its Qwen3.5 entries. A fair comparison needs the same benchmark at the same accelerator count in the closed division.

What is MLPerf Training, and what does time to train measure?

MLPerf Training measures how long a system takes to train a defined model to a specified quality target. MLCommons released the previous round, v6.0, six months before this one.

What is GRPO and why does it matter for AI training hardware?

GRPO is a reinforcement learning method that samples several attempts per prompt, scores each one and updates the model, and variants of it are widely used to build reasoning and coding models. It matters for hardware because generating attempts, training on them and running tests in sandboxes all have to keep pace with one another.

Did NVIDIA win every benchmark in MLPerf Training v6.1?

NVIDIA says Vera Rubin NVL72 led six of the seven benchmarks, and NVIDIA's own 10,240-GPU GB300 cluster set the fastest DeepSeek-V3 time. Those are NVIDIA's own tallies, and MLCommons' release does not name winners.