A logo of Nvidia is picture before the Nvidia/Japan AI Ecosystem Reception in Tokyo on July 16, 2026. Philip FONG/AFP via Getty Images

NVIDIA researchers published and open-sourced FlashREINFORCE on September 13, 2026, a reinforcement learning algorithm for training AI agents that matched or beat the dominant GRPO baseline across every tested task while consuming half as many rollouts — the fundamental currency of RL training compute — according to the FlashREINFORCE technical report. For any organization that fine-tunes AI agents on GPU clusters, the implication is direct: the same training budget could now run twice as many experiments, or the same experiment could run on half as much hardware.

The paper — authored by Jian Hu, Yifan Zhang, Hao Zhang, Binfeng Xu, Shaokun Zhang, Hongqing Peng, Zhiding Yu, Pavlo Molchanov, and Jan Kautz, all of NVIDIA — argues that the field's reliance on increasingly complex algorithmic scaffolding around policy gradient methods has been producing diminishing returns. The paper's central argument is captured in its title, Reinforcement Learning Should Do REINFORCE.

Why Agentic RL Training Has Become So Expensive

The core tension in training AI agents with reinforcement learning has always been between signal quality and compute cost. The agent must take actions, observe outcomes, and update its parameters in ways that reinforce successful behaviors — but extracting a reliable learning signal from a single trajectory is hard. The dominant solution that emerged through 2024 and 2025, Group Relative Policy Optimization (GRPO), which DeepSeek used to train the R1 model that drove a wave of efficiency-focused AI development, works by generating multiple responses to the same prompt and comparing them against each other. That group comparison is statistically clean, but it carries synchronized group training costs.

The first cost is raw throughput: every prompt requires G simultaneous rollouts (where G is the group size), meaning the model must generate G responses before learning anything from any of them. For short math problems where all responses finish in roughly equal time, this synchronization requirement is manageable. For the long-horizon tasks that define agentic AI — browsing the web, writing and executing code, multi-turn tool calls with variable response times — the slowest rollout in each group becomes the rate limiter. GPUs sit idle waiting for stragglers.

The second cost is the critic model. Although GRPO removed the learned critic that Proximal Policy Optimization (PPO) requires, conventional wisdom still held that some form of value estimation was necessary to stabilize policy gradient training without group structure. That assumption, FlashREINFORCE directly challenges.

A third failure mode emerged specifically when GRPO ran asynchronously — the configuration needed to avoid the synchronization bottleneck. Published research from mid-2026 documented that asynchronous GRPO exhibits a characteristic two-phase pattern: stable early improvement followed by abrupt, often catastrophic degradation. The Qwen team attributed similar collapses in large-scale GRPO runs to fundamental misapplication and invalidation of importance sampling weights, resulting in catastrophic and irreversible model collapse. On multi-turn tool-use tasks, GRPO's score collapsed to 30.3 at training step 600 and stopped invoking the Python tool altogether — a failure mode consistent with the documented GRPO tool-use instability pattern.

How FlashREINFORCE Fixes All Three Problems at Once

FlashREINFORCE returns to a principle that predates modern deep RL: the REINFORCE algorithm introduced by Ronald Williams in 1992, which updates a policy using a single trajectory weighted by its total return. The NVIDIA team's contribution is not the basic idea but the engineering solution to the three instabilities that have historically made naive single-rollout REINFORCE impractical at scale — high variance, policy staleness, and length bias.

One-Batch REINFORCE handles the variance problem. Rather than comparing a group of responses to the same prompt, FlashREINFORCE batches B independent trajectories from B distinct prompts, computes a batch-mean reward, and assigns each trajectory a signed advantage: trajectories scoring above the mean receive positive signal, those below receive negative signal. This gives the learner signed feedback without a critic and without sibling rollouts. Each fresh batch receives one full-batch update, then gets discarded — a design decision that prevents the additional policy drift that comes from reusing collected data for multiple sequential updates.

The coverage benefit compounds over training. A fixed rollout budget of N trajectories spent one per distinct prompt exposes the model to N different prompts; spending G rollouts per prompt reduces that to N/G distinct prompts. More prompt diversity means broader task coverage within the same compute envelope.

The Sequence Trust Region addresses policy staleness. In asynchronous training, a trajectory collected several policy updates ago carries a specific risk: the model that generated it (the "behavior policy") has drifted from the model currently being trained (the "current policy"). Token-level importance sampling — weighting each token's gradient by the ratio of current-policy probability to behavior-policy probability — corrects for this drift at the action level. But token-level importance sampling corrects only the conditional action distribution at stored histories; the histories themselves still come from the behavior policy. FlashREINFORCE's Sequence Trust Region addresses the accumulated drift across a full trajectory: it computes a mean Bernoulli KL proxy across all tokens in a trajectory, and rejects the entire trajectory if that mean exceeds a threshold delta. Rejection means zero gradient contribution — no corrupted update. The threshold is one hyperparameter rather than the dense machinery of value networks.

This design draws a direct architectural lineage to DeepMind's 2018 IMPALA framework, the asynchronous actor-learner architecture for game-playing agents that first demonstrated importance-weighted correction could stabilize asynchronous RL at scale. FlashREINFORCE extends IMPALA's core principle — decouple acting from learning, correct for staleness — into the specific constraints of language model training: token sequences instead of frame observations, variable-length trajectories instead of fixed-horizon episodes, and policy lag measured in update steps rather than wall-clock time.

Sample-Mean Optimization eliminates length bias. A naive token-averaged loss gives a failed thousand-token trajectory more negative signal than a failed ten-token trajectory purely because of its length. FlashREINFORCE normalizes each trajectory's contribution by its token count before summing across the batch, choosing to weight each trajectory equally regardless of how long it ran. Long failures don't dominate; short successes don't disappear.

Together these three mechanisms produce what the paper describes as "critic-free, one-pass asynchronous updates without waiting for sibling rollouts" — a training loop in which completed trajectories flow directly to the learner as soon as they finish, and the learner updates once per batch and moves on.

What the Benchmarks Show

The paper's headline result compared FlashREINFORCE to GRPO on mathematical reasoning, the task domain where GRPO has performed most reliably. Training Qwen2.5-Math-1.5B across five standard benchmarks — MATH-500, AMC23, Minerva, AIME25, and OlympiadBench — FlashREINFORCE scored 38.0 mean accuracy compared to 36.3 for GRPO, a gain of 1.7 percentage points. FlashREINFORCE used 256,000 rollouts to reach that result; GRPO used 512,000. Better accuracy at half the compute.

The contrast on tool-use tasks was sharper. Training Qwen2.5-7B-Instruct on Python tool use, FlashREINFORCE reached a mean score of 37.0 by step 600, with models averaging 3.25 tool calls per trajectory. The matched GRPO run scored 30.3 at the same step and had stopped invoking the Python tool — a collapse consistent with the documented instability of GRPO under asynchronous conditions. For any practitioner planning to train agents that use tools, this is the more consequential result: GRPO didn't just underperform, it stopped doing the thing the agent was supposed to learn.

Long-term training stability extended to larger models. Testing DeepSeek-R1-Distill-Qwen-1.5B, FlashREINFORCE remained stable through 6,000 update steps at policy lags of approximately four updates — conditions under which the model had updated four times since any given trajectory was collected. AIME24/25 mean accuracy improved from 21.7 to 33.7 over that run. Extending to a 30-billion-parameter Qwen3-30B-A3B mixture-of-experts model at lag-8 asynchrony — eight policy updates of staleness — FlashREINFORCE remained stable and outperformed a matched GRPO comparison by 6.8 points. On ALFWorld, an interactive text-based agent benchmark requiring multi-step decision-making, FlashREINFORCE achieved 98.3% success on seen tasks and 96.5% on unseen tasks after just 12,800 training trajectories.

These results are as of publication; no independent third-party reproduction has been published yet given the paper's September 13 release date.

What This Means for the Field

The paper's provocation — "Reinforcement Learning Should Do REINFORCE" — points at a broader implication that the results support: the algorithmic complexity that the RL-for-language-models field has accumulated since GRPO's introduction may represent engineering overhead rather than fundamental necessity. GRPO eliminated the learned critic that PPO required; FlashREINFORCE eliminates GRPO's group structure. Each step backward toward simpler methods has, in these experiments, produced better results.

Williams's 1992 REINFORCE algorithm was displaced as the standard policy gradient foundation largely because of its variance problem. The critic machinery that replaced it — and the group-sampling approach that simplified the critic — were engineering responses to that variance. FlashREINFORCE demonstrates that the variance problem can instead be managed with batch-mean normalization, trajectory-level drift screening, and length normalization, without adding network parameters or synchronization requirements. If those results replicate independently, the field has been paying a compute tax on a problem that has a lighter-weight solution.

For practitioners, the implication is immediate: the same training budget supports twice as many experiments, or equivalently, the same experiment runs on half as much hardware. For a domain where access to hundreds of H100s is the ceiling for most research teams, a 50% reduction in the rollout requirement is a meaningful expansion of what's tractable.

How to Use FlashREINFORCE Today

The algorithm shipped with an open-source reference implementation under the Apache 2.0 license, with integration into NVIDIA's Molt framework — the PyTorch-native agentic RL training stack that NVIDIA NeMo Labs open-sourced in July 2026 — completed in the same September 14 update. The Molt changelog entry — "2026-09: Molt now supports FlashREINFORCE, critic-free single-rollout RL with stable training beyond 6,000 steps" — marks this as native Molt capability rather than a research branch.

The GitHub repository includes PyTorch reference loss code, a CPU example for validating the loss without GPU infrastructure, and pinned Molt launchers for DeepSeek-R1-Distill-Qwen-1.5B on AIME and Qwen2.5-Math on reasoning tasks. Experiment settings cover reasoning, Python tool-use, mixture-of-experts runs, and ALFWorld. One honest disclosure the NVIDIA team included: the CPU example checks the optimizer update only and does not reproduce the paper's benchmark scores, which require GPU training.

A practical implementation note surfaces in the paper's design section: FlashREINFORCE stores the actual behavior probabilities at generation time rather than recomputing them later, because recomputation in a different inference context is not guaranteed to reproduce the exact probabilities the inference engine used. This is not a hidden limitation but a deliberate design choice that requires implementations to log the stored behavior log-probabilities as part of the trajectory. Molt handles this automatically.

Is NVIDIA Abandoning GRPO?

No. The Molt framework still ships all seven major advantage estimators including GRPO, and teams whose workflows are synchronized and whose tasks have short, regular rollout times may see no benefit from switching. The NVIDIA team's argument is not that GRPO is broken for every use case — it is that GRPO is structurally mismatched to the async, long-horizon, irregular-length trajectories that agentic tasks produce, and that those conditions are precisely where the field is headed as AI agents grow more capable. The benchmark results support that specific claim: on short mathematical reasoning tasks, FlashREINFORCE's advantage is real but modest (1.7 percentage points); on long-horizon tool use, where GRPO collapsed entirely, the advantage is categorical.

Frequently Asked Questions

What is GRPO and why has it been the dominant approach for training AI agents?

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that trains a language model by generating multiple responses to the same prompt, then reinforcing the responses that scored relatively better while penalizing those that scored worse. DeepSeek popularized it with the R1 model in early 2025, and it became the field's default partly because it eliminated the separate "critic" network that earlier algorithms like PPO required, cutting memory requirements roughly in half. Its limitation — which FlashREINFORCE targets directly — is that it requires synchronized groups of rollouts from the same prompt, which creates synchronization bottlenecks on long, variable-length agentic tasks, and which has proven unstable under asynchronous conditions where trajectories complete at different speeds.

How is FlashREINFORCE different from just running standard REINFORCE?

Standard REINFORCE (as introduced in 1992) uses the raw return of a trajectory — the sum of rewards — as the gradient weight. Without a baseline, this is high-variance: a single lucky trajectory can dominate training. FlashREINFORCE adds three targeted mechanisms that make the approach stable at scale. First, it subtracts a batch-mean baseline rather than a learned critic, providing signed advantages cheaply. Second, its Sequence Trust Region screens entire trajectories for accumulated policy drift using a token-level KL proxy, rejecting any trajectory where the policy has shifted too far from the one that generated it. Third, Sample-Mean Optimization normalizes each trajectory's contribution by its length before aggregating, preventing long failed trajectories from automatically dominating the gradient. Together these make the 1992 approach viable for the training conditions modern AI agents actually face.

Can I run FlashREINFORCE on models I'm already training with GRPO, and what do I need?

Yes. The code is available now at github.com/yifanzhang-pro/FlashREINFORCE under Apache 2.0, and integration into the NVIDIA Molt framework means it is available as a one-flag change for teams already using Molt. The key infrastructure requirement is that your rollout workers must store the actual behavior log-probabilities at generation time — the probabilities the inference engine assigned to each sampled token — alongside the trajectory, because the importance-sampling correction during training requires them. Molt handles this storage automatically. For teams using other frameworks, the reference implementation includes a standalone PyTorch loss and a CPU example to verify the implementation.

Does FlashREINFORCE replace the need for a GPU cluster, or just reduce how many GPUs are needed?

It reduces the rollout requirement, not the GPU requirement entirely. The paper's headline result — half the rollouts for equal or better accuracy — means a training run that previously required 512,000 rollouts can now achieve the same result with 256,000. In practice, this could mean running experiments in half the wall-clock time on the same cluster, or getting the same result on roughly half as many GPUs. The paper's 30-billion-parameter MoE experiment used a multi-node GPU cluster; FlashREINFORCE made that run stable at lag-8 asynchrony where GRPO was not, but large-scale agent training still requires substantial hardware. What changes is the denominator: for a fixed GPU budget, FlashREINFORCE expands what's achievable.