1stproof.org

On October 14, 2026, an independent benchmark released its third batch of results, confirming that frontier AI systems can now solve approximately 90 percent of unpublished, expert-graded research mathematics problems — while the best publicly available open-weight model failed to solve a single one. The performance gap between the two categories, roughly ten months in effective capability, is substantially wider than the four-month lag Epoch AI measured across general AI benchmarks in May 2026. For the mathematicians who use open-weight models specifically to protect their unpublished theorems from commercial AI servers — a concern that has grown more concrete since September 2026, when two researchers disputed whether OpenAI's training data incorporated their work — the finding presents a direct tradeoff: the capable tools and the private tools are, at present, mutually exclusive.

First Proof: Benchmarking AI on Problems That Have Never Been Online

First Proof was created to fill a structural gap in how AI mathematical capability is evaluated. Most existing benchmarks draw on competition problems whose solutions have appeared in textbooks, online forums, or training corpora — making it impossible to know whether a model is reasoning or recalling. First Proof uses a different method: working mathematicians contribute unpublished lemmas from their own research, problems whose solutions do not exist anywhere online, spanning geometry, topology, statistics, dynamics, and other research-level domains. There is no possibility of pattern-matching against prior work. The First Proof Foundation mission reflects this design philosophy.

The benchmark evaluates AI systems on three criteria for scoring proofs: mathematical correctness, quality of exposition, and quality of attribution to prior literature. This tripartite scoring distinguishes it from most AI benchmarks, which focus on whether an answer is right or wrong and ignore the mathematical practices of clarity and citation that define professional mathematical output.

The initiative is run by First Proof Foundation nonprofit editors — Mohammed Abouzaid (Stanford), Nikhil Srivastava (UC Berkeley), Rachel Ward (UT Austin), and Lauren Williams (Harvard) — under the First Proof Foundation, a registered 501(c)(3) non-profit. Batch 3 funders and donors include DARPA, the Survival and Flourishing Fund, the AI for Math Fund, and Google.org, as well as unrestricted donations from Anthropic and OpenAI.

What the Author-Critic Architecture Actually Does

The top-performing system in Batch 3, ProofCouncil, was built by ProofCouncil ETH Zürich team led by Tim Gehrunger and Johannes Schmitt. Understanding why it outperformed a bare frontier model by two additional problems — and at a cost of $1,888 versus $36 — requires understanding what a harness actually does differently from a single model query.

ProofCouncil works through an author-critic loop architecture. An "author" agent — running on GPT-6 Astra Pro at maximum reasoning effort — writes a mathematical proof in LaTeX, consulting web search and code execution tools as needed. A "critic" agent independently evaluates each version and returns structured feedback. The author revises in response. Every three rounds, the stateful critic, which has accumulated conversation history and may have developed path-dependent blind spots, is replaced by a "fresh critic" that has not seen any prior version of the proof. A proof is submitted as a solution only if both the stateful and fresh critics accept it — a double-blind audit mechanism that catches errors a lone model would miss.

When the author reaches an impasse, it can invoke a ProofCouncil frontier model council: a panel of separate frontier models — Gemini 3.1 Pro, Claude Fable 5.1, and an additional GPT-6 Astra instance — each receiving the same question without seeing one another's responses. For problems involving concrete calculation, a compute worker uses SageMath and related tools including GAP and PARI/GP to verify intermediate claims. The process continues until a solution passes its double audit, until the cost budget is exhausted, or until 24 hours have elapsed.

The Batch 2 cost breakdown percentages are instructive: the author consumed 48.8 percent of total spend, the critic 31.6 percent, the GPT council member 11.3 percent, the Claude council member 3.6 percent, the Gemini council member less than one percent, and the compute worker 4 percent. The significant spend difference between the raw model ($36) and the harness ($1,888) reflects not the cost of better reasoning but the cost of independent verification — dozens of additional model calls whose sole purpose is to catch errors before submission. That architecture is what improved attribution from four of ten to seven of ten: a second model reviewing citations is more likely to notice what the author missed.

What the Open-Weight Model Did Instead

The Humanize system, built by a team led by Zhengyang Zhang and running on Kimi K3 — the open-weight 2.8 trillion-parameter model released by Beijing-based Moonshot AI in July 2026 — received a failing grade on every one of its ten submissions. Expert reviewers described its work as roughly equivalent to GPT 5.2 Pro, the frontier model from December 2025 — approximately ten months behind frontier models. The Humanize system Kimi K3 results show a run cost of approximately $2,000 in self-hosted compute, more than bare GPT-6 Astra while producing zero correct results.

The comparison against Epoch AI's four-month general benchmark lag is direct: research mathematics is an unusually demanding domain, and the open-weight deficit there is more than twice what independent measurement shows across general capability. Lauren Williams, a Harvard mathematician and First Proof editor, called the result blunt: "It really fell flat on its face."

How First Proof Draws Mathematicians Into a Privacy Dilemma

The open-weight deficit matters beyond the benchmark for a specific reason: many research mathematicians choose open-weight models precisely because they prefer not to submit unpublished work to commercial API servers.

Under the business and API terms of major proprietary AI providers, user data is by default not used for training — but the question of what actually happens to API inputs has become contested. In August and September 2026, two mathematicians reported that OpenAI appeared to have reached results similar to their own unpublished Navier-Stokes work, which they had discussed in private Codex sessions. OpenAI denied direct data access — neither its researchers nor AI agents had accessed the mathematicians' specific sessions — but acknowledged it could not entirely rule out that de-identified data drawn from user interactions had contributed to model improvements. One researcher noted that stripping names from conversations leaves the underlying mathematical ideas intact — meaning privacy tools designed for personal data may not protect unpublished theorems. Williams noted that open-source models for privacy are chosen exclusively by some in the mathematical community for this reason.

This is why First Proof's open-weight finding is more than a performance report. Researchers who want to keep their work private now have quantified evidence of what that preference costs them: not a matter of months but a near-year of effective capability.

Why the Gap Is So Much Larger in Research Mathematics

Research mathematics is not a domain where partial knowledge transfers easily. Correctly producing a proof of an unpublished lemma requires: identifying the right strategy from a vast solution space, carrying out multi-step reasoning without losing track of earlier commitments, citing the literature accurately enough to avoid reproving known results, and writing clearly enough that a human expert can verify the argument line by line. Each requirement is independently demanding; the combination is far harder than the arithmetic and symbol-manipulation tasks on which standard benchmarks measure AI progress.

The Epoch Capabilities Index four-month gap reflects performance across dozens of benchmarks spanning many domains — some of which open-weight models handle close to par with proprietary systems. Research mathematics selects for exactly the capabilities where the frontier is furthest ahead: multi-step reasoning under sustained constraint, independent verification, and strategic adaptability. The harness architecture amplifies the frontier model's advantage specifically because it applies multiple independent frontier models as verifiers — an approach open-weight systems cannot replicate without also paying proprietary API costs.

Compute Access as a Structural Problem

First Proof's press release flagged a concern that goes beyond the open-weight divide: the ability to improve on raw frontier model performance by spending more compute is itself becoming a compute barrier for basic research in mathematics.

GPT-6 Astra used alone, without a harness, solved seven of ten problems at $36 total — already a strong result. ProofCouncil solved nine of ten at $1,888, a cost that is moderate by industrial standards but that most university mathematics departments would find difficult to sustain across the weeks of exploratory work that precede a publishable result. Abouzaid on frontier math performance was direct: "for the core task the benchmark measures — proving well-defined statements — the models have become very good," adding that running frontier models on this task "might not be that relevant in the future" as the ceiling is approached. But the near-term implication is that the researchers most likely to benefit from AI in mathematics are those with the institutional resources to run expensive, multi-model orchestration pipelines — which maps closely onto researchers at well-funded institutions, not across the discipline as a whole.

Adam Sikora, a mathematician at the University at Buffalo whose quantum topology problem appeared in Batch 3, described the tension after ProofCouncil — and only ProofCouncil — solved his problem correctly. In a University at Buffalo news release, Sikora on ProofCouncil's cost described the system as "a more sophisticated and expensive system that coordinates several leading AI models together, and requires technical programming expertise as well as paid access to multiple high-end AI models." The implication is that the democratization story of AI in mathematics applies unevenly.

Exposition and Citation Gaps That Still Separate AI from Mathematicians

Even ProofCouncil's nine correct answers contained notable weaknesses in presentation. Roughly one-third of its submissions received negative average ProofCouncil exposition scores — meaning expert reviewers judged them below-acceptable in clarity and organization. AI-generated proofs tend to overexplain elementary steps while glossing over the genuinely novel arguments, a pattern that inverts what mathematical exposition demands. A human expert finding a solution to an unpublished problem would typically lead with the key insight; ProofCouncil's solutions often buried it.

Citation improved substantially — the harness achieved seven of ten adequate citations, compared to four for the bare GPT-6 Astra model and far worse for earlier batches — but the improvement required a compute worker scanned the literature in parallel. Producing a proof that is correct, well-written, and properly sourced simultaneously remains beyond what any system tested in Batch 3 consistently achieved.

What Comes Next

First Proof announced that Batch 4 in January 2027 will take place in the first half of that month. Given the now-large gap between frontier and open-weight performance, the editorial board plans to test the two categories separately, running problems calibrated to each tier to better understand how open-weight systems fail and whether targeted scaffolding can close any of the gap.

The same day the Batch 3 results appeared, Terence Tao — the Fields Medalist at UCLA whose blog directed the mathematical community's attention to the results — published Tao's call for collaboration stories for mathematicians to share experiences of human-AI collaboration in recent mathematics. The juxtaposition captures where the field actually stands: a benchmark in which every problem was solved correctly by some AI system, and a community trying to understand not whether AI can prove theorems but how mathematicians and AI can work most effectively together — a question the compute-access divide complicates before it even gets to the privacy one.

Frequently Asked Questions

What is the First Proof benchmark, and why is it different from other AI math tests?

First Proof is an independent benchmark run by a non-profit with editors at Stanford, UC Berkeley, UT Austin, and Harvard. Unlike standard AI math benchmarks, which use competition problems whose solutions often appear in training data, First Proof uses problems contributed by working mathematicians from their own unpublished research. Solutions do not exist anywhere online before the test runs. The benchmark also scores exposition and attribution separately from mathematical correctness — assessing whether an AI writes a proof that a human reader can actually follow and verify, not just whether it reaches a correct result.

Why does it matter that the performance gap is ten months rather than the four months measured on general benchmarks?

Epoch AI's measure of a four-month open-weight lag covers AI performance across many domains — including tasks where open models are relatively competitive. Research mathematics tests exactly the capabilities where frontier proprietary systems are furthest ahead: sustained multi-step reasoning, independent self-verification, and strategic exploration under a hard time constraint. The ten-month figure is not a discrepancy; it is what domain-specific capability differences look like when you measure the right thing.

Can researchers protect their unpublished work if they use frontier AI tools for mathematics?

This is genuinely unresolved. The major commercial AI providers exclude business and API users from training by default, but the limits of that protection became contested in September and October 2026 when two mathematicians disputed whether their unpublished Navier-Stokes work had influenced OpenAI's training data. OpenAI denied direct access to their sessions but could not fully rule out de-identified data contributions. Open-weight models self-hosted on a researcher's own hardware send nothing to external servers, removing that risk entirely — but as Batch 3 demonstrated, they are currently ten months behind on the tasks researchers most need them for. There is no option that is both capable and fully private at present.

How does ProofCouncil's author-critic architecture produce better results than a single model query?

ProofCouncil runs an iterative loop: an author model writes a proof, a critic evaluates it, the author revises. Every few rounds, a fresh critic with no prior context audits the proof independently — catching acceptances the stateful critic made through accumulated bias. When the author is stuck, a council of three different frontier models (from different providers) weighs in independently on specific sub-questions. A separate compute worker runs mathematical software to verify concrete claims. The result is that no single model's errors go unchecked — a human analog to the review process any mathematics paper undergoes before publication.