CEA

Brain MRI and CT scans are conspicuously absent from the internet data that trained every major general-purpose AI model. Facial features embedded in neuroimaging make those scans legally untouchable for public datasets — which means GPT-5, Claude, and every other frontier model entered the clinic largely blind to what two decades of routine brain imaging actually looks like. A University of Michigan team published a paper in Nature Medicine on July 10 that addresses that blind spot directly: NeuroVFM, a specialist visual foundation model trained on 5.24 million consecutive clinical MRI and CT volumes from Michigan Medicine, outperformed GPT-5 by 21.4 percentage points on critical-findings triage in a prospective, real-world feasibility study run across the entire health system for one week in January 2026.

The result matters not just as a performance benchmark but as a proof of concept: a hospital's own accumulated scan archive — the full, unfiltered output of two decades of routine clinical care — is a viable pretraining corpus for specialist medical AI, without hand-labeled data, without curated datasets, and without the public internet's neuroimaging blind spot.

Private Clinical Data Is AI's Blind Spot in Medicine

Frontier AI models derive their broad capabilities from internet-scale data — billions of documents, images, and conversations scraped from the public web. Clinical medicine is systematically underrepresented in that corpus. Brain MRI and CT scans in particular rarely appear on the public internet because they contain identifiable facial features that restrict their distribution. The result is a structural gap: general-purpose multimodal models like GPT-5 have never encountered the breadth and depth of clinical neuroimaging that a practicing radiologist sees across a career.

The Michigan team describes this gap in blunt terms at the opening of their paper: frontier AI models "know the map." They have read about brain anatomy in textbooks, papers, and medical websites. What they lack is the territory — 566,915 consecutive clinical studies spanning trauma, neoplasms, ischemia, infection, and inflammation across every scanner, field strength, and acquisition protocol that a functioning academic health system accumulates over 20 years.

Their proposed solution is a concept they call "health system learning": rather than building AI from descriptions of the clinical world, train it directly on the raw data that clinical operations produce. The data is already there, sitting in every major hospital's picture archiving and communication system (PACS). The challenge has been building a learning algorithm suited to its structure, as the health system learning study published in Nature Medicine sets out to demonstrate.

Vol-JEPA: Predicting the Brain in Latent Space

The architectural innovation at the center of NeuroVFM is Vol-JEPA — Volumetric Joint-Embedding Predictive Architecture — a new self-supervised learning method that extends the JEPA family of techniques, originally developed for natural images and video by Meta AI Research based on Yann LeCun's theoretical proposals, to full three-dimensional medical volumes.

The mechanism works as follows. Each MRI or CT volume is first tokenized into non-overlapping 3D patches of 4×16×16 voxels. Every volume is then partitioned into a small visible context region and a larger masked target region — with both regions sampled exclusively from within a precomputed mask of the patient's head, so the model reasons about brain anatomy rather than memorizing background air. A trainable "student" encoder processes the context patches, producing context representations. A "predictor" module combines those context representations with positional information about the masked target patches to generate predictions. A "teacher" encoder — updated as an exponential moving average of the student, never directly trained by gradient descent — generates the ground-truth latent representations the predictor must match. The training objective is simply to minimize the smooth L1 distance between predicted and teacher latents.

What makes this fundamentally different from competing approaches is what is being predicted. Masked autoencoder methods (MAE-style) reconstruct raw voxel values, forcing the model to spend capacity on texture, noise, and acquisition artifacts. Contrastive methods require carefully constructed negative pairs. CLIP-style report-supervised approaches are bottlenecked by whatever a radiologist chose to mention in a free-text report — which systematically undercounts incidental findings and excludes everything that goes unremarked. Vol-JEPA predicts in latent space — abstract semantic representations, not pixels — which pushes the model toward neuroanatomically meaningful features while remaining entirely self-supervised and requiring no labels, no reports, and no paired text.

CT preprocessing added a further architectural detail worth noting: rather than a single grayscale rendering, each CT volume was processed through three windowing presets — brain, subdural, and bone — preserving distinct contrast ranges for hemorrhage, edema, and skeletal findings simultaneously. The model learned to handle all three during pretraining.

Pretraining the full NeuroVFM backbone required fewer than 1,000 GPU hours on NVIDIA L40S GPUs, running on the University of Michigan's Advanced Research Computing cluster. That computational cost is within reach of most major academic medical centers — a significant practical consideration for any health system that wants to replicate the approach with its own archive.

NeuroVFM Wins Every Comparison, Decisively

To isolate what the pretraining objective actually contributed, the researchers ran a controlled comparison across five baselines, all trained or evaluated on the same UM-NeuroImages dataset. The held-out test cohort covered more than 21,000 CT and 29,000 MRI studies from Michigan Medicine patients scanned between June 2023 and May 2024 — studies the model had never encountered during pretraining.

On the primary endpoint — macro-averaged AUROC across all 156 diagnostic tasks (74 MRI, 82 CT) spanning trauma, neoplasms, ischemia, infection, and inflammation — NeuroVFM achieved 92.68 on CT and 92.49 on MRI. It outperformed both report-supervised methods (HLIP: +0.98; PRIMA: +3.87) and voxel-reconstruction self-supervision (NeuroMAE: +1.55). The largest margins appeared against internet-scale baselines (DINOv3: +2.24; BiomedCLIP: +2.88), where differences in both objective and pretraining data compounded.

Because HLIP, PRIMA, NeuroMAE, and NeuroVFM all trained on the same UM-NeuroImages data, those gaps isolate the effect of the learning objective alone: latent prediction beats report supervision; report supervision beats voxel reconstruction; everything beats models trained on natural images. The hierarchy held across every pathology category in both modalities, as detailed in the published paper in Nature Medicine.

NeuroVFM also demonstrated substantially better label efficiency, requiring between 31.5% and 55.9% fewer labeled positive CT examples than each baseline to reach equivalent performance. In practical terms, a clinical application built on top of NeuroVFM needs less expensive expert annotation to reach deployment-grade diagnostic performance — an important economic advantage for health systems building downstream tools.

Performance continued to scale with both more data and larger encoders, with no signs of saturation even at 100% of UM-NeuroImages. That scaling behavior suggests health systems with even larger archives could push the model further.

Brain Anatomy Emerged, Without Being Taught

One of the most striking findings in the paper concerns what NeuroVFM learned without explicit instruction. When the team visualized patch-level embeddings using dimensionality reduction, they found that the model's internal representations spontaneously organized into neuroanatomically ordered clusters — gray matter clusters separately from white matter, cerebellar tissue separately from cortex — without any segmentation supervision or report-based anatomical labels during pretraining.

This emergent neuroanatomic map translated into measurable capabilities. In cross-protocol anatomical landmark matching — finding the same brain structure in images acquired with different MRI sequences and patient orientations — NeuroVFM achieved a 44.2% lower mean localization error compared to the voxel-reconstruction baseline (2.27 cm vs. 4.07 cm).

The model also proved capable of zero-shot cross-modal transfer: a diagnostic classifier trained on CT embeddings and then tested on MRI showed less than a 5-point AUROC drop, outperforming the voxel-reconstruction and internet-scale baselines on eight of nine shared diagnostic tasks. MRI and CT are physically very different modalities — one measures hydrogen relaxation, the other X-ray attenuation — but NeuroVFM learned a representation of brain anatomy that was modality-agnostic. That property, the authors argue, emerged specifically from the intersection of volumetric latent prediction and health system-scale data: it did not appear in any of the alternative training objectives applied to the same dataset.

Brain AI Report Generation: Beating GPT-5 and Claude

The second major experiment in the paper asked whether NeuroVFM's visual representations could serve as the perceptual backbone for an AI radiology reporting system. The researchers coupled the frozen NeuroVFM encoder with Qwen3-14B, an open-source language model, using a LLaVA-1.5-style visual instruction-tuning approach. A custom Perceiver-style connector compresses the potentially large number of visual tokens from a full neuroimaging study — which can include a dozen or more 3D scan sequences — into 64 fixed-length latent representations per sequence before passing them to the language model. The result is a system called NeuroVFM-LLaVA.

NeuroVFM-LLaVA was evaluated against GPT-5 and Claude Sonnet 4.5 on a curated test set of 300 expert-verified CT and MRI studies, balanced across three acuity levels: unremarkable, routine, and urgent. The comparison was designed to be fair to the frontier models: both GPT-5 and Claude Sonnet 4.5 were accessed through HIPAA-compliant agreements and given structured clinical prompts; NeuroVFM-LLaVA received the same studies as its native 3D-volumetric input.

According to the Nature Medicine study, NeuroVFM-LLaVA outperformed both frontier models on three-tier acuity accuracy (GPT-5: +11.0 points; Claude Sonnet 4.5: +20.3 points) and on detection of urgent findings (GPT-5: +10.5 points; Claude Sonnet 4.5: +21.9 points). Generated findings scored higher on automated language metrics against ground truth reports. The key finding error rate of NeuroVFM-generated reports was approximately half that of GPT-5 (10% vs. 20%), with fewer hallucinated findings and fewer laterality errors.

Laterality errors — reporting a finding on the wrong side of the brain — are among the most dangerous failure modes in radiology AI. That NeuroVFM showed meaningfully fewer of them than GPT-5 is a direct consequence of having learned from millions of actual clinical studies whose spatial structure is explicit in the imaging data, rather than from radiologists' descriptions of that structure in free-text reports.

Three blinded clinical experts preferred NeuroVFM-generated reports over GPT-5 reports more than two to one, with high agreement among the three reviewers (Fleiss' κ = 0.718).

One Week, 1,155 Real Patients: What Prospective Testing Showed

The most clinically significant component of the study was a prospective, silent feasibility evaluation run across the entire Michigan Medicine health system for one consecutive week — January 18 to 25, 2026 — without any influence on actual patient care.

Over that week, NeuroVFM-LLaVA processed every CT and MRI study of the brain, head, face, neck, and orbits performed in routine clinical operations: 1,155 studies in total, comprising 601 MRIs and 544 CTs. The test asked a specific clinical question: could model-generated reports accurately identify which studies contained critical findings requiring urgent clinical attention, as defined by the American Society of Neuroradiology's critical findings list?

The NeuroVFM arm achieved 92.6% balanced accuracy on critical findings and triage (95% CI 89.8–95.2%), compared to 71.2% for GPT-5 (95% CI 67.2–75.2%; P < 0.0001). Among the 187 studies that NeuroVFM flagged as potentially urgent, all 134 genuinely urgent cases were labeled urgent — a perfect within-flagged sensitivity. The over-triage rate was moderate: 53 non-urgent studies were flagged along with the 134 urgent ones.

The limitation is clear and the researchers state it plainly: 21 of 155 patients with a critical finding were missed entirely. These misses occurred because the NeuroVFM-generated report failed to identify the radiographic finding, so the screening model never flagged the study. That yields an overall sensitivity of 86.5% (95% CI 81.0–91.6%) — a meaningful clinical signal, but not the threshold required for unsupervised autonomous screening.

The model is not ready to replace radiologist oversight. It demonstrated, however, that a domain-specialist AI trained on health system data can operate with meaningfully better triage accuracy than frontier general-purpose models on real, consecutive, unselected clinical cases.

Can Any Hospital Build This?

The paper's most far-reaching claim is methodological: that uncurated electronic health records — the scan data that health systems already accumulate as a byproduct of delivering care — constitute a viable large-scale pretraining corpus for specialist AI, in the same way that public internet text serves as a pretraining corpus for large language models. The health system learning paradigm introduced in the paper formalizes this approach.

The parallel is intentional and explicit. Health system data, the Michigan team argues, offers something the internet never will for clinical medicine: ground truth about how disease actually presents in real patients, across the full distribution of scanner types, acquisition protocols, patient demographics, and pathological severity encountered in a functioning hospital. No curated dataset, however carefully constructed, captures the long tail of rare presentations and incidental findings that accumulate over two decades of unrestricted clinical care.

If the health system learning claim holds up under independent replication at other institutions, it suggests a scalable pathway for building specialist foundation models across medicine without extraordinary curation effort. The raw material is already there. The algorithm — Vol-JEPA — is now public, along with NeuroVFM's pretrained weights, available under open-source licenses at github.com/MLNeurosurg/neurovfm.

The Michigan team explicitly envisions NeuroVFM not as a replacement for frontier general models but as a specialist perceptual module that can plug into agentic AI systems to provide grounded clinical vision. In their prospective study, GPT-5 served as the downstream reasoning component assessing acuity from NeuroVFM-generated findings — a layered architecture in which domain-specific vision and general-purpose reasoning work together.

Not FDA-Cleared: What "Research Tool Only" Means for Deployment

Here is the deployment reality that the benchmark results do not resolve: NeuroVFM is explicitly not FDA-cleared for clinical use. The paper states this directly — "no treatment decisions were informed by NeuroVFM, as our model is not approved by the U.S. Food and Drug Administration" — and classifies the model as a research tool subject to institutional governance review before clinical deployment at any other institution.

That classification matters for how to read the 92.6% balanced accuracy figure. In a prospective silent study, where clinicians see and act only on the actual clinical record, an AI that achieves 92.6% accuracy on triage and misses 13.5% of urgent cases is impressive research progress. As an autonomous clinical screening tool without radiologist review, that same miss rate would be unacceptable. The distinction is between decision support — a tool that helps a radiologist prioritize a worklist — and autonomous screening — a tool that determines independently which patients get escalated.

No commercial AI radiology tool had achieved FDA clearance for standalone clinical reporting as of 2026. The field's standard deployment model, in place at organizations like Aidoc and Qure.ai, integrates AI as a second reader or worklist prioritizer that surfaces findings for clinician review, not as an autonomous decision-maker. NeuroVFM's architecture and the prospective results position it in that same category: a tool that can extend expert-level interpretation in under-resourced settings or assist radiologists in managing high-volume worklists, provided it operates under appropriate clinical oversight.

The path from these results to any clinical deployment also requires prospective validation under each institution's own regulatory and governance framework — a requirement the paper acknowledges explicitly and one that applies to any health system attempting to replicate the Michigan approach with its own data archive.

Frequently Asked Questions

What is NeuroVFM and how is it different from ChatGPT or GPT-5 for reading brain scans?

NeuroVFM is a specialist visual foundation model trained exclusively on 5.24 million clinical MRI and CT brain scans from Michigan Medicine's 20-year archive, using a self-supervised learning method called Vol-JEPA that teaches the model to predict what brain anatomy looks like in masked regions — without any labeled data or radiology report supervision. GPT-5 and other frontier general-purpose models were trained primarily on public internet data, which contains very few clinical brain scans because MRI and CT images include identifiable facial features. That means frontier models approach neuroimaging without having encountered the full diversity of clinical presentations. In a prospective one-week study on 1,155 real patients, NeuroVFM outperformed GPT-5 on critical-findings triage by 21.4 percentage points and produced radiology reports with half the key finding error rate.

What does the study's 86.5% sensitivity figure mean for patients?

In the prospective study, NeuroVFM correctly identified urgent cases — brain tumors, hemorrhages, strokes, and other findings requiring immediate clinical attention — in 134 of 155 patients who had such findings. It missed 21, for a sensitivity of 86.5%. For context: GPT-5's sensitivity on the same study was substantially lower, and NeuroVFM's miss rate was approximately four times lower than GPT-5's. However, 86.5% sensitivity is not sufficient for an autonomous screening tool that operates without radiologist oversight — missing nearly 1 in 7 urgent cases would be clinically unacceptable in a setting where delayed diagnosis causes permanent disability. The researchers are explicit that NeuroVFM is a research tool and a potential decision-support aid, not a replacement for radiologist review.

Is NeuroVFM ready for clinical use, and can other hospitals replicate it?

NeuroVFM is not FDA-cleared and was evaluated under secondary data usage with University of Michigan IRB approval. Any clinical deployment at another institution would require prospective validation under that institution's own regulatory and governance framework. The model code and pretrained weights are publicly available under open-source licenses at github.com/MLNeurosurg/neurovfm, and the paper describes the full methodology for assembling an equivalent training corpus from any large academic medical center's PACS system — meaning institutions with comparable scan archives can in principle attempt replication. The computational cost (fewer than 1,000 GPU hours on current hardware) is within reach of major academic medical centers. Independent replication at other health systems is the critical next step before broader deployment claims can be made.

Does NeuroVFM raise data privacy concerns given that it was trained on patient scans?

The NeuroVFM pretraining used deidentified clinical data under University of Michigan IRB approval (HUM00229133) as secondary data usage, and all processing occurred on the university's HIPAA-compliant computing infrastructure — no patient data was shared with external commercial AI providers during training. The language model component (Qwen3-14B, an open-source model from Alibaba) was run locally as part of the NeuroVFM-LLaVA system on the same secure infrastructure, so patient imaging data did not pass through external servers. The pretrained model weights released publicly contain no patient data — they encode learned representations of anatomy and pathology, not recoverable images or records.