Last week at Protege’s “Evaluating Healthcare AI — Benchmarks to Real-World Trust” event, our CEO Kevin Brown and I shared the first results from an independent, double-blind cancer outcome prediction benchmark. It pits the Standard Model against GPT-5.6-sol, Claude-Opus-5 and Gemini-3.7-flash.
The short version: on lung cancer, the Standard Model posted the highest headline score of any model tested, and on breast cancer was within the 95% confidence intervals of frontier models. Across both cancers, it delivered its predictions at 36–170 times lower cost and 54–83 times lower median inference time than the frontier models. It did so with only 3 billion parameters and zero network calls.
Why we needed a new benchmark
Until now, there has been no common framework for evaluating models on tasks related to real world outcomes, including disease progression and survival times, in oncology indications. . The usual obstacles are familiar to anyone who has run a clinical AI evaluation:
Key diagnostic modalities are often missing from public datasets.
Most datasets aren’t longitudinal, so they can’t show how a patient’s disease unfolds.
Restrictive data use agreements prevent sending patient data to frontier model APIs.
Locked-down compute environments prevent running custom models.
No guarantees that frontier models have not already trained on the public data.
For pharma and biotech teams deciding which model to trust with trial design or patient stratification, that leaves a gap. Vendor claims based on results from different test data sets are hard to compare, and performance on familiar datasets may reflect memorization rather than real predictive skill.
Protege controlled the test data, the execution and the scoring, so no model, including ours, could see the answers. The design:
Double blind, SMB never saw the test data or had the ability to hill climb on results, and Protege developed that data without knowledge of our model or its architecture
Two indications, breast and lung cancer, with pancreatic cancer coming soon
Net-new data that had never been sold or trained on, covering 700+ longitudinal clinical timelines that the hospital system has agreed not to sell again.
EHR and radiology (DICOM) inputs, with pathology (WSI) coming soon.
Two real-world tasks: overall survival (OS) and time to next treatment (TTNT), with four more clinical tasks in progress.
Ablations on each modality: every model was scored on EHR alone and on EHR plus other modalities, so that we could assess the differential evidence each modality provided vs EHR alone, as well as on a headline arm that included whatever was available.
Third-party harnesses for the frontier models, so we didn’t build the setups we were compared against.
The safeguards matched the stakes. Every task and ablation ran in a fresh, isolated container on frozen, SHA-256-verified inputs. The Standard Model ran with no network access at all; the API models reached only approved destinations through a proxy. Hidden labels arrived only after every inference container had been torn down, and scoring used frozen rules with pinned models and recorded predictions.
Results: the Standard Model is comparable to frontier models with far fewer parameters and less training data
We used this benchmark to test whether the Standard Model’s world model architecture could deliver competitive performance at its current scale: 3 billion parameters trained on data from roughly 50,000 patients. Comparing it directly with models from frontier labs helps us assess how well the approach works today. It also gives us a baseline for measuring what we gain as we increase model size and training data.
On lung cancer, the Standard Model scored the highest headline C-index of the four models:, ahead of Claude-Opus-5, Gemini-3.7-flash and GPT-5.6-sol. Across the 18 paired lung comparisons (three competitors, two tasks, three data settings), it came out ahead in 11. Those margins sit within the confidence intervals, so the fairest reading is that a 3B-parameter specialist model held its own against the largest frontier systems on one of oncology’s hardest prediction problems.
Despite training on data from only about 1,000 breast cancer patients with limited longitudinal follow-up, our model performed nearly as well as the frontier models. Only Claude Opus 5’s advantage on TTNT was statistically significant. Combined with our high performance in lung cancer, this gives us confidence that our world model architecture can capture real signal, even at smaller scales.
The Standard Model returned each prediction in a median of 2 seconds for under a cent. The frontier models took 108–166 seconds and cost 0.32–1.53 per prediction.
At that scale, the whole benchmark cost about $20 to run on the Standard Model, versus roughly $720–$3,450 on the frontier models. This gap compounds as the data scales to fill real-world data programs. Additionally, because the Standard Model makes zero network calls, it can run inside a sponsor’s own secure environment, where restrictive data agreements often rule out sending patient records to an external API.
This is the first round of what will be a much larger comparison: Pancreatic cancer, pathology inputs and four more clinical tasks are in progress, and we’ll publish those results as they land. As we extend the comparison to more objectives and patient groups, we will keep growing the Standard Model itself. We plan to increase its parameter count and train it on even more high-quality patient data, then measure whether each new version improves prediction on the benchmark tasks. We’re grateful to Protege for building a benchmark that holds every model, including ours, to the same standard.
Frontier-level prediction on hard cancer endpoints no longer has to mean frontier-level cost, latency or data exposure. The Standard Model is a 3-billion-parameter model trained on roughly 50,000 patients. It turns a patient’s assessments, imaging, pathology, genomics and interventions into a single treatment-conditioned patient state. Teams can use this patient representation to then read out the endpoint they care about at any horizon: survival, treatment response, toxicity, stratification, or their own custom endpoint, through a lightweight task head trained on their own labels.
If you’re evaluating models for trial design optimization, patient identification and stratification, or real-world evidence, we’d love to talk.
Reach us at info@standardmodel.bio.