Science Prediction Universe (SPU), introduced in a preprint submitted to arXiv on November 11, 2026, tests whether AI agents can rank experimental outcomes they have not seen—not predict exact measurements. Its benchmark spans 116 tasks and 4,207 experiments across chemistry, materials science, biology and medicine.

The benchmark includes 100 public tasks drawn from 85 papers in 11 Nature Portfolio journals, plus 16 private tasks based on unpublished experimental series conducted with partner laboratories.

Agents submit rankings that are compared with withheld measurements using Spearman rank correlation; they can search the internet and use scientific software, but the information policy bars retrieving answers from the source studies.

To limit answer leakage and chance success, scoring runs in disconnected containers that block access to original results, and each task’s score is checked against a chance threshold based on 4,000 random rankings.

Each session gives an agent eight hours, one H20 GPU, an 8-core CPU and 16 GiB of memory, while the evaluation compares complete configurations that combine a model, its reasoning effort and its harness.

In the first evaluation, 12 models were tested across 13 configurations using Codex and Claude Code; public-task-adjusted mean correlations ranged from 0.227 to 0.478, and 34% to 52% of scored tasks cleared their task-specific chance thresholds. These are ranking correlations, not percentages of experiments correctly predicted.

Claude Opus 5.5 had the highest reported public-task mean at 0.478, but was scored on only 44 of 100 tasks because of safety refusals; statistical adjustment cannot establish how it would perform in those refused domains.

The leading configuration varied by field: Opus 5.5 in chemistry, Fable 5.1 in materials science, GPT-6 Astra Medium in biology and Opus 5 in medicine; Fable 5.1 and Opus 5.5 sometimes refused to answer.

Six leading configurations averaged about 0.15 points above the other seven and had fewer negatively scored tasks—about 9%, compared with as many as 22%—while also performing more reliably on experiment pairs with a correct combined reference ranking.

On the 16 private tasks, 12 configurations significantly outperformed random rankings after adjustment for multiple comparisons, but no statistically detectable pairwise differences emerged, so the small sample does not support a definitive ranking of models.

The public dataset release includes measurements, graders and reference solutions, so its tasks are not a held-out test set; private tasks remain unreleased. The release also omits audited web tools and source-identifier screening used in the paper, making results without those controls not directly comparable.

The authors note that public-task selection relied mainly on pilot scores from GPT-6 Astra and caution that the benchmark does not show AI agents can replace experimental validation; universities and research offices should weigh task coverage and evaluation controls alongside headline scores, and record the system configuration used.