AI Agents Struggle to Predict Unseen Experiments in New Science Benchmark

November 14, 2026
AI Agents Struggle to Predict Unseen Experiments in New Science Benchmark
  • To limit answer leakage and chance success, scoring runs in disconnected containers, and each task’s score is checked against a threshold based on 4,000 random rankings.

  • Across 13 configurations using 12 models, public-task-adjusted mean correlations ranged from 0.227 to 0.478, and configurations beat task-specific chance thresholds on only 34% to 52% of scored tasks; these correlations measure ranking agreement, not the percentage of experiments predicted correctly.

  • Claude Opus 5.5 had the highest reported public-task mean, 0.478, but was scored on only 44 of 100 tasks because of safety refusals, so statistical adjustment cannot show how it would perform in the domains it declined.

  • The top average score varied by field: Opus 5.5 led in chemistry, Fable 5.1 in materials science, GPT-6 Astra Medium in biology and Opus 5 in medicine; Fable 5.1 and Opus 5.5 sometimes refused to answer.

  • Coverage was uneven: all 13 configurations had predictions for just 28 tasks in common, and two Claude configurations were scored on only 44 and 30 public tasks, largely because of safety refusals.

  • An isolated grader evaluates those rankings against withheld measurements, while the information policy bars agents from retrieving answers from the source studies.

  • A preprint introducing the Science Prediction Universe (SPU) benchmark asks whether AI agents can rank outcomes of experiments they have not seen, reporting results across 116 tasks and 4,207 experiments in chemistry, materials science, biology and medicine.

  • The benchmark includes 100 public tasks based on 85 published papers and 16 private tasks drawn from unpublished collaborative experiments.

  • Agents get up to eight hours per session, one H20 GPU, an 8-core CPU and 16 GiB of memory, and can use internet retrieval and scientific software; they submit rankings that are compared with experimental results using Spearman correlation.

  • Agents agreed more closely with one another than with measured outcomes, averaging a Spearman correlation of 0.52 between configurations versus 0.33 against measurements; their agreement exceeded their correlation with measurements on 81 of 100 tasks.

  • Consensus was not reliably more accurate: on one oncology task, 11 configurations agreed at 0.9 while their average correlation with measurements was 0.27, and the best individual system outperformed consensus on matched tasks.

  • On the 16 private tasks, 12 configurations performed significantly better than random rankings after adjustment for multiple comparisons, but no statistically detectable pairwise differences emerged, leaving too little evidence for a definitive model ranking.

Summary based on 3 sources


Get a daily email with more AI stories

More Stories