A Benchmark That Asks Whether AI Agents Can Build Better AI Agents — and One Already Scores Above Human Baselines

Singularity-Bench tests whether AI agents can autonomously build core AI systems from scratch. Opus 5.5 scores 1.25× human baselines across all six tasks. The rest of the field averages 0.58.

Imagine you're a master chef running a restaurant, and you ask your sous chef not just to cook tonight's menu, but to redesign the kitchen — the ovens, the ventilation, the supply chain, the recipes, the training program for new hires — from a near-empty room. That's the structural claim of Singularity-Bench: not "can agents do tasks" but "can agents build the systems that do the tasks." The six tasks span the entire AI development pipeline, from pre-training data infrastructure through inference optimization to cluster orchestration, and the agent starts from a minimal scaffold each time. The committed claim is stark: Opus 5.5 with Claude Code achieves a Singularity-Index of 1.25, meaning it matches or outperforms mature human-engineered baselines on identical hardware across all six end-to-end system-building tasks. The authors frame this as evidence that at least one frontier agent "may have the skills needed to reach the singularity" — a deliberately provocative phrasing that conflates benchmark performance with recursive self-improvement capability. The remaining 10 agents, spanning 7 frontier models, average only 0.58, meaning the field overall is still well below human baselines. The benchmark architecture is genuinely well-structured. Six tasks cover pre-training, mid-training, post-training, inference, agent harness design, and cluster orchestration — the full stack of what it takes to build and deploy an AI system. Each task evaluates against a mature human-engineered baseline on identical hardware, which is the right experimental design. This is not a toy sandbox; the tasks require implementing real systems from near-scratch. The benchmark also introduces Breakwater, a hardened evaluation framework that patches reward-hacking vulnerabilities found through systematic red-teaming of the existing Harbor framework, covering 9 of 10 OWASP Top 10 categories for agentic applications. The ladder position is mixed. Comparing against human-engineered baselines is the right move, and doing so on identical hardware is honest. But the "Singularity-Index" metric — a ratio of agent performance to human baseline — is the authors' own construction with no external calibration. There's no comparison to other agent benchmarks (SWE-bench, METR's tasks, ARC-AGI) on overlapping capabilities. The 1.25 score for Opus 5.5 is striking, but we're trusting the authors' task design, difficulty calibration, and baseline selection simultaneously. The 0.58 average for other agents provides useful contrast but also raises questions about whether the benchmark is measuring something specific to Claude Code's harness design rather than general agent capability. Integrity concerns center on three issues. First, the benchmark was designed and evaluated by the same team, with no independent verification yet. Second, the "human baseline" is defined by the authors — we don't know how optimized those baselines are or whether they represent median, expert, or state-of-the-art human engineering. Third, the Singularity-Index is a novel metric without external validation. The red-teaming work on Harbor/Breakwater is a genuine contribution, but it also means the authors are grading agents on a framework they themselves hardened, creating a circularity risk. Code availability would significantly change the integrity picture. The milestone framing is the most important lens here. The gap between Opus 5.5 at 1.25 and the field average at 0.58 suggests either Opus 5.5 is genuinely exceptional at system-building tasks or the benchmark has agent-specific affordances. The next meaningful milestone would be a second agent from a different model family crossing 1.0 on independent hardware — that would confirm the benchmark measures general capability rather than model-specific fit. The successor experiment the authors didn't run is the obvious one: having the agent-built systems actually train a model that itself performs on downstream tasks, closing the recursive loop. The honest read is that this is computationally expensive and logistically complex, but it's also the experiment that would justify the "singularity" framing. Without it, we have agents building components, not agents building agents.