You know how a mechanic can listen to your car idle for ten seconds and tell you the transmission is going? They're not running every diagnostic — they're extracting a rich signal from a small observation and placing your car in a mental map of every car they've seen. This paper does that for AI agent evaluation: instead of running an agent through hundreds of expensive benchmark tasks, watch it solve one, extract a structured summary of its behavior, and predict where it lands relative to every agent you've already evaluated. The committed claim: by representing agent execution traces as LLM-generated structured summaries (not raw logs), you can build a geometric embedding space where a new agent's position among known reference systems predicts its full benchmark score. On SWE-bench Verified, this drops mean absolute error from 0.13 to 0.08 given a single executed task — a 38% reduction with Wilcoxon p < 10⁻⁴. On Terminal-Bench 2.0, the improvement is 0.14 to 0.12 (p < 10⁻³). The key insight is that raw traces are contaminated with system-identity and model-family signatures that make the embedding space reflect 'who built this agent' rather than 'what can this agent do.' The LLM summary step strips those confounds. The architecture is a two-stage pipeline bolted onto an item-response theory (IRT) model. Stage one: for each task execution, an LLM extracts a structured summary of what happened — tool calls, reasoning steps, failure modes — and a standard text encoder embeds that summary. Stage two: given embeddings for all reference systems across all tasks, they build a low-dimensional representation of each system (its position in capability space) and use geometric proximity to predict a new system's score. The IRT backbone is classical psychometrics — the same math that makes adaptive standardized tests work. The novelty is augmenting it with trace geometry rather than relying solely on binary pass/fail. The ladder matters here. The baseline is a standard IRT model using only correctness signals from reference systems — a well-established efficient benchmarking approach. This paper doesn't replace IRT; it augments it. The 0.13 → 0.08 MAE improvement on SWE-bench is meaningful because at these scales (predicting scores between 0 and 1), shaving 5 points of absolute error from a single-task observation is the difference between 'noisy guess' and 'actionable estimate.' The paper is honest that the Terminal-Bench improvement is smaller (0.14 → 0.12), and they report dominance across all task budgets on both benchmarks when using the full reference set. Integrity is reasonable but bounded. SWE-bench Verified and Terminal-Bench 2.0 are community benchmarks with public task sets — not synthetic toy problems. The Wilcoxon signed-rank tests are appropriate for paired comparisons and the p-values are strong. However, the evaluation is simulation-only: they're predicting scores of agents that have already been fully evaluated, using held-out subsets. There's no independent replication, no pre-registration, and the choice of these two specific benchmarks (both coding-heavy) leaves open the question of whether the trace-summary approach transfers to non-coding agent domains. The reference systems are whatever agents have already been run on these benchmarks — a convenience sample, not a designed experiment. The practical milestone is clear: if you run an agent evaluation lab, the threshold that matters is 'can I trust a 5-task estimate enough to skip the other 495?' This paper says yes for coding benchmarks, with ~0.08 MAE. The next number to watch is whether this holds below 0.05 MAE on a third, non-coding benchmark — that would make trace-based prediction a standard tool in agent evaluation pipelines, not just a SWE-bench trick. The obvious experiment not run: applying this to non-coding agent benchmarks (web browsing, tool use, multi-modal tasks) where trace structure is fundamentally different. My read is (a) — they had access to SWE-bench and Terminal-Bench reference runs and didn't have comparable trace corpora for other domains. The LLM summary step might need different prompting or fail entirely when traces aren't code-centric. This is the load-bearing question for generalizability.
