Research Agents Fail When the Desk Is Messy: A Benchmark That Finally Tests AI in Real Working Conditions

ARE-bench shows frontier AI agents pass 66% of tasks in clean conditions but crater to 28-40% when forced to respect prior decisions and messy workspaces — the gap most benchmarks hide.

Imagine you hire a brilliant new intern and hand them a pristine spec doc on day one. They crush it. Now imagine handing that same intern a laptop with six half-finished branches, three conflicting config files, a Slack thread of decisions they weren't part of, and saying 'pick up where I left off.' That second scenario is what ARE-bench measures — and it's where today's research agents fall apart. The core claim is direct: existing research-agent benchmarks overestimate agent capability by testing in clean-room conditions. ARE-bench constructs 75 tasks drawn from real researcher sessions — not synthetic puzzles but actual mid-project requests where the workspace contains unfinished files, outdated versions, and failed prior attempts. Each task is anchored at a real user request with accumulated context from earlier session decisions still in effect. The benchmark spans seven research stages (including Design and Execution, which prove hardest), 33 requirements, 150 sub-requirements, and 1,173 binary checklist items. The headline result is devastating for agent optimists. The best frontier model passes every checklist item on only 66.2% of tasks. But the more telling number is the gap: when you check only delivery contracts and general norms — basically, 'did you produce output that looks right?' — three models hit about 69%. When you add historical context requirements and workspace-state requirements — 'did you respect what came before and work with what's actually there?' — those same models crash to 28.4%–40.0%. That 30-point gap is the benchmark's core contribution. It quantifies the difference between 'looks correct' and 'actually advances the project.' The benchmark design deserves credit for ecological validity. Tasks come from recruited researchers working on their own projects, not contrived scenarios. Workspaces are replayed to the task boundary without cleanup — the mess is the point. The evaluation rubric is intent-aligned, meaning any valid solution path gets credit, not just the path the benchmark designer imagined. This addresses a common criticism of rigid automated evaluation. Architecturally, ARE-bench is an evaluation scaffold, not a model. It tests black-box agents against a structured rubric of binary checklist items, which means it's model-agnostic and can track progress across the field. The seven-stage taxonomy (spanning research lifecycle from literature review through execution) gives granular signal about where agents break. Design and Execution being hardest for every model tested suggests a real capability ceiling around multi-step reasoning with state, not just information retrieval. The integrity picture is mixed. The tasks come from real researchers, which is strong, but the evaluation is author-team scoring against their own rubric — no independent replication yet, no pre-registration. The 1,173 binary checklist items provide granularity, but it's unclear how much inter-rater reliability was measured. Seven frontier models were tested, which is reasonable breadth, but the paper doesn't name them (at least not in the abstract), making it hard to track which architectures struggle where. The practical upshot for anyone building or deploying research agents: your agent's benchmark score on clean tasks is probably 30 points higher than its real-world delivery rate. The next milestone this field needs is not a higher score on ARE-bench but an agent that closes the context-gap — passing workspace and historical requirements at the same rate as delivery-only checks. That gap is the delta between 'demo-ready' and 'actually useful.'