A New Benchmark Tests Whether LLMs Can Play Project Manager, Not Just Coder

WorkflowArena formalizes 'workflow orchestration' as a distinct capability — can an LLM decide how to split work among agents? 145 tasks, 9 models, and the answer is: delegation strategy matters as much as raw intelligence.

Imagine you're a film director who's also the screenwriter, casting director, and line producer. You don't just need to be good at each job — you need to decide which jobs to delegate, to whom, in what order, and how the pieces fit back together. That organizational layer is a skill distinct from any individual craft. WorkflowArena tests whether LLMs have this skill. The committed claim: workflow orchestration — an LLM's ability to autonomously design and execute multi-agent delegation structures — is a measurable, separable capability that existing benchmarks miss. Prior benchmarks either measure final output quality (did the code compile?) or test coordination under human-specified structures. WorkflowArena is the first to let the model decide its own org chart and then evaluate both the deliverable and the org chart jointly. 145 tasks across 8 professional domains (software engineering, formal math, legal analysis, 3D production, and more), each requiring an integrated deliverable with interdependent components, graded by hidden verifiers. The ladder here is interesting because there's no clean prior benchmark to beat. The closest relatives are SWE-bench (code tasks, single-agent focus), GAIA (multi-step reasoning), and various agent-coordination testbeds that predefine the workflow structure. WorkflowArena's contribution is orthogonal: it doesn't compete on the same metric as these benchmarks, it opens a new axis. Nine models were tested, and no single model-configuration dominates all eight domains — a healthy sign that the benchmark has genuine discriminative texture rather than just measuring raw scale. Architecturally, this is an evaluation framework, not a new model. It wraps existing agent harnesses (Claude Code, OpenAI Codex) that already support multi-agent spawning and workflow authoring. The key measurement innovation is recording all delegation events — who spawned whom, with what instructions, in what topology — so you can analyze organizational patterns post-hoc. Claude models in Claude Code tend toward fewer, larger delegation batches; GPT models in Codex prefer more frequent, smaller-unit delegation. These are genuinely different strategies, not noise. The integrity picture is mixed. The hidden verifier design is smart — it prevents models from gaming the evaluation — but the verifiers themselves are author-designed, not community-standardized. No pre-registration. No independent replication. The 145 tasks were curated by the team, which introduces selection bias risk. The most striking result — that Claude Opus 5.5's workflow plans substantially improve weaker executor models — is tested on a 'challenging subset' whose selection criteria aren't fully transparent. The cross-model transfer experiment is the strongest evidence that orchestration is separable, but it needs replication. The milestone question is where this gets practical. Today we have 145 tasks and qualitative strategy differences. The next meaningful number is independent task-set construction — when a second team builds 200+ tasks in overlapping domains and the same strategy differences replicate. That would confirm orchestration as a stable capability axis, not an artifact of this particular task set. Roughly 12-18 months out if the benchmark gains traction. The obvious experiment not run: testing whether models can improve their own orchestration through feedback. The paper measures static strategy differences but never closes the loop — if you show a model that its delegation pattern scored poorly, can it restructure? This is likely (c) being saved for the next paper, because it's the natural extension and would require a more complex experimental setup the authors clearly have the infrastructure to build.