Can You Trust an LLM's To-Do List for a Robot Fleet? This Paper Maps Exactly Where Typed Plans Break

A typed-plan validation system for NL-controlled robot fleets reveals 70-87% pass rates and specific failure modes: wrong formations, dropped robots, duplicate pickups. The gaps matter more than the wins.

Imagine you're a restaurant manager who gives verbal orders to twenty line cooks through a single dispatcher. The dispatcher writes each order on a structured ticket — protein, temp, sides, table number — and the kitchen executes from the ticket, not the voice. The interesting question isn't whether tickets work (they do, mostly). It's what goes wrong between the spoken order and the plated dish: a ticket that says 'medium rare' but gets read as 'medium,' a side that falls off the ticket entirely, two cooks both grabbing the same steak. This paper is about that gap — but for robot fleets. The committed claim: a three-level ROS 2 pipeline that converts natural-language commands into typed plans, applies deterministic corrections, coordinates a fleet of twenty simulated robots, and checks outcomes — plus a 44-case offline benchmark and an 18-run simulation series that together reveal exactly where the chain from language to action breaks. This is not a performance-maximization paper. It is a failure-cataloging paper, and that distinction matters. The architecture is a layered typed-plan system built on Robot Operating System 2, where LLM outputs are parsed into structured plan objects, validated against type constraints, corrected deterministically where possible, and then dispatched to fleet coordination. The offline benchmark checks plan structure and selected command requirements across five local LLM configurations without enforcing a response schema — yielding 70.0–87.3% pass rates. The simulation series runs eighteen cases where all runs meet pose-based task conditions, though two runs are replacements for incomplete runs after a software change, which the authors document rather than hide. What the paper actually contributes is a taxonomy of failure modes at the boundary between plan validation and execution correctness. The documented failures are specific and instructive: duplicate pickup assignments where two robots claim the same object, goal adjustments that stay within original task zones but deviate from the literal command, a formation plan that passes structural validation but produces the wrong shape, a route that gets correctly rejected, and an arrival report that silently drops a robot after replanning. These are not edge cases — they are the characteristic failure modes of any system that translates unstructured intent into structured execution. The integrity picture is honest but narrow. Validation is entirely internal: the authors built the system, wrote the benchmark, ran the simulations, and graded the results. There is no independent replication, no community benchmark for NL-to-fleet-plan correctness, and no physical robot deployment. The 44-case benchmark is hand-constructed and small. The 18-run simulation series is carefully documented — including the two replacement runs — but the absence of physical hardware means the entire validation stack is one layer of simulation checking another layer of simulation. The authors acknowledge this scope explicitly. The milestone question is where this paper is most useful as a lens. The field of natural-language robot control has been accumulating demos of LLMs generating plausible plans, but has been notably weak on systematic failure analysis. This paper moves the conversation from 'can an LLM produce a plan?' (yes, 70-87% of the time in constrained settings) to 'what categories of incorrectness survive plan validation?' That reframing — from generation success rate to correctness boundary taxonomy — is the actual contribution, even if the numbers are modest. The obvious next experiment is physical deployment with real robots, and the honest read is resource constraint. Running twenty physical robots through eighteen scenario series is expensive and logistically complex. The second obvious gap is scaling the benchmark beyond 44 cases — ideally with adversarial or community-contributed scenarios that probe the failure taxonomy more aggressively. The authors are clearly building toward both, but this paper is the simulation-stage checkpoint.