
Read the paper
doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving
on GitHub
Imagine you're a sous chef and the head chef says: "Sear the steak, then once the asparagus arrives from the walk-in, blanch it, and keep the hollandaise warm the whole time." That's three tasks — one immediate, one event-triggered, one persistent — and you have to hold all three in your head while the kitchen evolves around you. Current autonomous driving planners are like a sous chef who can only remember the last five seconds of instructions. doPlan is the first dataset that writes down the full multi-stage ticket. The committed claim: doPlan is, to the authors' knowledge, the first publicly available, human-annotated real-world dataset designed to study passenger language as persistent task context for autonomous driving. Built on top of Motional's nuPlan platform, it contains 5,154 human-written passenger instructions spanning 169.1 hours of cumulative instruction-aligned driving context across 50.9 hours of unique driving data. Annotation windows range from 30 seconds to 508.8 seconds — an order of magnitude beyond the typical 5-second planning horizon used by most language-conditioned driving models. The ladder result is damning for existing models. The authors evaluate four language-conditioned driving architectures and find that "sensitivity to passenger language does not reliably translate into behavior consistent with the requested direction." Among 2,161 examples containing a matched future maneuver, the median time to the first relevant maneuver is 24.6 seconds after the evaluation point, and only 9.8% of those maneuvers fall within the standard 5-second prediction window. This is not a marginal miss — it's a structural mismatch between what passengers actually say and what planners can see. Architecturally, doPlan itself is a dataset and benchmark, not a model. The contribution sits in the annotation ontology: five categories of passenger intent (immediate, deferred, event-conditioned, persistent, and multi-stage) mapped onto real sensor logs with human-written natural language. The evaluation pipeline feeds these instructions to existing language-conditioned planning models and measures whether the model's planned trajectory aligns with the intended maneuver direction. The four tested models represent the current crop of language-to-trajectory architectures — the paper exposes their shared blind spot rather than proposing a fix. The integrity picture is solid for a dataset paper. Annotations are human-written (not LLM-generated), the underlying driving data comes from nuPlan (a well-known community resource), the dataset and annotation interface are publicly released on GitHub, and the evaluation is transparent about what the models fail at. The weakness is that the four evaluated models are not named with full SOTA context — we know they fail, but the paper positions itself more as a diagnostic than a horse race. No pre-registration, but that's standard for dataset contributions. The milestone framing is clear: current planners operate on a ~5-second horizon; 90.2% of real passenger intent requires reasoning beyond that window. The next concrete threshold is a planner that can retain and ground unresolved goals across at least 25 seconds (the median maneuver delay), ideally across the full annotation windows of 30–509 seconds. No existing model demonstrated this capability. The field needs to move from reactive trajectory prediction to persistent goal tracking — and doPlan provides the first public benchmark to measure progress. The obvious experiment not run: training or fine-tuning a model on doPlan's multi-stage annotations and showing that persistent-context architectures (memory-augmented transformers, hierarchical planners, or explicit goal stacks) actually close the gap. The authors built the diagnostic but stopped short of the treatment. Honest read: this is likely scoped as a dataset paper for a venue submission, with the model paper coming next. The infrastructure is in place — the training loop is the sequel.
The analogy
You're a sous chef and the head chef gives you a compound ticket: sear the steak now, blanch the asparagus when it arrives from the walk-in, keep the hollandaise warm throughout. Current self-driving planners only remember the last 5 seconds of the ticket — they sear the steak and forget the rest. doPlan writes down the full ticket and times how long it takes the kitchen to need each instruction.
Kitchen ticket
Multi-stage cooking orders with immediate, deferred, and persistent tasks that a sous chef must track across a shifting kitchen
doPlan dataset
Multi-stage passenger instructions with immediate, deferred, event-conditioned, and persistent intent that a planner must track across evolving driving scenes
Honest assessment
doPlan is a well-constructed diagnostic that makes a sharp, quantified case for a real gap: current language-conditioned driving planners cannot see past 5 seconds, but 90% of real passenger intent requires it. The contribution is the benchmark, not a solution. This is a load-bearing dataset paper for a subfield that was previously benchmarking against artificially simple commands. The limitation is obvious — no model trained on this data is shown to close the gap — but that's the right scope for a first dataset release. Practitioners building language-enabled AV systems should use this as their new smoke test.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine gap-fill: first public dataset for persistent multi-stage language intent in driving. Not a model contribution — a benchmark contribution. The claim is narrow but the gap is real and well-documented.
The field fight
Can end-to-end language-conditioned planners handle persistent multi-step passenger intent, or do autonomous vehicles need explicit goal-tracking modules?
Side A
End-to-end models will learn to retain context implicitly as architectures scale and training data improves
Held by: Language-conditioned planning community, LLM-as-planner approaches
Side B
Explicit persistent goal tracking is structurally necessary — short-horizon planners cannot solve this by scaling alone
Held by: Hierarchical planning, task-and-motion planning, this paper's diagnostic evidence
Milestone tracker
Unlocks: Planners that can retain and execute multi-stage passenger instructions in real driving — prerequisite for conversational autonomous vehicles
Current
5
seconds
current planner horizon
Next
25
seconds
median maneuver delay
No model demonstrated this; timeline unknown
Verified Facts (8)
doPlan contains 5,154 human-written passenger instructions spanning 169.1 hours of cumulative instruction-aligned context over 50.9 hours of unique driving
Annotation windows ranging from 30.0 to 508.8 s
Among 2,161 examples with a matched future maneuver, the first associated maneuver occurs a median of 24.6 s after the evaluation point
Only 9.8% occur within the models' common 5 s prediction horizon
Built on nuPlan
Dataset, annotation interface, and supporting resources are publicly available at https://github.com/Mi3-Lab/doPlan
Annotations capture immediate, deferred, event-conditioned, persistent, and multi-stage passenger intent
Four language-conditioned driving models evaluated
Practical Implications (4)
AV teams should benchmark language-conditioned planners against multi-stage instructions, not just single-turn commands — doPlan provides the first public test for this
Planner architectures need persistent goal tracking mechanisms (memory modules, goal stacks) to handle the 25s+ intent horizon revealed here
nuPlan users can layer doPlan annotations directly onto their existing sensor logs for language-grounded planning research
Product teams building conversational AV interfaces should expect current models to silently drop deferred and event-conditioned instructions
More latest news
Read all