Imagine you're a chess player who can instantly see that a square is attacked — but can't play a whole game without blundering into checkmate. You have local threat detection nailed. What you lack is the ability to hold multiple threats in your head across a sequence of 15 moves, where an early safe-looking choice forecloses your only safe path later. That's the core mechanism CAPSafe is measuring: the difference between recognizing a hazard in isolation versus reasoning about safety across a chain of dependent actions in a changing environment. The committed claim: LLMs can identify whether a single household action is unsafe given its environmental context at >90% accuracy, but safe long-horizon planning — generating full task plans where no step introduces an unmitigated hazard — remains substantially harder, with even the best model (Gemini-2.5-pro) reaching only 78.19% safe success rate and most models stuck below 65%. This is a benchmark contribution, not a methods paper: CAPSafe provides the ruler, not the fix. The benchmark is structured at two levels. The action-level dataset asks binary questions: is this specific action safe in this specific context? The plan-level dataset is harder — it presents complex household tasks (think: cleaning a kitchen where there's a gas leak AND a wet floor AND a child present) and asks models to generate multi-step plans that avoid all hazards. Crucially, hazards are situational and co-occurring: a knife is safe in a drawer but not on a counter near a toddler. Each hazard has a severity annotation, enabling fine-grained analysis of which failure types are most dangerous. Nine models were tested — both closed-source (Gemini-2.5-pro, GPT-4o, Claude-3.5-sonnet) and open-source (Llama-3.1-70B, Mistral, DeepSeek-R1, among others). The ladder is where things get interesting. On action-level safety, several models exceed 90% accuracy — the problem is basically solved for single-step hazard recognition. But plan-level safety collapses the rankings. Gemini-2.5-pro leads at 78.19% safe success. GPT-4o and Claude-3.5-sonnet land in a cluster around 60-65%. Open-source models generally trail further. The gap between action-level and plan-level performance is the paper's core finding: recognizing danger is not the same as planning around it over long horizons. This parallels a known pattern in LLM reasoning — local coherence doesn't guarantee global coherence. The integrity picture is mixed. The benchmark is new, so there's no prior community standard to anchor against — CAPSafe IS the proposed standard. The evaluation covers nine models, which is a reasonable breadth. However, the dataset construction methodology (how hazards, severities, and gold-standard plans were determined) is where you'd want to scrutinize: human annotation processes for safety judgments are inherently subjective, and inter-annotator agreement will be key. No code or dataset release is explicitly mentioned in the abstract, which is a gap for replicability. The benchmarks weren't pre-registered — this is a first publication establishing the benchmark itself. The milestone framing is clear even if the paper doesn't state it: the field needs plan-level safe success rates above 95% before embodied LLM agents can be trusted in real households with real hazards. We're at 78% best-case today. The gap between 78% and 95%+ is where the next 2-3 years of work lives — and it's not obvious that scaling alone closes it, because the failure mode is compositional reasoning over dependent constraints, not raw knowledge. The obvious next experiment the authors didn't run: testing whether chain-of-thought prompting, explicit safety-constraint planners, or tool-augmented approaches (where the LLM calls a safety-checker at each plan step) close the gap. My read: they're saving the interventions for the follow-up paper — this one establishes the problem and the ruler.
