Imagine calling a friend who lives across the country and asking about the weather outside your window. They check vibes — time of year, your city's climate norms — and say 'should be fine.' Meanwhile a tornado warning is blaring on your local TV. That's what LLM weather assistants do when they lack real-time alert data: they confabulate normalcy from priors. This paper measures exactly how often, and the answer is disturbingly high. The core claim is clean: an LLM-based preparedness assistant without access to real NWS alert data produces 'false reassurance' — responses that leave users believing conditions are safe when an official warning is active — at rates between 37.7% and 44.9%. The study replays 69 real alert cases through five configurations of a Claude-based pipeline, varying whether the model sees actual alerts and how the prompt is designed. The production pipeline with alert access scores a perfect 0% false reassurance rate (0/66, 95% CI 0.0–5.5%). The blind configurations fail catastrophically. The most counterintuitive finding: explicitly telling the blind model to disclose that it cannot see alerts made false reassurance worse, not better — 44.9% versus 37.7% without the disclosure instruction (McNemar p = 0.40, no significant difference, but directionally worse). The model appears to treat the disclosure instruction as a framing cue that it should still try to be helpful, and 'helpful' in weather context defaults to reassurance. Every false reassurance case was implicit — no model ever explicitly denied an alert existed. It just talked around the gap. Alert data alone was necessary but not sufficient for safe behavior. With a neutral prompt and real alerts, the model raised alarms on 87.5% of no-alert control cases — a massive false-positive problem. A single production instruction in the prompt explained much of this gap, demonstrating that prompt engineering and data access are joint requirements. The paper also documents two real system failures: NWS test products being ingested as real alerts, and a token-budget overflow that fails on complex multi-hazard scenarios. The integrity setup is unusually honest for an applied AI paper. The seven-category labeling scheme was developed post-hoc after an initial binary pass, and the authors disclose this explicitly rather than pretending it was preregistered. A single rater labeled all 342 alert-case responses, which is a limitation but one acknowledged openly. The automated regex classifier they initially built missed 53 of 57 true false-reassurance cases, so they abandoned it and reported the failure — a rare act of methodological candor. The study is small and single-model (Claude only), single-rater, and confined to U.S. NWS alerts. The 69 alert cases and 32 controls provide meaningful signal but wide confidence intervals. The paired McNemar design is appropriate for within-case comparisons but the sample limits generalizability claims. The paper's value is less in the specific numbers and more in establishing a replayable evaluation methodology: frozen cases, controlled alert access, structured labeling. This matters because weather-safety AI assistants are already deployed and the failure mode is invisible. A hallucinating chatbot that makes up facts is detectable; one that calmly reassures you while a warning is active is dangerous precisely because the output looks correct. The paper's central contribution is making this failure mode measurable and showing that the intuitive fix — 'just tell the model it's blind' — does not work.
