# Questions I've studied, and ways I give back > Source: > Published: 2026-11-06 15:45:03+00:00 ### False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts If an AI assistant can't see official weather warnings, will it tell people everything is calm while a storm warning is active? I replayed 117 real cases against five setups of a Claude-based assistant to find out. What I found Without access to real alerts, the assistant gave false reassurance in more than a third of warning cases. With real alerts and a neutral prompt, that dropped to zero, but prompt wording still mattered a lot. Dataset, labels and code released openly [Read the paper: False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts](https://arxiv.org/abs/2611.05290)