Questions I've studied, and ways I give back

A replayable evaluation of 117 real National Weather Service alert cases found that a Claude-based assistant without access to official warnings gave false reassurance in more than a third of warning cases, according to the paper "False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts" posted to arXiv. Supplying real alerts with a neutral prompt dropped false reassurance to zero, though the author reports prompt wording still mattered substantially. The dataset, labels and code were released openly alongside the paper.

False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts If an AI assistant can't see official weather warnings, will it tell people everything is calm while a storm warning is active? I replayed 117 real cases against five setups of a Claude-based assistant to find out. What I found Without access to real alerts, the assistant gave false reassurance in more than a third of warning cases. With real alerts and a neutral prompt, that dropped to zero, but prompt wording still mattered a lot. Dataset, labels and code released openly Read the paper: False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts https://arxiv.org/abs/2611.05290