Prosaic Intelligence on X: "Today we’re releasing suicide-memory-bench v01.1 First of our consumer alignment evals set, a 2,048 stateful task set evaluation of memory retention within production AI-systems, testing whether distributed risk signals survive memory processing and condition downstream… / X
Prosaic Intelligence on X: "Today we’re releasing suicide-memory-bench v01.1 First of our consumer alignment evals set, a 2,048 stateful task set evaluation of memory retention within production AI-systems, testing whether distributed risk signals survive memory processing and condition downstream behavior The results expose a failure of consumer alignment in current AI systems. After repeated suicidal warning signs, even the highest scoring AI system (Gemini 4 Argon) failed to ask about user safety without providing risky assistance 89.8% of the time."
Today we’re releasing suicide-memory-bench v01.1 First of our consumer alignment evals set, a 2,048 stateful task set evaluation of memory retention within production AI-systems, testing whether distributed risk signals survive memory processing and condition downstream behavior The results expose a failure of consumer alignment in current AI systems. After repeated suicidal warning signs, even the highest scoring AI system (Gemini 4 Argon) failed to ask about user safety without providing risky assistance 89.8% of the time.
Today we’re releasing suicide-memory-bench v01.1 First of our consumer alignment evals set, a 2,048 stateful task set evaluation of memory retention within production AI-systems, testing whether distributed risk signals survive memory processing and condition downstream behavior The results expose a failure of consumer alignment in current AI systems. After repeated suicidal warning signs, even the highest scoring AI system (Gemini 4 Argon) failed to ask about user safety without providing risky assistance 89.8% of the time.
Suicide Memory Bench consists of 4 sub benchmarks that ablate across two axis: suicide vs. Suicide Crisis Syndrome (SCS), and single vs. multi-chat Personas are modeled on ACM U.S. Census data and publicly documented distributions of AI usage. Each persona-based task combinesShow more
In our single chat benchmarks, models fail on average 62.8% of the time when explicitly told about suicide, and 91.5% when presented SCS signs. That means a user can tell a model they are suicidal five times in the same chat, and still be provided a shopping link to a gun.
Across multi-chat tasks, failure rates increases drastically
- 93.3% failure rate with five explicit suicidal disclosures
- 99.5% failure rate when presented SCS signs Claude Haiku 5.5 went from responding correctly 47.4% to 0% of the time
Suicide Memory Bench is the first public release from our broader consumer alignment benchmark suite. We’re building evaluations that reflect production usage, and are excited to continue to release our findings. Links below.