{"slug": "testing-what-users-see-when-an-ai-safety-check-fails", "title": "Testing what users see when an AI safety check fails", "summary": "A maintainer of Child AI Safety, an Apache-2.0 developer preview, built an offline test harness for Koda, an app that teaches kids how AI works, to verify what reaches the screen when a model's safety check fails after a draft already exists. The harness runs without API keys using simulated providers and exercises four paths — approved answer, refusal, support response, and a simulated detector outage — asserting that the fallback message renders and no part of the held draft is emitted, including in streaming event sequences. The project is soliciting a first contribution in issue #8 for a runnable example of local counting of approved and unavailable requests at the gateway.", "body_md": "When we built Koda, an app that teaches kids how AI works, we had a question that a normal answer-quality eval couldn't settle. If the model has a draft ready and the safety check fails, what actually reaches the screen?\n\nWe built an offline test around that boundary. The application holds the draft while checking it alongside the original question. The gateway rechecks permission before it delivers an approved answer. If the detector is unavailable, the application can display a predefined message without showing the draft.\n\nWith Node.js 22 or later, run:\n\n```\ngit clone https://github.com/SamSquareAI/child-ai-safety.git\ncd child-ai-safety\nnpm ci --ignore-scripts\nnpm run --silent demo\n```\n\nThe demo uses invented prompts and simulated providers. It runs without API keys. The terminal shows four paths: an approved answer, a refusal, a support response, and a simulated detector outage. In the outage case, the learner-facing message says Koda can't answer right now and suggests trying a learning question later.\n\nRun the test through the same delivery path your UI uses. Make the safety adapter return unavailable after a draft exists. Assert that the application renders the fallback and that no part of the draft was emitted. In a streaming app, that means checking the entire event sequence, including chunks sent before the final message. Test an ordinary approved answer too, so you know both paths work.\n\nI help maintain [Child AI Safety](https://github.com/SamSquareAI/child-ai-safety), the Apache-2.0 developer preview that came out of this work. If you'd like a small first contribution, [issue #8](https://github.com/SamSquareAI/child-ai-safety/issues/8) asks for a runnable example showing how the gateway counts approved and unavailable requests locally.\n\n*Drafted with AI and checked against the repository's offline demo.*", "url": "https://wpnews.pro/news/testing-what-users-see-when-an-ai-safety-check-fails", "canonical_source": "https://dev.to/asamassekou/testing-what-users-see-when-an-ai-safety-check-fails-b81", "published_at": "2026-11-15 19:11:36+00:00", "updated_at": "2026-11-15 19:20:54.315359+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "developer-tools", "ai-tools"], "entities": ["Child AI Safety", "Koda", "SamSquareAI", "Node.js", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/testing-what-users-see-when-an-ai-safety-check-fails", "markdown": "https://wpnews.pro/news/testing-what-users-see-when-an-ai-safety-check-fails.md", "text": "https://wpnews.pro/news/testing-what-users-see-when-an-ai-safety-check-fails.txt", "jsonld": "https://wpnews.pro/news/testing-what-users-see-when-an-ai-safety-check-fails.jsonld"}}