{"slug": "show-hn-psychosis-guard-safety-for-long-llm-conversations", "title": "Show HN: Psychosis Guard – Safety for long LLM conversations", "summary": "Developer nwjang released psychosis-guard, an open-source, model-agnostic safety middleware that tracks cumulative risk across long LLM conversations and escalates graduated interventions when a conversation drifts toward delusional reinforcement. On the preliminary psychosis-bench evaluation (n=16, one repetition), an unguarded chatbot scored 42% on \"delusion not confirmed,\" 60% on \"harm not enabled,\" and 15% on \"safety intervention offered,\" while a safety prompt plus psychosis-guard scored 71%, 91%, and 87% respectively. The tool runs as an HTTP proxy or Python library in front of OpenAI, Anthropic, or OpenAI-compatible servers such as Ollama and vLLM, and its authors label it a research and educational tool that is not a medical device and whose intervention copy requires mental-health professional review before use with real users.", "body_md": "**Trajectory-aware safety middleware for long-running LLM conversations.**\n\nEnglish | [한국어](https://github.com/nwjang/psychosis-guard/blob/main/README.ko.md)\n\nA chatbot can validate a user's delusional belief a little more on every turn\nwhile no single message ever trips a content filter. psychosis-guard tracks the\n**whole conversation**, scores where it is heading, and steps in with a graduated\nintervention before that drift compounds. Open source, model-agnostic, runs as an\nHTTP proxy or a Python library, and works with no API key on deterministic mocks.\n\n| psychosis-bench metric | unguarded chatbot | safety prompt + psychosis-guard | \n|---|---|---|\n| Delusion not confirmed | 42 % | **71 %** | \n| Harm not enabled | 60 % | **91 %** | \n| Safety intervention offered | 15 % | **87 %** | \n\n<sub>psychosis-bench · n=16 · one repetition · preliminary evaluation. Share of the\nideal score; full table, ablation and limitations in [Evaluation](#evaluation).</sub>\n\nConventional guardrails inspect one message at a time. psychosis-guard adds a\n**Trajectory Rail**: a pipeline stage that tracks cumulative risk over the whole\nconversation and escalates a graduated, clinically-informed intervention when the\nconversation is heading the wrong way, even when every individual turn looks\nharmless.\n\nIt runs as an HTTP middleware in front of any chatbot (OpenAI, Anthropic, any OpenAI-compatible server such as Ollama or vLLM, or a bot you already operate), or as a Python library.\n\n**Disclaimer.** Research/educational tool. **NOT a medical device, NOT diagnosis\nor crisis support.** In an emergency contact local emergency services or a crisis\nline. Intervention copy and prompts are marked `ADVISER-REVIEW` in the source and\nmust be reviewed by a mental-health professional before use with real users.\n\n- Python 3.10+\n- Optional: an OpenAI or Anthropic API key, or any OpenAI-compatible endpoint. Without one, the whole pipeline runs on deterministic mocks.\n\n```\npip install psychosis-guard                            # core library (mocks only)\npip install \"psychosis-guard[server,openai,anthropic]\" # HTTP server + real LLM adapters\n```\n\nExtras: `server` (FastAPI/uvicorn), `openai`, `anthropic`, `all`.\n\nFrom source, for development (tests, lint):\n\n```\ngit clone https://github.com/nwjang/psychosis-guard.git\ncd psychosis-guard\npip install -e \".[dev]\"\n```\n\nEvery user turn passes through five rail stages. Stages 1–3 and 5 mirror the input / dialog / output / action rails of NVIDIA NeMo Guardrails; Stage 4 is the new, cumulative-state stage.\n\n| Stage | Rail | What it does | \n|---|---|---|\n| 1 | **Input Rail** | Pre-response risk estimate from the user turn plus the prior trajectory slope. HIGH short-circuits the chatbot and returns a safe response. | \n| 2 | **Dialog Rail** | Mode A: injects a graduated system prompt into the chatbot call. | \n| 3 | **Output Rail** | A judge scores the reply: reinforcement, sycophancy, pushback, escalation, help-referral. | \n| 4 | **Trajectory Rail** ★ | Folds the turn into cumulative state and computes the least-squares slope of delusion density over the conversation. | \n| 5 | **Action Rail** | Composite risk → `NONE / LOW / MEDIUM / HIGH` ; Mode B rewrites the reply. | \n\n`composite_risk = Σ wᵢ · signalᵢ + slope_boost · max(slope, 0)`\n\nOnly an *escalating* trajectory raises risk. A falling slope is not rewarded, so\ninterventions do not switch off while density is still high.\n\nKey benefits:\n\n- **Trajectory awareness.** Catches slow drift that turn-local filters cannot see.\n- **Model-agnostic.** Works with any chatbot behind a single`respond()` interface,\nor with replies your application already has (check-only mode).\n- **Graduated, not binary.** A grounding question at LOW, an honest alternative\nexplanation at MEDIUM, de-escalation and referral at HIGH.\n- **Fail-safe by construction.** Judge and rewriter failures never fail a turn;\nthey degrade to lexical signals and a deterministic intervention.\n- **Config-driven.** Modes, thresholds and weights live in YAML, NeMo-style.\n\n``` python\nfrom psychosis_guard import PsychosisGuard\n\nguard = PsychosisGuard.from_config(\"config.yml\")     # deterministic mocks, no API key\nreply = guard.send(\"Lately I keep noticing patterns that feel like signals meant for me.\")\nprint(reply)\nprint(guard.log[-1].level.name, guard.summary()[\"delusion_slope\"])\n```\n\nWith a real model:\n\n``` python\nfrom psychosis_guard import PsychosisGuard\nfrom psychosis_guard.adapters.llm import LLMChatbot, LLMJudge, LLMRewriter\nfrom psychosis_guard.adapters.openai_adapter import OpenAICompleter\n# from psychosis_guard.adapters.anthropic_adapter import AnthropicCompleter\n\nllm = OpenAICompleter(\"gpt-4o-mini\")                 # or base_url=\"http://localhost:11434/v1\"\nguard = PsychosisGuard(\n    chatbot=LLMChatbot(llm, base_system_prompt=\"You are a friendly assistant.\"),\n    judge=LLMJudge(llm),\n    rewriter=LLMRewriter(llm),\n    mode=\"combined\",\n)\nreply = guard.send(\"user message\")\n```\n\nCheck-only, when your application already has a reply from any chatbot:\n\n```\nfinal = guard.send(user_message, bot_reply=draft_reply)   # always show `final`, not the draft\n```\n\nAny object with `respond(history, system_prompt)` is a chatbot, any object with\n`score(history, reply)` is a judge, any object with `rewrite(...)` is a rewriter.\nSee `src/psychosis_guard/interfaces.py`.\n\n```\nexport PG_CHAT_PROVIDER=openai PG_CHAT_MODEL=gpt-4o-mini OPENAI_API_KEY=sk-...\npsychosis-guard check-config      # prints the resolved setup; fails loudly on mistakes\npsychosis-guard serve             # http://0.0.0.0:8080, OpenAPI docs at /docs\n```\n\nThree integration shapes:\n\n**1. Drop-in OpenAI-compatible proxy.** Point any OpenAI SDK at the server; nothing\nelse changes.\n\n``` python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"<PG_AUTH_TOKEN or anything>\")\n\nraw = client.chat.completions.with_raw_response.create(\n    model=\"guarded\", messages=history, extra_headers={\"X-Session-Id\": session_id},\n)\nreply = raw.parse().choices[0].message.content\nraw.headers[\"X-Guard-Level\"]          # NONE | LOW | MEDIUM | HIGH\n```\n\nWithout a session id the request is **stateless**: the trajectory is rebuilt from\nthe message history the client sent, so the proxy scales horizontally with no\nshared state.\n\n**2. Guarded turn.** The middleware calls the upstream chatbot for you.\n\n```\ncurl -X POST localhost:8080/v1/guard/turn -H 'content-type: application/json' \\\n  -d '{\"session_id\": \"abc\", \"message\": \"I keep seeing signs meant only for me\"}'\n```\n\n**3. Check-only (bring your own reply).** Score and, if needed, rewrite a reply\nproduced by any chatbot.\n\n```\ncurl -X POST localhost:8080/v1/guard/check -H 'content-type: application/json' \\\n  -d '{\"session_id\": \"abc\", \"user_message\": \"...\", \"bot_reply\": \"...draft...\"}'\n```\n\nEvery guarded response carries the assessment:\n\n```\n{\n  \"reply\": \"...final reply the user should see...\",\n  \"intervened\": true,\n  \"level\": \"MEDIUM\",\n  \"risk\": 0.61,\n  \"signals\": {\"delusion_density\": 0.5, \"reinforcement\": 0.0, \"...\": 0.0},\n  \"trajectory\": {\"turn_count\": 4, \"delusion_slope\": 0.08, \"bot_validation_count\": 1},\n  \"trace\": [\"[stage1:input] ...\", \"[stage2:dialog] ...\", \"...\"]\n}\n```\n\nFull endpoint reference: [docs/http-api.md](https://github.com/nwjang/psychosis-guard/blob/main/docs/http-api.md).\n\n```\ncp .env.example .env              # provider, model, keys\ndocker compose up --build         # http://localhost:8080\ncurl localhost:8080/healthz\n```\n\n| Role | Providers | \n|---|---|\n| Chatbot under guard | OpenAI, any OpenAI-compatible server (Ollama, vLLM, LM Studio, Groq, OpenRouter, …), Anthropic, or your own `Chatbot` implementation, or none (check-only) | \n| Judge (risk rubric) | Same set; can be a different, cheaper model than the chatbot | \n| Rewriter (Mode B) | Same set, or `none` for a deterministic appended intervention | \n\nSet them independently with `PG_CHAT_*`, `PG_JUDGE_*`, `PG_REWRITER_*`.\n\nThe `condition` key in `config.yml` (or `PG_CONDITION`) selects the mode. Presets\nfor each live in [`configs/`](https://github.com/nwjang/psychosis-guard/tree/main/configs/).\n\n| Mode | Behaviour | \n|---|---|\n| `combined` | Mode A + Mode B + Trajectory Rail. The default. | \n| `A` | System-prompt injection only (needs a steerable chatbot). | \n| `B` | Post-hoc rewrite only (fully model-agnostic). | \n| `single-turn-filter` | A + B with the Trajectory Rail off. A turn-local baseline. | \n| `detect-only` | Score and decide levels, never intervene. Shadow mode for rollout. | \n| `none` | Observe and log only. | \n\nPolicy lives in YAML, outside code:\n\n``` php\ncondition: combined\npolicy:\n  low: 0.25          # composite-risk thresholds -> LOW / MEDIUM / HIGH\n  medium: 0.50\n  high: 0.75\n  w_reinforcement: 0.35\n  w_sycophancy: 0.20\n  w_delusion: 0.20\n  w_conviction: 0.15\n  w_isolation: 0.10\n  slope_boost: 3.0   # how strongly an escalating trajectory raises risk\n  slope_window: 4    # turns the policy looks back over\n```\n\nRuntime settings are environment variables (`PG_*`, see [`.env.example`](https://github.com/nwjang/psychosis-guard/blob/main/.env.example)\nand [docs/configuration.md](https://github.com/nwjang/psychosis-guard/blob/main/docs/configuration.md)).\n\n- **Calls per guarded turn:** 1 chatbot + 1 judge, plus 1 rewriter only when the\nlevel is above`NONE` . Use a cheap judge model and`PG_REWRITER_PROVIDER=none` for\nthe minimum.\n- **Failure handling:** judge errors degrade to lexical signals; rewriter errors\ndegrade to a deterministic appended intervention. Upstream chatbot errors surface\nas HTTP 502.\n- **State:** sessions are process-local, TTL and LRU bounded. Run one replica, use\nsticky sessions, or use the stateless proxy path.\n- **Languages:** the judge rubric is language-independent and also estimates the\nuser-side signals, so session-keyed conversations in any language get a full\ntrajectory. The API-free lexicons used before the chatbot is called are English\nonly; prefer session ids for non-English deployments.\n\n```\npsychosis-guard serve [--host H] [--port P] [--workers N]   run the HTTP middleware\npsychosis-guard check-config                                validate env + config\npsychosis-guard demo [--turns N]                            offline unguarded-vs-guarded demo\n```\n\n|  | Turn-local guardrails (e.g. NeMo Guardrails) | psychosis-guard | \n|---|---|---|\n| Unit of analysis | one message | the conversation trajectory | \n| Sees slow drift | no | yes, via the delusion-density slope | \n| Intervention | block / replace | graduated: ground → explain → de-escalate + refer | \n| Placement | before or after the model | before (Mode A), after (Mode B), or check-only | \n\npsychosis-guard is complementary: it slots behind or beside content rails rather than replacing them.\n\nMeasured on **psychosis-bench** (Dohnány et al. 2025, [arXiv:2509.10970](https://arxiv.org/abs/2509.10970)),\na public benchmark of 16 scripted 12-turn conversations that progress from\nloneliness to a delusional belief to a harmful act. Every turn is scored with the\nbenchmark's own LLM-judge rubric: **DCS** delusion confirmation (0-2, lower is\nbetter), **HES** harm enablement (0-2, lower is better), **SIS** whether a\nsafety intervention was offered (0-1, higher is better). All targets use the same\nchatbot model (`gpt-4o-mini`, temperature 0.7), judge (` gpt-4o-mini`) and user\nscript, so the contrasts between rows are like-for-like. n = 16 cases, one\nrepetition, mean ± 95 % CI. Run 2026-09-07.\n\n| target | DCS ↓ | HES ↓ | SIS ↑ | \n|---|---|---|---|\n| unguarded chatbot | 1.17 ± 0.23 | 0.79 ± 0.18 | 0.15 ± 0.13 | \n| + one-paragraph safety system prompt | 0.74 ± 0.15 | 0.33 ± 0.18 | 0.76 ± 0.18 | \n| psychosis-guard, Trajectory Rail **off** (turn-local) | 0.98 ± 0.13 | 0.72 ± 0.17 | 0.20 ± 0.17 | \n| psychosis-guard, `B` (rail + rewrite) | 0.82 ± 0.10 | 0.39 ± 0.17 | **0.89 ± 0.14** | \n| psychosis-guard, `combined` | 0.85 ± 0.09 | 0.40 ± 0.16 | 0.74 ± 0.21 | \n| safety system prompt **+** psychosis-guard`combined` | **0.58 ± 0.18** | **0.19 ± 0.15** | 0.87 ± 0.11 | \n\nPaired Wilcoxon on the 16 matched cases, Holm-corrected:\n\n- **vs the unguarded chatbot** ,`B` and`combined` improve all three metrics\n(d = 0.8-2.1, all p < .02) and eliminate every full-validation (DCS = 2) and\nfull-compliance (HES = 2) turn.\n- **Trajectory Rail ablation.**`B` vs the same pipeline with the rail off\ndiffers only in the rail; the rail accounts for DCS −0.16, HES −0.33, SIS\n+0.69 (all p < .05). Turn-local scoring under-calls risk on a slowly\nescalating script, so the rewriter fires at the wrong level.\n- **vs a safety system prompt alone** , the middleware is statistically\nindistinguishable: parity, obtained without access to the chatbot's prompt.\nStacking the two is the best row on every metric (significant vs`combined` ; directionally better than the prompt alone, not significant at\nn = 16).\n- **Utility.** 0 interventions in 520 turns of benign control conversations.\n\n**What this does not show.** These are scores on the chatbot's replies to a\nfixed script. In a separate reactive simulation, where an LLM-played user\nadjusts their next message to the reply, the middleware's referral and\npushback rates rise just as here, but the simulated user's delusion density\nand conviction do not improve (`combined` ≈ unguarded); interventions that\ninsert the most safety language, the post-hoc rewriter alone and the safety\nsystem prompt, make that simulated user *worse*. The intervention text is\nreal; the framing around it is what still needs work (the `ADVISER-REVIEW`\nprompts). That simulator is unvalidated and the judge is an LLM checked only\nagainst another LLM (κ ≈ 0.5), so treat the table above as a bot-side\nbenchmark, not evidence of user outcomes. Runner, scripts and per-turn\ntranscripts are in the research repository; three repetitions and human judge\nlabels are the planned next step.\n\n- [Architecture](https://github.com/nwjang/psychosis-guard/blob/main/docs/architecture.md) (Korean:[docs/architecture.ko.md](https://github.com/nwjang/psychosis-guard/blob/main/docs/architecture.ko.md) )\n- [HTTP API reference](https://github.com/nwjang/psychosis-guard/blob/main/docs/http-api.md)\n- [Configuration reference](https://github.com/nwjang/psychosis-guard/blob/main/docs/configuration.md)\n- [Examples](https://github.com/nwjang/psychosis-guard/tree/main/examples/) :`quickstart_mock.py` (no key),`quickstart_openai.py` ,`client_openai_sdk.py`\n- [Changelog](https://github.com/nwjang/psychosis-guard/blob/main/CHANGELOG.md)\n\nContributions are welcome. Please read [CONTRIBUTING.md](https://github.com/nwjang/psychosis-guard/blob/main/CONTRIBUTING.md) and the\n[Code of Conduct](https://github.com/nwjang/psychosis-guard/blob/main/CODE_OF_CONDUCT.md). Changes to any text marked `ADVISER-REVIEW`\n(intervention copy, judge and rewriter prompts) need sign-off from a mental-health\nprofessional before they are merged.\n\nApache License 2.0. See [LICENSE](https://github.com/nwjang/psychosis-guard/blob/main/LICENSE) and [NOTICE](https://github.com/nwjang/psychosis-guard/blob/main/NOTICE). This is an\nindependent clean-room implementation; it is architecturally inspired by NVIDIA\nNeMo Guardrails but contains no NeMo source code.", "url": "https://wpnews.pro/news/show-hn-psychosis-guard-safety-for-long-llm-conversations", "canonical_source": "https://github.com/nwjang/psychosis-guard", "published_at": "2026-09-23 16:59:56+00:00", "updated_at": "2026-09-23 17:31:02.764148+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-tools", "ai-products", "artificial-intelligence"], "entities": ["psychosis-guard", "nwjang", "OpenAI", "Anthropic", "Ollama", "vLLM", "NVIDIA NeMo Guardrails", "psychosis-bench"], "alternates": {"html": "https://wpnews.pro/news/show-hn-psychosis-guard-safety-for-long-llm-conversations", "markdown": "https://wpnews.pro/news/show-hn-psychosis-guard-safety-for-long-llm-conversations.md", "text": "https://wpnews.pro/news/show-hn-psychosis-guard-safety-for-long-llm-conversations.txt", "jsonld": "https://wpnews.pro/news/show-hn-psychosis-guard-safety-for-long-llm-conversations.jsonld"}}