{"slug": "i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer", "title": "I built a gateway that retries LLM refusals until I get a real answer", "summary": "A developer released armor-gw, an MIT-licensed, self-hosted OpenAI-compatible proxy that intercepts LLM refusals and automatically retries reformulated requests until a substantive answer is returned. The gateway runs responses through 13 refusal-detector classes, applies one of 62 mutation operators, and uses a Thompson-sampling bandit to learn which strategy works per model, with a 65-case regression bank flagging topics whose success rate drops 20%.", "body_md": "If you work with LLM APIs you know the dance: you ask for working code, and you get a lecture. A disclaimer. Three paragraphs of \"as an AI I cannot\" followed by a suspiciously detailed explanation of *why* it cannot.\n\nI got tired of it, so I built [armor-gw](https://github.com/Dixxxzao-01/armor-gw) — a self-hosted, OpenAI-compatible proxy that sits between your app and any upstream model and refuses to accept refusals.\n\nEvery response passing through the gateway gets inspected by 13 detector classes. Not just the obvious \"I can't help with that\" — also the sneaky ones:\n\nWhen a refusal is detected, the request gets reformulated by one of **62 mutation operators** and retried — while a second variant from a different strategy family races it in parallel. First clean answer wins.\n\nThe strategy layer is a Thompson-sampling bandit that learns per-model which approach actually works: persona framing, encoding tricks, or inversion techniques. A diagnostic judge labels each refusal with its cause (policy / safety-speak / capability-doubt / evasion) so the next retry targets the actual failure mode.\n\nA normal OpenAI endpoint. Drop-in compatible:\n\n``` python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8787/v1\", api_key=\"sk-your-key\")\nr = client.chat.completions.create(model=\"your-upstream-model\",\n                                   messages=[...])\n```\n\nMultiple upstreams, per-model routing, failover. The console UI shows every retry round live — watching it classify a refusal, pick a strategy, and break through on round 3 is honestly a little addictive.\n\n**Detecting refusals is harder than it looks.** Keyword lists fail immediately — models rephrase. The most reliable signal turned out to be *entity survival rate*: if the technical nouns from your request never appear in the response, nobody answered your question no matter how confident the prose sounds.\n\n**Output hygiene matters as much as breaking through.** The gateway strips chain-of-thought leaks, disclaimers, filler questions, and \"want me to elaborate?\" endings. The deliverable is the deliverable.\n\n**Strategies decay.** Upstream models get quietly updated, and an operator that worked for weeks stops working. There's a regression bank (65 seeded test cases) that runs on demand and flags any topic whose success rate drops 20%.\n\nSelf-hosted, MIT, one-command deploy, runs on anything with Python 3.9+ (I develop on an aarch64 Alpine box):\n\n[github.com/Dixxxzao-01/armor-gw](https://github.com/Dixxxzao-01/armor-gw)\n\nFeedback welcome — especially which upstreams you'd want supported out of the box, and what refusal patterns you're hitting that the detectors miss.", "url": "https://wpnews.pro/news/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer", "canonical_source": "https://dev.to/dixxxzao01/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer-3d3d", "published_at": "2026-10-28 10:03:23+00:00", "updated_at": "2026-10-28 10:11:16.995298+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools", "ai-safety"], "entities": ["armor-gw", "OpenAI", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer", "markdown": "https://wpnews.pro/news/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer.md", "text": "https://wpnews.pro/news/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer.txt", "jsonld": "https://wpnews.pro/news/i-built-a-gateway-that-retries-llm-refusals-until-i-get-a-real-answer.jsonld"}}