{"slug": "how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts", "title": "How I built a 3ms self-hosted AI drift proxy in Python (and wiped deleted prompts from SQLite WAL files)", "summary": "A 15-year-old developer built Wallras, a single-container self-hosted transparent LLM proxy that adds roughly 3–5ms of overhead and detects silent model drift, model swaps and confident hallucinations while keeping prompt data on the user's own servers. The proxy combines three statistical signals with 150-iteration permutation shuffling to hold false alarms to about 1.7% on noisy baselines, scores internal certainty from token logprobs, and scrubs plaintext from SQLite WAL files and unallocated pages after GDPR deletions. It also ships a zero-dependency standard-library Python MCP server (wallras-mcp) exposing drift, trace, blind-spot and prompt-diff tools to coding agents such as Claude Code and Cursor.", "body_md": "If you run LLMs in production—whether OpenAI, Anthropic, Gemini, or self-hosted Ollama/vLLM—you have probably been burned by **silent model drift**. An upstream provider tweaks quantization or safety filters overnight, refusal rates jump, or an update subtly breaks edge-case prompts without any announcement.\n\nMost observability platforms solve this in two frustrating ways:\n\nOver the past few months (I’m 15, building on a basic setup with no GPU cluster or funding), I built **[Wallras](https://wallras.com)**—a single-container, self-hosted transparent proxy (`base_url` swap, ~3–5ms overhead) that catches silent drift, model swaps, and confident hallucinations while keeping 100% of your data on your own servers.\n\nHere is a look under the hood at four engineering problems I had to solve while building it.\n\nA static rule like *\"alert if average response length shifts by 15%\"* breaks immediately on real traffic because some prompts naturally produce longer answers than others.\n\nInstead, Wallras combines three statistical signals:\n\n`|z| > 3`) for multilingual refusal-rate spikes\nTo avoid crying wolf, it runs **150-iteration permutation shuffling** on baseline vs. recent outputs to measure that specific model's own natural random variance. It only raises a drift flag when a shift exceeds both a practical floor *and* the model's 99th-percentile random noise ceiling (~1.7% false-alarm rate on noisy baselines).\n\nYou can read the exact formulas and thresholds on the [measurement math page](https://wallras.com).\n\nLLMs frequently sound 95% confident in plain English even when they are guessing a date, API parameter, or legal citation.\n\nWhen token logprobs are available (from OpenAI, OpenRouter, Ollama, or vLLM), Wallras computes Shannon entropy across the top alternative tokens alongside the geometric mean and minimum token probability:\n\n```\ninternal_certainty = 0.5 * geometric_mean_prob + 0.5 * min_token_prob\n```\n\nBlending the average token probability with the **single weakest token** ensures that one guessed fact buried inside an otherwise fluent paragraph still triggers a mismatch flag when the model's text sounds overly certain.\n\nOne of the trickiest security bugs I found during testing was with **SQLite deletion requests**.\n\nWhen a user triggers a GDPR deletion—or turns on field-level **AES-256-GCM encryption** for existing records—running a standard SQL `DELETE` or `UPDATE` removes the row logically, **but leaves the plaintext sitting inside SQLite's write-ahead log (`.wal`) and unallocated database file pages** until reused.\n\nTo fix this properly:\n\n`.wal` file after deletion to verify zero plaintext remains.\nAll 366 automated/attack tests, Semgrep/Bandit/Trivy scans, and live A+ Mozilla Observatory / SSL Labs checks are documented on the [Wallras security page](https://wallras.com).\n\nDevelopers increasingly debug inside **Claude Code** or **Cursor** rather than clicking through separate dashboards.\n\nWallras ships a zero-dependency standard-library Python MCP server (`wallras-mcp`) and an [` llms.txt`](https://wallras.com) spec:\n\n`wallras_drift`, `wallras_traces`, `wallras_blind_spots`, `wallras_prompt_diff`) so your coding agent can diagnose production issues safely.`--enable-writes`) allow Cursor or Claude to promote a prompt update to `production` Payments aren't even turned on yet while I validate proof-of-concept—every plan up to Scale is **100% free to try** with zero telemetry or prompt data leaving your machine.\n\nSince I don't have a multi-GPU rig or heavy API budget to stress-test high concurrency, I would love feedback from anyone running local models or production LLM apps:\n\nFeel free to roast the math, proxy overhead, or SQLite/Postgres architecture in the comments!", "url": "https://wpnews.pro/news/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts", "canonical_source": "https://dev.to/harshith_kala/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts-from-sqlite-wal-3aoh", "published_at": "2026-11-14 18:08:19+00:00", "updated_at": "2026-11-14 18:20:52.512031+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "ai-agents", "agent-protocols", "ai-safety"], "entities": ["Wallras", "SQLite", "OpenAI", "Anthropic", "Gemini", "Ollama", "vLLM", "Claude Code"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts", "markdown": "https://wpnews.pro/news/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts.md", "text": "https://wpnews.pro/news/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-3ms-self-hosted-ai-drift-proxy-in-python-and-wiped-deleted-prompts.jsonld"}}