If you run LLMs in production—whether OpenAI, Anthropic, Gemini, or self-hosted Ollama/vLLM—you have probably been burned by silent model drift. An upstream provider tweaks quantization or safety filters overnight, refusal rates jump, or an update subtly breaks edge-case prompts without any announcement.
Most observability platforms solve this in two frustrating ways:
Over the past few months (I’m 15, building on a basic setup with no GPU cluster or funding), I built Wallras—a single-container, self-hosted transparent proxy (base_url swap, ~3–5ms overhead) that catches silent drift, model swaps, and confident hallucinations while keeping 100% of your data on your own servers.
Here is a look under the hood at four engineering problems I had to solve while building it.
A static rule like "alert if average response length shifts by 15%" breaks immediately on real traffic because some prompts naturally produce longer answers than others.
Instead, Wallras combines three statistical signals:
|z| > 3) for multilingual refusal-rate spikes
To avoid crying wolf, it runs 150-iteration permutation shuffling on baseline vs. recent outputs to measure that specific model's own natural random variance. It only raises a drift flag when a shift exceeds both a practical floor and the model's 99th-percentile random noise ceiling (~1.7% false-alarm rate on noisy baselines).
You can read the exact formulas and thresholds on the measurement math page.
LLMs frequently sound 95% confident in plain English even when they are guessing a date, API parameter, or legal citation.
When token logprobs are available (from OpenAI, OpenRouter, Ollama, or vLLM), Wallras computes Shannon entropy across the top alternative tokens alongside the geometric mean and minimum token probability:
internal_certainty = 0.5 * geometric_mean_prob + 0.5 * min_token_prob
Blending the average token probability with the single weakest token ensures that one guessed fact buried inside an otherwise fluent paragraph still triggers a mismatch flag when the model's text sounds overly certain.
One of the trickiest security bugs I found during testing was with SQLite deletion requests.
When a user triggers a GDPR deletion—or turns on field-level AES-256-GCM encryption for existing records—running a standard SQL DELETE or UPDATE removes the row logically, but leaves the plaintext sitting inside SQLite's write-ahead log (.wal) and unallocated database file pages until reused.
To fix this properly:
.wal file after deletion to verify zero plaintext remains.
All 366 automated/attack tests, Semgrep/Bandit/Trivy scans, and live A+ Mozilla Observatory / SSL Labs checks are documented on the Wallras security page.
Developers increasingly debug inside Claude Code or Cursor rather than clicking through separate dashboards.
Wallras ships a zero-dependency standard-library Python MCP server (wallras-mcp) and an llms.txt spec:
wallras_drift, wallras_traces, wallras_blind_spots, wallras_prompt_diff) so your coding agent can diagnose production issues safely.--enable-writes) allow Cursor or Claude to promote a prompt update to production Payments aren't even turned on yet while I validate proof-of-concept—every plan up to Scale is 100% free to try with zero telemetry or prompt data leaving your machine.
Since I don't have a multi-GPU rig or heavy API budget to stress-test high concurrency, I would love feedback from anyone running local models or production LLM apps:
Feel free to roast the math, proxy overhead, or SQLite/Postgres architecture in the comments!