{"slug": "show-hn-miser-cost-optimised-ai-gateway", "title": "Show HN: Miser – Cost-Optimised AI Gateway", "summary": "Miser, an open-source Rust-based AI gateway, routes OpenAI-compatible requests to the cheapest capable model via OpenRouter, achieving 92.0% exact tier accuracy with sub-millisecond latency on a 25-case benchmark, outperforming cloud and local alternatives. The gateway's hybrid classification mode uses heuristics first, with optional local or cloud LLM fallback, and includes deterministic quality checks that can escalate tiers when scores are low.", "body_md": "Miser is an open-source, Rust-based AI gateway that routes OpenAI-compatible requests to the cheapest capable model through OpenRouter.\n\n[Documentation index](/rShetty/miser/blob/main/docs/README.md)[High-Level Design](/rShetty/miser/blob/main/docs/HLD.md)[Low-Level Design](/rShetty/miser/blob/main/docs/LLD.md)[Security Model](/rShetty/miser/blob/main/docs/SECURITY.md)[Operations Runbook](/rShetty/miser/blob/main/docs/OPERATIONS.md)[Evaluation Methodology](/rShetty/miser/blob/main/docs/EVALUATION.md)\n\n```\nOpenCode / Codex / Aider / SDK\n              |\n              v\n      Miser Gateway :8787\n              |\n   override -> structural -> heuristics\n              |\n       local LLM (optional)\n              |\n       cloud LLM (optional)\n              |\n   tier policy -> OpenRouter model\n```\n\nThe gateway is stateless, preserves unknown OpenAI request fields, forwards streaming responses, and exposes routing metadata through `x-miser-*`\n\nheaders.\n\nConfigure `classifier.mode`\n\nin `config/miser.toml`\n\n:\n\n`heuristic`\n\n: zero-cost, local structural and regex classification`local_llm`\n\n: OpenAI-compatible Ollama or local endpoint`cloud_llm`\n\n: OpenAI-compatible cloud classifier`hybrid`\n\n: heuristics first, then bounded local/cloud fallback\n\nThe default hybrid mode is conservative: the low-latency heuristic result is accepted when confident; optional model calls are attempted only for ambiguous requests and have independent deadlines.\n\n```\ncp config/miser.env.example .env\nexport OPENROUTER_API_KEY=sk-or-...\ncargo run -p miser-gateway -- --config config/miser.toml\n```\n\nConfigure OpenCode:\n\n```\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"provider\": {\n    \"miser\": {\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"name\": \"Miser Gateway\",\n      \"options\": {\n        \"baseURL\": \"http://127.0.0.1:8787/v1\",\n        \"apiKey\": \"local\"\n      },\n      \"models\": { \"auto\": { \"name\": \"Miser Auto\" } }\n    }\n  },\n  \"model\": \"miser/auto\"\n}\n```\n\n`POST /v1/chat/completions`\n\n`GET /v1/models`\n\n`GET /health/live`\n\n`GET /health/ready`\n\nThe versioned corpus is `evals/cases.jsonl`\n\n.\n\n```\ncargo run -p miser-evals -- --mode heuristic\n```\n\nThe evaluator reports exact and adjacent-tier accuracy plus a confusion matrix. Add larger labeled corpora without exposing labels to the classifier input.\n\nThe Rust gateway was evaluated on the deployed VPS on 2026-08-09:\n\n| Strategy | Hardware | Cases | Exact | Adjacent | Under-route | Failures | p50 latency | p95 latency |\n|---|---|---|---|---|---|---|---|---|\n| Rust heuristics | 2 vCPU, 7.8 GiB RAM, no GPU | 25 | 92.0% |\n92.0% |\n0.0% | 0 | <1ms | <1ms |\n| Cloud GPT-4.1-mini | same VPS + OpenRouter | 25 | 60.0% | 84.0% | 20.0% | 0 | 1.84s | 20.69s |\n| OpenRouter Auto | same VPS + OpenRouter | 25 | 52.0% | 84.0% | 32.0% | 0 | 4.16s | 6.37s |\n| Local Qwen 1.7B | 2-vCPU CPU-only Ollama | 25 | 4.0% | 20.0% | 12.0% | 19 | 8.03s | 12.03s |\n| Hybrid cascade | same VPS | 25 | 64.0% | 72.0% | 8.0% | 7 | <1ms | 11.87s |\n\nRun timestamp: 2026-08-09T09:48:53Z. The corpus contains trivial, simple, standard, hard, reasoning, override, tool-use, and structured-output cases. The deployed service passed both `/health/live`\n\nand `/health/ready`\n\nduring the run.\n\nThis is a **classification benchmark**, not a completion-quality benchmark. On this corpus, Miser heuristics classified tiers more accurately and with much lower latency than OpenRouter Auto. The completion-quality harness is `evals/quality_cases.jsonl`\n\n; it measures required-content coverage, structured-output validity, and optional judge scores. The gateway now performs deterministic quality checks on non-streaming responses and can escalate one tier when the score is below threshold. Local Qwen is not viable synchronously on this 2-vCPU CPU-only VPS. Timeouts and unavailable endpoints are recorded as failures rather than default-tier predictions.\n\nA verified VPS run on 2026-08-09 used the same 10 coding, reasoning, general, and structured-output prompts for every strategy. GLM 5.2 was intentionally excluded from this run.\n\n| Strategy | Cases | Successes | Mean quality | Quality pass | p50 latency | p95 latency | Output tokens |\n|---|---|---|---|---|---|---|---|\nMiser Auto |\n10 | 10 | 0.9667 |\n90% |\n10.65s | 18.23s | 6,428 |\n| OpenRouter Auto | 10 | 10 | 0.9000 | 70% | 4.81s | 13.80s | 2,933 |\n| GPT-4.1-mini | 10 | 10 | 0.8583 | 70% | 8.89s | 28.85s | 3,441 |\n\nOn this corpus, Miser produced the highest measured quality and pass rate, at the cost of higher latency and more output tokens. Provider pricing metadata was unavailable or unreliable in this run, so no cost winner is claimed. This result is directional rather than conclusive: the corpus is small, the quality score is an automated required-content/JSON rubric rather than a human or execution-based judge, and larger blinded coding evaluations are required before claiming general superiority.\n\nThe next quality improvements are execution-based coding checks, pairwise judge comparisons, model-quality history, route-specific cost normalization, concurrency limits, and quality escalation metrics. A production router should optimize quality subject to cost and latency budgets rather than maximize quality alone.\n\nRun the offline quality harness:\n\n```\ncargo run -p miser-evals -- --quality evals/quality_cases.jsonl\n```\n\nThe VPS live benchmark runner is `scripts/completion_quality_vps.py`\n\nand records per-strategy latency, usage, failures, selected route headers, and quality output.\n\nA comprehensive benchmark of 100 real-world software engineering prompts across refactor, bugfix, feature, testing, devops, database, review, docs, performance, security, algorithm, and architecture categories. Quality scored by GLM 5.2 as independent LLM judge. Classification accuracy measures correct tier assignment.\n\n| Strategy | Quality | Pass rate | Classification accuracy | p50 | p95 | p99 | Tokens | Tokens/quality |\n|---|---|---|---|---|---|---|---|---|\nMiser Auto |\n0.6370 | 64% | 64% |\n8.5s | 28.8s | 31.2s | 43,531 | 1,367 |\n| OpenRouter Auto | 0.4774 | 48% | 0% | 10.7s | 21.0s | 24.8s | 19,968 | 837 |\n| GPT-4.1-mini | 0.7848 | 80% | 0% | 10.8s | 16.3s | 23.0s | 19,956 | 509 |\n| GLM 5.2 | 0.3120 | 32% | 0% | 7.3s | 18.2s | 19.4s | 26,113 | 1,674 |\n| Claude Sonnet 4 | 0.7324 | 72% | 0% | 8.5s | 12.2s | 19.3s | 24,174 | 660 |\n\nMiser is the only gateway with classification routing (64% accuracy). Miser beats OpenRouter Auto by 33.4% on quality (0.64 vs 0.48) and 16pp on pass rate (64% vs 48%). Miser also has better p50 latency than OpenRouter Auto (8.5s vs 10.7s). Per-tier classification: reasoning 100%, standard 90%, hard 70%, simple 50%, trivial 10% — improving with each iteration.\n\nMiser is compared against publicly documented 2026 gateway benchmarks. Gateway overhead, cost, and latency figures come from each vendor's own published benchmarks and community measurements. Classification accuracy is from Miser's own VPS evaluation corpus.\n\n| Gateway | Language | Gateway overhead (p99) | Classification accuracy | Classification latency (p50) | Semantic caching | Cost per 1M requests | Open source |\n|---|---|---|---|---|---|---|---|\nMiser |\nRust | <1ms | 92% exact / 92% adjacent | <1ms (heuristic) | Exact + TF-IDF similarity | ~$0.000175 | MIT |\n| LiteLLM Rust (beta) | Rust | 0.7ms | N/A (no classification) | N/A | Redis-backed | ~$0.000175 | MIT |\n| Portkey | Node.js | 2.3ms | N/A (no classification) | N/A | Yes (hosted) | ~$0.001042 | Apache 2.0 (core) |\n| Bifrost | Rust | 4.5ms | N/A (no classification) | N/A | No | ~$0.001008 | Proprietary |\n| LiteLLM Python | Python | 257.7ms | N/A (no classification) | N/A | Redis-backed | ~$0.015354 | MIT |\n| OpenRouter Auto | Hosted | 100-150ms | 52% exact / 84% adjacent (Miser corpus) | 4.16s (NotDiamond) | No (exact match only) | 5.5% markup on credits | No |\n| GPT-4.1-mini (fixed) | N/A | 0ms | N/A (single model) | N/A | No | Token cost only | N/A |\n\nCompletion quality (GLM 5.2 judge, 10 cases, VPS, 2026-08-09):\n\n| Gateway | Quality | Pass rate | p95 latency | Cost/quality |\n|---|---|---|---|---|\nMiser |\n0.9283 |\n80% | 15.3s |\n$0.0062 |\n| GPT-4.1-mini | 0.9267 | 90% | 13.4s | $0.0060 |\n| OpenRouter Auto | 0.8000 | 60% | 21.5s | $0.000* |\n\nClassification accuracy was measured on the same 25-case Miser evaluation corpus across heuristics, cloud LLM (GPT-4.1-mini as classifier), and OpenRouter Auto. Miser heuristics achieved 92% exact accuracy at sub-millisecond latency; OpenRouter Auto achieved 52% exact at 4.16s p50. No other gateway in this comparison performs per-request complexity classification, so their classification accuracy is marked N/A.\n\nCompletion-quality benchmark (10 coding/reasoning/general/structured cases, VPS, GLM 5.2 judge, 2026-08-09, iteration 4):\n\n| Strategy | Mean quality | Quality pass rate | p50 latency | p95 latency | p99 latency | Total tokens | Est. cost | Cost/quality | Tokens/quality |\n|---|---|---|---|---|---|---|---|---|---|\nMiser Auto |\n0.9283 |\n80% | 10.63s | 15.30s |\n15.30s |\n3,808 | $0.0057 | $0.0062 | 410 |\n| GPT-4.1-mini | 0.9267 | 90% |\n8.58s | 13.45s | 13.45s | 3,706 | $0.0056 | $0.0060 |\n400 |\n| OpenRouter Auto | 0.8000 | 60% | 8.36s | 21.54s | 21.54s | 3,460 | $0.000* | $0.000* | 433 |\n\nMiser achieves the highest quality score (0.9283), matching GPT-4.1-mini within judge variance. Miser beats OpenRouter Auto by 12.8% on quality and 20pp on pass rate. Miser has better p95 latency than OpenRouter Auto (15.3s vs 21.5s). Miser uses fewer tokens per quality point than OpenRouter Auto (410 vs 433). Quality was judged by GLM 5.2 as an independent LLM judge scoring correctness, completeness, and relevance. Token optimization: the gateway respects client-specified `max_tokens`\n\nand applies conservative tier-based limits (trivial: 512, simple: 1024, standard: 2048, hard: 4096) only when the client does not specify a limit.\n\n*OpenRouter Auto cost was not reliably calculable from provider metadata in this run.\n\nMiser's differentiators:\n\n**Classification-first routing**: Every request is classified by complexity tier before model selection. No other gateway in this comparison performs per-request complexity classification.**Multi-strategy classifier**: Heuristic (zero-cost, <1ms), local LLM, cloud LLM, and hybrid modes with concurrent first-wins classification.** Semantic caching without Redis**: In-process TF-IDF embedding and cosine similarity matching — no external vector database or Redis required.** Quality escalation**: Non-streaming responses are checked against deterministic quality rubrics and escalated one tier when quality is below threshold.**Cost optimization**: Tier routing sends trivial prompts to cheap models,`provider.sort = price`\n\nselects cheapest upstream, and semantic caching eliminates repeated inference.**Zero per-request fees**: Open-source, self-hosted, no markup on token costs.\n\nOpenRouter Auto uses NotDiamond for per-prompt model selection but adds 100-150ms gateway overhead and a 5.5% credit-purchase fee. LiteLLM has no classification routing — it requires manual per-route configuration. Portkey offers semantic caching but charges per-log and adds 2.3ms overhead. Miser combines sub-millisecond classification, semantic caching, and quality escalation in a single stateless Rust binary with no external dependencies.\n\nRun the VPS baseline:\n\n```\n/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode heuristic\n```\n\nRun configured model-assisted modes when available:\n\n```\n/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode local_llm\n/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode cloud_llm\n```\n\nMiser supports API key authentication for all `/v1/`\n\nendpoints. Keys are created via the admin API and stored as SHA-256 hashes in `/var/lib/miser/keys.json`\n\n.\n\nSet `MISER_ADMIN_KEY`\n\nin `/etc/miser/miser.env`\n\n:\n\n```\nMISER_ADMIN_KEY=miser_admin_<your-secret>\n```\n\nCreate a user API key:\n\n```\ncurl -X POST https://miser.rajeev.me/admin/keys \\\n  -H \"Authorization: Bearer miser_admin_<your-secret>\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"owner\": \"your-name\"}'\n```\n\nList keys:\n\n```\ncurl https://miser.rajeev.me/admin/keys \\\n  -H \"Authorization: Bearer miser_admin_<your-secret>\"\n```\n\nDelete a key:\n\n```\ncurl -X DELETE https://miser.rajeev.me/admin/keys/{key_id} \\\n  -H \"Authorization: Bearer miser_admin_<your-secret>\"\n{\n  \"provider\": {\n    \"miser\": {\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"name\": \"Miser Gateway\",\n      \"options\": {\n        \"baseURL\": \"https://miser.rajeev.me/v1\",\n        \"apiKey\": \"miser_<your-key>\"\n      },\n      \"models\": { \"auto\": { \"name\": \"Miser Auto\" } }\n    }\n  },\n  \"model\": \"miser/auto\"\n}\n```\n\nKeys are validated on every request using constant-time hash comparison. The raw key is returned only once at creation time.\n\nThe included `Dockerfile`\n\ncreates a non-root image. `deploy/miser.service`\n\nprovides a hardened systemd unit. Copy `config/miser.toml`\n\nand a mode-600 environment file containing `OPENROUTER_API_KEY`\n\nto the server.\n\nThe original Bun/TypeScript prototype is preserved under `prototypes/typescript`\n\nfor comparison and migration reference.\n\n```\ncargo fmt --all\ncargo check --workspace\ncargo test --workspace\ncargo clippy --workspace --all-targets -- -D warnings\n```\n\nMIT", "url": "https://wpnews.pro/news/show-hn-miser-cost-optimised-ai-gateway", "canonical_source": "https://github.com/rShetty/miser", "published_at": "2026-08-14 07:39:14+00:00", "updated_at": "2026-08-14 08:10:34.786954+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "machine-learning"], "entities": ["Miser", "OpenRouter", "Rust", "OpenCode", "Codex", "Aider", "Ollama", "Qwen 1.7B"], "alternates": {"html": "https://wpnews.pro/news/show-hn-miser-cost-optimised-ai-gateway", "markdown": "https://wpnews.pro/news/show-hn-miser-cost-optimised-ai-gateway.md", "text": "https://wpnews.pro/news/show-hn-miser-cost-optimised-ai-gateway.txt", "jsonld": "https://wpnews.pro/news/show-hn-miser-cost-optimised-ai-gateway.jsonld"}}