{"slug": "show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t", "title": "Show HN: Pith – a proxy that cuts LLM output tokens, and refuses when it can't", "summary": "A developer released Pith, an open-source proxy (installable via `pip install pith-optimizer`) that cuts LLM output-token spend by replaying sampled traffic under cheaper output profiles and pinning a profile only when a judge model finds it as good as the baseline is with itself. Across ten live runs on the Anthropic API with 200 prompts each and judge `claude-sonnet-5-5`, Pith pinned only two routes: repeated support tickets on Haiku 4.5 at −51% tokens for a $1.30 sweep, and GSM8K on Sonnet 5.5 at 25k requests/month for −37% tokens and −26% cost, confirmed on a 30% holdout. Pith refused the other eight routes, including SAMSum summarisation at −53% tokens (0.64 vs floor 0.84) because it dropped information, and ticket classification at −44% tokens because the sweep was not repaid at 13k requests/month.", "body_md": "Cut LLM output-token spend without losing information. pith watches a route, replays a sample under cheaper output profiles, has a judge compare every answer with the baseline, and pins a profile only when it is as good as the baseline is with itself. When it is not, pith says so and changes nothing.\n\n```\npip install pith-optimizer && pith demo\n```\n\nThe demo runs the whole loop in-process against a mock model and a mock judge, with no API key and no data leaving the machine. Its numbers are synthetic. The measured ones are below.\n\nTen live runs on the Anthropic API, 200 prompts each unless noted, three trials per prompt, judge `claude-sonnet-5-5`.\n\"Floor\" is how often the unconstrained model agrees with itself, which is the bar a candidate has to meet.\n\n| Route | Model | Best candidate | Outcome | Sweep cost | \n|---|---|---|---|---|\n| Support tickets, 10 repeated, 30 requests | Haiku 4.5 | −51% tokens at the floor | **pinned** | $1.30 | \n| Support tickets, 200 distinct | Haiku 4.5 | −53% tokens, 0.78 vs floor 0.83 | refused, below floor | $5.99 | \n| Ticket classification | Haiku 4.5 | −44% tokens at the floor | refused, sweep not repaid at 13k requests/month | $2.49 | \n| JSON extraction | Haiku 4.5 | −5% tokens | refused, nothing to remove | $2.44 | \n| SAMSum summarisation | Haiku 4.5 | −53% tokens, 0.64 vs floor 0.84 | refused, drops information | $5.61 | \n| Dolly summarisation | Haiku 4.5 | −55% tokens, 0.45 vs floor 0.77 | refused, drops information | $6.20 | \n| Bitext support replies | Haiku 4.5 | −54% tokens, 0.51 vs floor 0.72 | refused, drops information | $6.31 | \n| GSM8K | Haiku 5.5 | −35% tokens at the floor | refused, sweep not repaid at 10k requests/month | $4.47 | \n| GSM8K | Sonnet 5.5 | −40% tokens at the floor | refused, sweep not repaid at 10k requests/month | $7.13 | \n| GSM8K, 25k requests/month | Sonnet 5.5 | −37% tokens, −26% cost | **pinned, confirmed on a 30% holdout** | $8.38 | \n\nTwo lessons, stated in full in [docs/results.md](https://github.com/dex0shubham/pith/blob/main/docs/results.md).\nA concision instruction saves tokens by dropping content, and on summaries and support replies the judge catches\nthat every time. An effort step-down on a thinking model saves thinking tokens the task did not need, and then the\nonly question is whether the route's volume repays the sweep. Decision rules and their reasons:\n[docs/methodology.md](https://github.com/dex0shubham/pith/blob/main/docs/methodology.md).\n\n```\npip install pith-optimizer\npith serve                                    # defaults; or --config pith.toml\n```\n\n`pith.toml` starts from [pith.example.toml](https://github.com/dex0shubham/pith/blob/main/pith.example.toml).\n\nPoint your client at it and keep your own API key:\n\n```\nANTHROPIC_BASE_URL=http://localhost:8787      # Anthropic SDKs\nOPENAI_BASE_URL=http://localhost:8787/v1      # OpenAI SDKs\n```\n\nSweep from the CLI, which is the only place a provider key lives:\n\n```\nexport ANTHROPIC_API_KEY=...                  # and/or OPENAI_API_KEY\npith sweep --config pith.toml --dry-run       # estimate, print, write nothing\npith sweep --config pith.toml --holdout 0.3   # pin the cheapest profile that clears the bar\npith recheck --config pith.toml               # re-judge live traffic, revert on drift\n```\n\nReports: `GET /optimizer/report.html`. Every scalar setting is also an `OPTIMIZER_<FIELD>` environment variable.\nFull flag reference: [docs/sweeps.md](https://github.com/dex0shubham/pith/blob/main/docs/sweeps.md).\n\n1. **Fingerprint.** Requests are grouped into routes by provider, model and system prompt.\n2. **Record.** Usage and a sample of bodies go into one SQLite file. Nothing is rewritten yet.\n3. **Replay.** Once a route has 50 sampled requests, the sweep replays them under each profile: effort one step\ndown, a concision instruction appended as user text, the instruction plus a one-line exemplar, or both levers.\n4. **Judge.** A judge model labels every candidate answer against the baseline answer. Every pair of baseline\nanswers is judged too, which gives the floor.\n5. **Pin.** The cheapest profile that matches the floor within sampling error, confirmed on a holdout if one was\nrequested, and whose saving repays the sweep within a month, is pinned. Everything else is refused with a reason.\n\nRewrites never touch the system prompt or tools, so prompt caches keep hitting. Pins are tied to the instruction text they were validated under and drop after an upgrade that changes it.\n\nOne SQLite, one sweep CLI, four ways to put it in front of traffic. All honour `X-Optimizer: off|bypass` and\n`X-Optimizer-Route`, fail open on any pith error, and never log exception text. Pick one per route.\n\n|  | Standalone proxy | LiteLLM | Headroom | Portkey | \n|---|---|---|---|---|\n| Attaches as | `pith serve` | guardrail in `config.yaml` | two extensions, enabled by env | `default.webhook` hooks | \n| Profiles | effort and concision | concision | effort and concision | concision | \n| Streams recorded | yes | yes | yes | no | \n| Sweeps go through | provider APIs | the LiteLLM proxy | provider APIs | the Portkey gateway | \n\nAdd `pith.guardrail.PithGuardrail` as a guardrail with `mode: [pre_call, post_call]`. It records every chat\ncompletion and appends the pinned instruction to the last user message. Setup, keys and limits:\n[docs/litellm.md](https://github.com/dex0shubham/pith/blob/main/docs/litellm.md).\n\n`HEADROOM_PROXY_EXTENSIONS=pith HEADROOM_PIPELINE_EXTENSIONS=pith headroom proxy`. The proxy extension records and\nrewrites effort, the pipeline extension appends the instruction after compression. Keep Headroom's cache and output\nshaper off while pins are in use: [docs/headroom.md](https://github.com/dex0shubham/pith/blob/main/docs/headroom.md).\n\nTwo `default.webhook` hooks in the Portkey config call `POST /optimizer/portkey` on a running `pith serve`. OpenAI\nformat only, streams are not recorded, and the endpoint needs `webhook_token` when it is reachable beyond the\ngateway: [docs/portkey.md](https://github.com/dex0shubham/pith/blob/main/docs/portkey.md).\n\n- Per request: `X-Optimizer: off` forces the baseline.`X-Optimizer: bypass` also skips recording.\n- Per route: `[routes.\"<key>\"] enabled = false` in`pith.toml` .\n- Global: `OPTIMIZER_ENABLED=0` .\n\nAny pith-side failure forwards the original request unchanged. The proxy has no authentication of its own, so bind\n`listen` to a private interface.\n\n`pip install -e '.[dev]' && pytest` runs the offline suite. Live tests, the gateway mock tests and the weekly\nintegrations job are described in [docs/development.md](https://github.com/dex0shubham/pith/blob/main/docs/development.md).\nDesign spec: [docs/design/specs](https://github.com/dex0shubham/pith/tree/main/docs/design/specs).\n\nMIT, see `LICENSE`.", "url": "https://wpnews.pro/news/show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t", "canonical_source": "https://github.com/dex0shubham/pith", "published_at": "2026-10-22 10:32:26+00:00", "updated_at": "2026-10-22 10:40:06.520380+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools", "ai-agents"], "entities": ["Pith", "Anthropic", "Haiku 4.5", "Haiku 5.5", "Sonnet 5.5", "claude-sonnet-5-5", "GSM8K", "SAMSum"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t", "markdown": "https://wpnews.pro/news/show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t.md", "text": "https://wpnews.pro/news/show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t.txt", "jsonld": "https://wpnews.pro/news/show-hn-pith-a-proxy-that-cuts-llm-output-tokens-and-refuses-when-it-can-t.jsonld"}}