Show HN: Pith – a proxy that cuts LLM output tokens, and refuses when it can't A developer released Pith, an open-source proxy (installable via `pip install pith-optimizer`) that cuts LLM output-token spend by replaying sampled traffic under cheaper output profiles and pinning a profile only when a judge model finds it as good as the baseline is with itself. Across ten live runs on the Anthropic API with 200 prompts each and judge `claude-sonnet-5-5`, Pith pinned only two routes: repeated support tickets on Haiku 4.5 at −51% tokens for a $1.30 sweep, and GSM8K on Sonnet 5.5 at 25k requests/month for −37% tokens and −26% cost, confirmed on a 30% holdout. Pith refused the other eight routes, including SAMSum summarisation at −53% tokens (0.64 vs floor 0.84) because it dropped information, and ticket classification at −44% tokens because the sweep was not repaid at 13k requests/month. Cut LLM output-token spend without losing information. pith watches a route, replays a sample under cheaper output profiles, has a judge compare every answer with the baseline, and pins a profile only when it is as good as the baseline is with itself. When it is not, pith says so and changes nothing. pip install pith-optimizer && pith demo The demo runs the whole loop in-process against a mock model and a mock judge, with no API key and no data leaving the machine. Its numbers are synthetic. The measured ones are below. Ten live runs on the Anthropic API, 200 prompts each unless noted, three trials per prompt, judge claude-sonnet-5-5 . "Floor" is how often the unconstrained model agrees with itself, which is the bar a candidate has to meet. | Route | Model | Best candidate | Outcome | Sweep cost | |---|---|---|---|---| | Support tickets, 10 repeated, 30 requests | Haiku 4.5 | −51% tokens at the floor | pinned | $1.30 | | Support tickets, 200 distinct | Haiku 4.5 | −53% tokens, 0.78 vs floor 0.83 | refused, below floor | $5.99 | | Ticket classification | Haiku 4.5 | −44% tokens at the floor | refused, sweep not repaid at 13k requests/month | $2.49 | | JSON extraction | Haiku 4.5 | −5% tokens | refused, nothing to remove | $2.44 | | SAMSum summarisation | Haiku 4.5 | −53% tokens, 0.64 vs floor 0.84 | refused, drops information | $5.61 | | Dolly summarisation | Haiku 4.5 | −55% tokens, 0.45 vs floor 0.77 | refused, drops information | $6.20 | | Bitext support replies | Haiku 4.5 | −54% tokens, 0.51 vs floor 0.72 | refused, drops information | $6.31 | | GSM8K | Haiku 5.5 | −35% tokens at the floor | refused, sweep not repaid at 10k requests/month | $4.47 | | GSM8K | Sonnet 5.5 | −40% tokens at the floor | refused, sweep not repaid at 10k requests/month | $7.13 | | GSM8K, 25k requests/month | Sonnet 5.5 | −37% tokens, −26% cost | pinned, confirmed on a 30% holdout | $8.38 | Two lessons, stated in full in docs/results.md https://github.com/dex0shubham/pith/blob/main/docs/results.md . A concision instruction saves tokens by dropping content, and on summaries and support replies the judge catches that every time. An effort step-down on a thinking model saves thinking tokens the task did not need, and then the only question is whether the route's volume repays the sweep. Decision rules and their reasons: docs/methodology.md https://github.com/dex0shubham/pith/blob/main/docs/methodology.md . pip install pith-optimizer pith serve defaults; or --config pith.toml pith.toml starts from pith.example.toml https://github.com/dex0shubham/pith/blob/main/pith.example.toml . Point your client at it and keep your own API key: ANTHROPIC BASE URL=http://localhost:8787 Anthropic SDKs OPENAI BASE URL=http://localhost:8787/v1 OpenAI SDKs Sweep from the CLI, which is the only place a provider key lives: export ANTHROPIC API KEY=... and/or OPENAI API KEY pith sweep --config pith.toml --dry-run estimate, print, write nothing pith sweep --config pith.toml --holdout 0.3 pin the cheapest profile that clears the bar pith recheck --config pith.toml re-judge live traffic, revert on drift Reports: GET /optimizer/report.html . Every scalar setting is also an OPTIMIZER