{"slug": "where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt", "title": "Where Did the LLM Tokens Go? A 100-Line Proxy That Finds Your Priciest Prompt", "summary": "A 100-line Python script called tokenlens lets teams attribute LLM spending to individual app features by sitting as a proxy between the application and an OpenAI-compatible provider, tagging each call with an X-Feature header and logging token usage to SQLite. The proxy, which uses only the standard library on Python 3.9+, records a short hash of the system prompt rather than prompt text, retries on a fallback provider when the primary returns 429 or 5xx errors, and reports cost and share per prompt through a `python tokenlens.py report 7` command. The author frames the tool around a concrete case: a $144 weekly LLM bill that providers break down only by model, API key or project, leaving teams to guess which feature drove the spend.", "body_md": "Your LLM bill says $144 this week. Which feature spent it? Providers typically break usage down by model, API key or project, not by the part of your app that made the call. So unless every feature has its own key, you guess, and guessing is how you spend a day shrinking a prompt that was 2% of the bill.\n\nThe fix is a small proxy between your app and the provider. It tags each call with the feature that made it, adds up the tokens, and lets one query show who is spending what.\n\n``` php\nflowchart LR\n    A[Your app] -->|X-Feature header| P[tokenlens proxy]\n    P -->|primary| U1[LLM provider]\n    P -.->|on 429 or 5xx| U2[Fallback provider]\n    P --> D[(SQLite tokens per call)]\n    D --> R[Report cost and share per prompt]\n```\n\n## How it works\n\n- Your app talks to the proxy instead of the provider, with one extra header per call: `X-Feature: doc-summary` .\n- The proxy forwards the request, reads the `usage` numbers from the response, and writes one row to SQLite. It never stores prompt text, only a short hash of the system prompt, so two versions of a prompt show up as two separate rows. If your system prompt embeds per-request data (the document itself, say), that hash changes on every call. Send an`X-Prompt: doc-summary-v1` header instead and the proxy uses that name.\n- If the provider answers 429 or a 5xx error, or can't be reached, the proxy retries on a fallback provider before your app sees anything.\n\n## The script\n\nStandard library only, Python 3.9+. Save it as `tokenlens.py`:\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"tokenlens: a logging proxy for OpenAI-compatible APIs.\n   python tokenlens.py            # proxy on 127.0.0.1:8787\n   python tokenlens.py report 7   # who spent what, last 7 days\n   Headers: X-Feature = which part of your app; X-Prompt = optional prompt name\n\"\"\"\nimport hashlib, http.client, json, os, sqlite3, sys, time, urllib.parse\nfrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer\n\nDB = os.environ.get(\"TOKENLENS_DB\", \"tokenlens.db\")\nUPSTREAMS = [  # tried in order; \"model\" overrides the model name (None = keep it)\n    {\"base\": os.environ.get(\"PRIMARY_BASE\", \"https://api.openai.com/v1\"),\n     \"key\": os.environ.get(\"PRIMARY_KEY\", \"\"), \"model\": None}]\nif os.environ.get(\"FALLBACK_BASE\"):\n    UPSTREAMS.append({\"base\": os.environ[\"FALLBACK_BASE\"],\n                      \"key\": os.environ.get(\"FALLBACK_KEY\", \"\"),\n                      \"model\": os.environ.get(\"FALLBACK_MODEL\")})\n# USD per 1M tokens (input, output). Placeholders: use your provider's current prices.\nPRICES = {\"model-a\": (0.50, 1.50), \"model-b\": (3.00, 15.00)}\n\ndef db():\n    c = sqlite3.connect(DB, timeout=10)\n    c.execute(\"PRAGMA journal_mode=WAL\")\n    c.execute(\"CREATE TABLE IF NOT EXISTS calls(ts REAL, feature TEXT, prompt TEXT, model TEXT,\"\n              \" tin INT, tout INT, ms INT, status INT, upstream INT)\")\n    return c\n\nclass Proxy(BaseHTTPRequestHandler):\n    def log_message(self, *args):\n        pass\n\n    def do_POST(self):\n        body = json.loads(self.rfile.read(int(self.headers.get(\"Content-Length\", 0))) or b\"{}\")\n        feature = self.headers.get(\"X-Feature\", \"untagged\")\n        system = next((m.get(\"content\") for m in body.get(\"messages\") or []\n                       if m.get(\"role\") in (\"system\", \"developer\")), None)  # hashed, never stored\n        prompt = self.headers.get(\"X-Prompt\") or hashlib.sha1(json.dumps(system).encode()).hexdigest()[:8]\n        stream = bool(body.get(\"stream\"))\n        if stream and self.path.endswith(\"/chat/completions\"):  # ask for the usage chunk, or streams log 0 tokens\n            body[\"stream_options\"] = {**(body.get(\"stream_options\") or {}), \"include_usage\": True}\n        t0, usage, status, model, used = time.time(), {}, 502, body.get(\"model\"), -1\n        for i, up in enumerate(UPSTREAMS):\n            sent = dict(body, model=up[\"model\"] or body.get(\"model\"))\n            u = urllib.parse.urlparse(up[\"base\"])\n            Conn = http.client.HTTPSConnection if u.scheme == \"https\" else http.client.HTTPConnection\n            try:\n                conn = Conn(u.netloc, timeout=120)\n                conn.request(\"POST\", u.path + self.path.removeprefix(\"/v1\"), json.dumps(sent),\n                             {\"Content-Type\": \"application/json\", \"Authorization\": \"Bearer \" + up[\"key\"]})\n                r = conn.getresponse()\n            except OSError:\n                continue\n            if (r.status == 429 or r.status >= 500) and i < len(UPSTREAMS) - 1:\n                r.read(); conn.close(); continue  # fail over before sending a byte to the client\n            status, model, used = r.status, sent[\"model\"], i\n            self.send_response(status)\n            self.send_header(\"Content-Type\", r.getheader(\"Content-Type\", \"application/json\"))\n            try:\n                if stream:\n                    self.end_headers()\n                    for line in iter(r.readline, b\"\"):\n                        self.wfile.write(line); self.wfile.flush()\n                        d = line[5:].strip() if line.startswith(b\"data:\") else b\"\"\n                        if d.startswith(b\"{\"):\n                            try:  # chat: top-level usage; Responses API: inside \"response\"\n                                ev = json.loads(d)\n                                usage = ev.get(\"usage\") or (ev.get(\"response\") or {}).get(\"usage\") or usage\n                            except (ValueError, AttributeError): pass\n                else:\n                    data = r.read()\n                    self.send_header(\"Content-Length\", str(len(data)))\n                    self.end_headers(); self.wfile.write(data)\n                    try: usage = json.loads(data).get(\"usage\") or {}\n                    except (ValueError, AttributeError): pass\n            except (OSError, http.client.HTTPException):\n                pass  # client or upstream dropped mid-response; still log the call\n            finally:\n                conn.close()\n            break\n        else:\n            self.send_error(502, \"all upstreams failed\")\n        tin = usage.get(\"prompt_tokens\") or usage.get(\"input_tokens\") or 0\n        tout = usage.get(\"completion_tokens\") or usage.get(\"output_tokens\") or 0\n        c = db()\n        c.execute(\"INSERT INTO calls VALUES(?,?,?,?,?,?,?,?,?)\", (t0, feature, prompt, model, tin,\n                  tout, int((time.time() - t0) * 1000), status, used))\n        c.commit(); c.close()\n\ndef report(days=7):\n    rows = db().execute(\"SELECT feature, prompt, model, COUNT(*), SUM(tin), SUM(tout) FROM calls \"\n                        \"WHERE ts > ? AND status < 400 GROUP BY 1, 2, 3\",\n                        (time.time() - days * 86400,)).fetchall()\n    cost = lambda m, i, o: (i * PRICES.get(m, (0, 0))[0] + o * PRICES.get(m, (0, 0))[1]) / 1e6\n    table = sorted(((cost(m, i, o), f, p, m, n, i, o) for f, p, m, n, i, o in rows), reverse=True)\n    total = sum(r[0] for r in table) or 1\n    print(f\"{'feature':<18}{'prompt':<14}{'model':<9}{'calls':>6}{'in':>10}{'out':>9}{'cost $':>9}{'share':>7}\")\n    for c, f, p, m, n, i, o in table:\n        print(f\"{f:<18}{p:<14}{m:<9}{n:>6}{i:>10}{o:>9}{c:>9.2f}{c / total:>7.0%}\")\n\nif __name__ == \"__main__\":\n    if sys.argv[1:2] == [\"report\"]:\n        report(int(sys.argv[2]) if len(sys.argv) > 2 else 7)\n    else:\n        db().close()\n        print(\"tokenlens listening on http://127.0.0.1:8787\")\n        ThreadingHTTPServer((\"127.0.0.1\", 8787), Proxy).serve_forever()\n```\n\nStart it, keeping your provider key here and out of the app:\n\n```\nexport PRIMARY_KEY=sk-your-key        # PRIMARY_BASE defaults to https://api.openai.com/v1\npython tokenlens.py                   # listens on 127.0.0.1:8787\n```\n\nFor failover, also set `FALLBACK_BASE`, `FALLBACK_KEY` and `FALLBACK_MODEL` (the fallback's name for the model). Then point your app at the proxy and tag each feature. With the OpenAI Python SDK:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://127.0.0.1:8787/v1\", api_key=\"unused\",\n                default_headers={\"X-Feature\": \"doc-summary\"})\n```\n\nUse `extra_headers={\"X-Feature\": \"email-reply\"}` on a single call to override the default. Last, edit `PRICES` so the report uses your models' current prices.\n\n## Reading the report\n\nAfter a few days of traffic:\n\n```\npython tokenlens.py report 7\n```\n\nExample output, from a simulated week with placeholder prices. Yours will differ:\n\n``` bash\nfeature           prompt        model     calls        in      out   cost $  share\ndoc-summary       9f3a1c02      model-b    1700  22000000  1300000    85.50    60%\nemail-reply       c71d0e55      model-b    1200   3600000   900000    24.30    17%\nsearch-rerank     4be8a9d1      model-a   18000  27000000  1800000    16.20    11%\nsupport-chat      2a6f77b3      model-a    5000  15000000  4000000    13.50     9%\ntagging           e03b5c48      model-a    9500   4700000   200000     2.65     2%\nuntagged          5d90ae17      model-a     900   2000000   300000     1.45     1%\n```\n\nOne row is 60% of the spend. `doc-summary` sends about 13,000 input tokens per call (22M tokens over 1,700 calls), most likely because the whole document goes in with every request, and it runs on the expensive model. `search-rerank` makes more than ten times as many calls and costs 11%. Call counts mislead. Tokens times price doesn't.\n\nFix the top row first. These are the usual levers, roughly in order of payoff:\n\n1. **Send less.** Pass the relevant chunks instead of the whole document.\n2. **Route easy cases to the cheaper model.** Keep the expensive one for the hard ones.\n3. **Use prompt caching** for the repeated prefix, if your provider supports it.\n4. **Cap** `max_tokens` on replies that never need to be long.\n\nRun the report again a week later. The row should shrink, and now you can prove it.\n\n## Things to know\n\n- **It has no auth and holds your provider keys.** It binds to`127.0.0.1` . Keep it on localhost or inside a private network.\n- **Streaming needs one extra field.** For Chat Completions streams, the proxy adds`stream_options: {\"include_usage\": true}` , the OpenAI-compatible way to get token counts at the end of a stream. If your provider rejects that field, remove that line, but streamed calls will then log zero tokens.\n- **Costs are estimates.** The report applies one input price and one output price per model, so cached-input discounts and other special rates aren't reflected. Calls cut off before the final usage chunk (a dropped stream, a hung-up client) are logged with zero tokens, so they are under-counted. Compare the totals with your invoice.\n- **Scope.** It proxies OpenAI-compatible`POST` endpoints and reads usage from Chat Completions replies and from Responses-style replies, where a stream carries the usage in its final event. A fallback provider must speak the same format.\n\nAdd one header per call site, wait a week, and optimise whichever row is at the top.", "url": "https://wpnews.pro/news/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt", "canonical_source": "https://zyvop.com/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt-b1pxy", "published_at": "2026-11-07 07:39:19+00:00", "updated_at": "2026-11-09 01:43:37.445413+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["tokenlens", "OpenAI", "SQLite", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt", "markdown": "https://wpnews.pro/news/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt.md", "text": "https://wpnews.pro/news/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt.txt", "jsonld": "https://wpnews.pro/news/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt.jsonld"}}