Where Did the LLM Tokens Go? A 100-Line Proxy That Finds Your Priciest Prompt

A 100-line Python script called tokenlens lets teams attribute LLM spending to individual app features by sitting as a proxy between the application and an OpenAI-compatible provider, tagging each call with an X-Feature header and logging token usage to SQLite. The proxy, which uses only the standard library on Python 3.9+, records a short hash of the system prompt rather than prompt text, retries on a fallback provider when the primary returns 429 or 5xx errors, and reports cost and share per prompt through a `python tokenlens.py report 7` command. The author frames the tool around a concrete case: a $144 weekly LLM bill that providers break down only by model, API key or project, leaving teams to guess which feature drove the spend.

Your LLM bill says $144 this week. Which feature spent it? Providers typically break usage down by model, API key or project, not by the part of your app that made the call. So unless every feature has its own key, you guess, and guessing is how you spend a day shrinking a prompt that was 2% of the bill. The fix is a small proxy between your app and the provider. It tags each call with the feature that made it, adds up the tokens, and lets one query show who is spending what. php flowchart LR A Your app -- |X-Feature header| P tokenlens proxy P -- |primary| U1 LLM provider P -.- |on 429 or 5xx| U2 Fallback provider P -- D SQLite tokens per call D -- R Report cost and share per prompt How it works - Your app talks to the proxy instead of the provider, with one extra header per call: X-Feature: doc-summary . - The proxy forwards the request, reads the usage numbers from the response, and writes one row to SQLite. It never stores prompt text, only a short hash of the system prompt, so two versions of a prompt show up as two separate rows. If your system prompt embeds per-request data the document itself, say , that hash changes on every call. Send an X-Prompt: doc-summary-v1 header instead and the proxy uses that name. - If the provider answers 429 or a 5xx error, or can't be reached, the proxy retries on a fallback provider before your app sees anything. The script Standard library only, Python 3.9+. Save it as tokenlens.py : bash /usr/bin/env python3 """tokenlens: a logging proxy for OpenAI-compatible APIs. python tokenlens.py proxy on 127.0.0.1:8787 python tokenlens.py report 7 who spent what, last 7 days Headers: X-Feature = which part of your app; X-Prompt = optional prompt name """ import hashlib, http.client, json, os, sqlite3, sys, time, urllib.parse from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer DB = os.environ.get "TOKENLENS DB", "tokenlens.db" UPSTREAMS = tried in order; "model" overrides the model name None = keep it {"base": os.environ.get "PRIMARY BASE", "https://api.openai.com/v1" , "key": os.environ.get "PRIMARY KEY", "" , "model": None} if os.environ.get "FALLBACK BASE" : UPSTREAMS.append {"base": os.environ "FALLBACK BASE" , "key": os.environ.get "FALLBACK KEY", "" , "model": os.environ.get "FALLBACK MODEL" } USD per 1M tokens input, output . Placeholders: use your provider's current prices. PRICES = {"model-a": 0.50, 1.50 , "model-b": 3.00, 15.00 } def db : c = sqlite3.connect DB, timeout=10 c.execute "PRAGMA journal mode=WAL" c.execute "CREATE TABLE IF NOT EXISTS calls ts REAL, feature TEXT, prompt TEXT, model TEXT," " tin INT, tout INT, ms INT, status INT, upstream INT " return c class Proxy BaseHTTPRequestHandler : def log message self, args : pass def do POST self : body = json.loads self.rfile.read int self.headers.get "Content-Length", 0 or b"{}" feature = self.headers.get "X-Feature", "untagged" system = next m.get "content" for m in body.get "messages" or if m.get "role" in "system", "developer" , None hashed, never stored prompt = self.headers.get "X-Prompt" or hashlib.sha1 json.dumps system .encode .hexdigest :8 stream = bool body.get "stream" if stream and self.path.endswith "/chat/completions" : ask for the usage chunk, or streams log 0 tokens body "stream options" = { body.get "stream options" or {} , "include usage": True} t0, usage, status, model, used = time.time , {}, 502, body.get "model" , -1 for i, up in enumerate UPSTREAMS : sent = dict body, model=up "model" or body.get "model" u = urllib.parse.urlparse up "base" Conn = http.client.HTTPSConnection if u.scheme == "https" else http.client.HTTPConnection try: conn = Conn u.netloc, timeout=120 conn.request "POST", u.path + self.path.removeprefix "/v1" , json.dumps sent , {"Content-Type": "application/json", "Authorization": "Bearer " + up "key" } r = conn.getresponse except OSError: continue if r.status == 429 or r.status = 500 and i < len UPSTREAMS - 1: r.read ; conn.close ; continue fail over before sending a byte to the client status, model, used = r.status, sent "model" , i self.send response status self.send header "Content-Type", r.getheader "Content-Type", "application/json" try: if stream: self.end headers for line in iter r.readline, b"" : self.wfile.write line ; self.wfile.flush d = line 5: .strip if line.startswith b"data:" else b"" if d.startswith b"{" : try: chat: top-level usage; Responses API: inside "response" ev = json.loads d usage = ev.get "usage" or ev.get "response" or {} .get "usage" or usage except ValueError, AttributeError : pass else: data = r.read self.send header "Content-Length", str len data self.end headers ; self.wfile.write data try: usage = json.loads data .get "usage" or {} except ValueError, AttributeError : pass except OSError, http.client.HTTPException : pass client or upstream dropped mid-response; still log the call finally: conn.close break else: self.send error 502, "all upstreams failed" tin = usage.get "prompt tokens" or usage.get "input tokens" or 0 tout = usage.get "completion tokens" or usage.get "output tokens" or 0 c = db c.execute "INSERT INTO calls VALUES ?,?,?,?,?,?,?,?,? ", t0, feature, prompt, model, tin, tout, int time.time - t0 1000 , status, used c.commit ; c.close def report days=7 : rows = db .execute "SELECT feature, prompt, model, COUNT , SUM tin , SUM tout FROM calls " "WHERE ts ? AND status < 400 GROUP BY 1, 2, 3", time.time - days 86400, .fetchall cost = lambda m, i, o: i PRICES.get m, 0, 0 0 + o PRICES.get m, 0, 0 1 / 1e6 table = sorted cost m, i, o , f, p, m, n, i, o for f, p, m, n, i, o in rows , reverse=True total = sum r 0 for r in table or 1 print f"{'feature':<18}{'prompt':<14}{'model':<9}{'calls': 6}{'in': 10}{'out': 9}{'cost $': 9}{'share': 7}" for c, f, p, m, n, i, o in table: print f"{f:<18}{p:<14}{m:<9}{n: 6}{i: 10}{o: 9}{c: 9.2f}{c / total: 7.0%}" if name == " main ": if sys.argv 1:2 == "report" : report int sys.argv 2 if len sys.argv 2 else 7 else: db .close print "tokenlens listening on http://127.0.0.1:8787" ThreadingHTTPServer "127.0.0.1", 8787 , Proxy .serve forever Start it, keeping your provider key here and out of the app: export PRIMARY KEY=sk-your-key PRIMARY BASE defaults to https://api.openai.com/v1 python tokenlens.py listens on 127.0.0.1:8787 For failover, also set FALLBACK BASE , FALLBACK KEY and FALLBACK MODEL the fallback's name for the model . Then point your app at the proxy and tag each feature. With the OpenAI Python SDK: python from openai import OpenAI client = OpenAI base url="http://127.0.0.1:8787/v1", api key="unused", default headers={"X-Feature": "doc-summary"} Use extra headers={"X-Feature": "email-reply"} on a single call to override the default. Last, edit PRICES so the report uses your models' current prices. Reading the report After a few days of traffic: python tokenlens.py report 7 Example output, from a simulated week with placeholder prices. Yours will differ: bash feature prompt model calls in out cost $ share doc-summary 9f3a1c02 model-b 1700 22000000 1300000 85.50 60% email-reply c71d0e55 model-b 1200 3600000 900000 24.30 17% search-rerank 4be8a9d1 model-a 18000 27000000 1800000 16.20 11% support-chat 2a6f77b3 model-a 5000 15000000 4000000 13.50 9% tagging e03b5c48 model-a 9500 4700000 200000 2.65 2% untagged 5d90ae17 model-a 900 2000000 300000 1.45 1% One row is 60% of the spend. doc-summary sends about 13,000 input tokens per call 22M tokens over 1,700 calls , most likely because the whole document goes in with every request, and it runs on the expensive model. search-rerank makes more than ten times as many calls and costs 11%. Call counts mislead. Tokens times price doesn't. Fix the top row first. These are the usual levers, roughly in order of payoff: 1. Send less. Pass the relevant chunks instead of the whole document. 2. Route easy cases to the cheaper model. Keep the expensive one for the hard ones. 3. Use prompt caching for the repeated prefix, if your provider supports it. 4. Cap max tokens on replies that never need to be long. Run the report again a week later. The row should shrink, and now you can prove it. Things to know - It has no auth and holds your provider keys. It binds to 127.0.0.1 . Keep it on localhost or inside a private network. - Streaming needs one extra field. For Chat Completions streams, the proxy adds stream options: {"include usage": true} , the OpenAI-compatible way to get token counts at the end of a stream. If your provider rejects that field, remove that line, but streamed calls will then log zero tokens. - Costs are estimates. The report applies one input price and one output price per model, so cached-input discounts and other special rates aren't reflected. Calls cut off before the final usage chunk a dropped stream, a hung-up client are logged with zero tokens, so they are under-counted. Compare the totals with your invoice. - Scope. It proxies OpenAI-compatible POST endpoints and reads usage from Chat Completions replies and from Responses-style replies, where a stream carries the usage in its final event. A fallback provider must speak the same format. Add one header per call site, wait a week, and optimise whichever row is at the top.