Stop Counting Calls, Start Counting Accepted Outputs: A Five-Stage Token Ledger Tutorial

A developer published a five-stage tutorial for building a token ledger that tracks tokens per accepted output rather than tokens per call, arguing the former is the metric that determines whether free-tier LLM inference is genuinely cheap or merely deferred cost. The ledger uses a frozen JSON-serializable receipt dataclass that labels each run as free or paid and records an acceptance verdict, paired with a provider-agnostic adapter that prefers provider-reported token counts and explicitly flags estimates. The tutorial is disclosed as part of MonkeyCode's product outreach and uses that service's free model access as its metering backend.

Every free-tier LLM project I look at has the same blind spot. The dashboard says requests , the bill says zero , and nobody can answer the one question that matters: how many tokens did we burn to get one output we actually kept? So here is the conclusion up front. Tokens per call is a vanity metric. Tokens per accepted output is the number that decides whether free inference is cheap or just deferred cost. In this tutorial we build the ledger that produces that number — from an empty file to a report you can paste into a design review. Five stages. Each one has code and a verification step, so you never advance on a stage you haven't seen work. The code is provider-agnostic on purpose: it talks to any chat endpoint behind a thin adapter, so the ledger stays useful no matter which backend you point it at. One practical note on where to run it. I use MonkeyCode's free model access behind the adapter in Stage 2 so the meter has something real to measure without a card on file, and the free server option to keep the tiny stub endpoint alive while I iterate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both of those are availability claims supplied by the project, not benchmarks — quotas, limits, and terms change, so read the current ones on the site before you plan a sprint around them. The ledger below works the same either way. Why start with the data shape? Because every metering project I've watched die, died the same way: somebody added a field in week three and the historical numbers stopped being comparable. Freeze the receipt first. Keep it boring and JSON-serializable. python ledger/receipt.py from dataclasses import dataclass, asdict, field import json, time, uuid @dataclass class Receipt: request id: str = field default factory=lambda: uuid.uuid4 .hex :12 stage: str = "draft" which pipeline step asked backend: str = "free" "free" | "paid" — label it, never infer it model: str = "unknown" prompt chars: int = 0 output chars: int = 0 usage prompt: int | None = None provider-reported, None when absent usage output: int | None = None latency ms: int = 0 status: str = "ok" ok | error | truncated accepted: bool | None = None None = not judged yet def to json self - str: return json.dumps asdict self , sort keys=True Two fields carry all the weight. backend because a free run and a paid run are not the same observation and must never be averaged together. accepted because a receipt with no verdict is just a timestamp with extra steps. Verify: python python -c "from ledger.receipt import Receipt; r=Receipt ; print r.to json ; assert r.accepted is None" If that assertion fails, you've already started guessing. Stop and fix it. Now the adapter. The trap here is pretending you know the token count. You often don't — many endpoints return a usage block, some don't, and the ones that do disagree about what counts. So: prefer provider numbers, fall back to an estimate, and mark the estimate as an estimate . python ledger/meter.py import time from .receipt import Receipt def estimate tokens text: str - int: rough, deliberately conservative; NOT a tokenizer return max 1, len text // 4 def metered call adapter, prompt: str, stage: str, model: str - tuple str, Receipt : rec = Receipt stage=stage, model=model, backend=adapter.backend, prompt chars=len prompt t0 = time.monotonic try: out, usage = adapter.chat prompt usage: dict | None rec.status = "ok" except Exception: rec.status = "error" rec.latency ms = int time.monotonic - t0 1000 return "", rec rec.latency ms = int time.monotonic - t0 1000 rec.output chars = len out if usage: rec.usage prompt = usage.get "prompt tokens" rec.usage output = usage.get "completion tokens" else: rec.usage prompt = estimate tokens prompt rec.usage output = estimate tokens out return out, rec The adapter.chat signature is a stub by design — write your own transport for whatever endpoint you're testing. Mine is twenty lines and returns text, usage or none . That's the whole contract. Verify with a fake, before you spend a single token: python tests/test meter.py from ledger.meter import metered call class FakeAdapter: backend = "free" def chat self, prompt : return "ok", {"prompt tokens": 11, "completion tokens": 2} class NoUsageAdapter FakeAdapter : def chat self, prompt : return "ok", None def test provider numbers win : , rec = metered call FakeAdapter , "hello there", "draft", "m" assert rec.usage prompt, rec.usage output == 11, 2 def test estimate is still recorded : , rec = metered call NoUsageAdapter , "hello there", "draft", "m" assert rec.usage prompt 0 and rec.status == "ok" python -m pytest tests/test meter.py -q Green? Now point the adapter at a real free endpoint and run ten prompts. You should see usage prompt populated or your estimate filling in — either way, no receipts with zeros. Metering after the fact is accounting. Metering before the call is control. The gate answers: is the next request even allowed? Two rules earn their keep immediately. A projected-token ceiling, and an error-rate circuit breaker — because a free endpoint that starts 500-ing is burning your retry budget, not your token budget. python ledger/gate.py from collections import deque class Gate: def init self, max tokens: int, window: int = 20, max error rate: float = 0.25 : self.max tokens = max tokens self.window = deque maxlen=window self.max error rate = max error rate self.spent = 0 def record self, rec : if rec.status == "error": self.window.append 1 else: self.window.append 0 self.spent += rec.usage prompt or 0 + rec.usage output or 0 def allow self, projected prompt tokens: int - tuple bool, str : if self.spent + projected prompt tokens self.max tokens: return False, "ceiling" if self.window and sum self.window / len self.window self.max error rate: return False, "error rate" return True, "ok" Verify with a table of cases, not vibes: python from ledger.gate import Gate from ledger.receipt import Receipt def test gate stops at ceiling : g = Gate max tokens=100 g.record Receipt usage prompt=60, usage output=30 assert g.allow 20 == False, "ceiling" assert g.allow 5 0 is True def test gate trips on errors : g = Gate max tokens=10 000, window=4, max error rate=0.25 for in range 2 : g.record Receipt status="error" assert g.allow 10 == False, "error rate" The ceiling number is yours to pick — start smaller than you think is polite. The point is that the gate refuses before the request leaves your process, so a surprise quota change produces a clean stop, not a half-finished batch. Here's where most metering tutorials quietly cheat. They count tokens and stop. But an output that fails your parser cost you the same tokens as one that passed. So attach a verdict. Not a human opinion — a mechanical check: does it parse, does the test pass, does the field exist? php ledger/accept.py import json def judge json rec, text: str - bool: try: json.loads text rec.accepted = True except json.JSONDecodeError: rec.accepted = False return rec.accepted Verify both directions: python from ledger.receipt import Receipt from ledger.accept import judge json def test accepts valid : r = Receipt assert judge json r, '{"a": 1}' is True def test rejects garbage : r = Receipt assert judge json r, 'Sure Here is your JSON:' is False That second test is the one people skip, and it's the one that shows up in production. A model that answers conversationally instead of emitting JSON hasn't failed loudly — it has failed expensively. Now the payoff. Read your receipts back, split by backend, and compute the ratio. python ledger/report.py import json from collections import defaultdict def report path: str - dict: buckets = defaultdict lambda: {"tokens": 0, "accepted": 0, "calls": 0, "errors": 0} with open path as f: for line in f: r = json.loads line b = buckets r "backend" b "calls" += 1 b "tokens" += r "usage prompt" or 0 + r "usage output" or 0 b "errors" += r "status" == "error" b "accepted" += bool r "accepted" for name, b in buckets.items : b "tokens per accepted" = round b "tokens" / b "accepted" , 1 if b "accepted" else None return dict buckets python python -c "from ledger.report import report; import json; print json.dumps report 'runs/2026-10-26.jsonl' , indent=2 " Verify the report against a hand-built fixture first. Two receipts, one accepted, 100 tokens total, ratio must be 100.0 . If your fixture disagrees with the function, the function is wrong — not the fixture. What do you do with the ratio? Compare it honestly. | Situation | Free backend wins? | Why | |---|---|---| | Prompt tuning, exploratory work | Yes | Cost of a bad call is zero; the ledger tells you the retry tax | | Batch job with a hard deadline | Rarely | Error-rate breaker trips more often than you'd like | | Pipeline with strict schema output | Depends | If tokens-per-accepted is high, free is not free | | Anything user-facing and latency-bound | No | You're measuring the wrong axis entirely | That third row is the whole article. A free endpoint at 3,000 tokens per accepted output can lose to a paid one at 400. The estimator is not a tokenizer. len text // 4 is a blunt instrument and will mislead you on non-English text, code, or anything with heavy punctuation. Treat estimated numbers as directional only. Provider usage blocks are not standardized. If yours is missing fields, your ledger will silently under-count — add an assertion, not a comment. Free availability is not an SLA. Both claims I mentioned earlier are claims, not guarantees; the gate exists precisely because the ground can move under you. And the acceptance check is the weakest link. json.loads is easy. A semantic grader is not. If you can't define "accepted" mechanically, you can't compute the ratio — build that first. Skip this approach if you're in a regulated path needing audit-grade billing, if your latency budget is under a second end-to-end, or if your volume is high enough that the free tier is a rounding error anyway. Five stages, maybe ninety minutes, and you end up with a number that survives contact with a design review: tokens per accepted output, split by backend, with the errors counted instead of hidden. If standing up the endpoint is the part that keeps holding you back, MonkeyCode's free server option is one way to get a live target for Stage 2 — verify the current terms yourself and let the ledger do the arguing.