{"slug": "stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger", "title": "Stop Counting Calls, Start Counting Accepted Outputs: A Five-Stage Token Ledger Tutorial", "summary": "A developer published a five-stage tutorial for building a token ledger that tracks tokens per accepted output rather than tokens per call, arguing the former is the metric that determines whether free-tier LLM inference is genuinely cheap or merely deferred cost. The ledger uses a frozen JSON-serializable receipt dataclass that labels each run as free or paid and records an acceptance verdict, paired with a provider-agnostic adapter that prefers provider-reported token counts and explicitly flags estimates. The tutorial is disclosed as part of MonkeyCode's product outreach and uses that service's free model access as its metering backend.", "body_md": "Every free-tier LLM project I look at has the same blind spot. The dashboard says *requests*, the bill says *zero*, and nobody can answer the one question that matters: how many tokens did we burn to get one output we actually kept?\n\nSo here is the conclusion up front. **Tokens per call is a vanity metric. Tokens per accepted output is the number that decides whether free inference is cheap or just deferred cost.** In this tutorial we build the ledger that produces that number — from an empty file to a report you can paste into a design review.\n\nFive stages. Each one has code and a verification step, so you never advance on a stage you haven't seen work. The code is provider-agnostic on purpose: it talks to any chat endpoint behind a thin adapter, so the ledger stays useful no matter which backend you point it at.\n\nOne practical note on where to run it. I use MonkeyCode's free model access behind the adapter in Stage 2 so the meter has something real to measure without a card on file, and the free server option to keep the tiny stub endpoint alive while I iterate.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nBoth of those are availability claims supplied by the project, not benchmarks — quotas, limits, and terms change, so read the current ones on the site before you plan a sprint around them. The ledger below works the same either way.\n\nWhy start with the data shape? Because every metering project I've watched die, died the same way: somebody added a field in week three and the historical numbers stopped being comparable.\n\nFreeze the receipt first. Keep it boring and JSON-serializable.\n\n``` python\n# ledger/receipt.py\nfrom dataclasses import dataclass, asdict, field\nimport json, time, uuid\n\n@dataclass\nclass Receipt:\n    request_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])\n    stage: str = \"draft\"          # which pipeline step asked\n    backend: str = \"free\"         # \"free\" | \"paid\" — label it, never infer it\n    model: str = \"unknown\"\n    prompt_chars: int = 0\n    output_chars: int = 0\n    usage_prompt: int | None = None    # provider-reported, None when absent\n    usage_output: int | None = None\n    latency_ms: int = 0\n    status: str = \"ok\"                # ok | error | truncated\n    accepted: bool | None = None      # None = not judged yet\n\n    def to_json(self) -> str:\n        return json.dumps(asdict(self), sort_keys=True)\n```\n\nTwo fields carry all the weight. `backend` because a free run and a paid run are not the same observation and must never be averaged together. `accepted` because a receipt with no verdict is just a timestamp with extra steps.\n\n**Verify:**\n\n``` python\npython -c \"from ledger.receipt import Receipt; r=Receipt(); print(r.to_json()); assert r.accepted is None\"\n```\n\nIf that assertion fails, you've already started guessing. Stop and fix it.\n\nNow the adapter. The trap here is pretending you know the token count. You often don't — many endpoints return a usage block, some don't, and the ones that do disagree about what counts.\n\nSo: prefer provider numbers, fall back to an estimate, and **mark the estimate as an estimate**.\n\n``` python\n# ledger/meter.py\nimport time\nfrom .receipt import Receipt\n\ndef estimate_tokens(text: str) -> int:\n    # rough, deliberately conservative; NOT a tokenizer\n    return max(1, len(text) // 4)\n\ndef metered_call(adapter, prompt: str, stage: str, model: str) -> tuple[str, Receipt]:\n    rec = Receipt(stage=stage, model=model, backend=adapter.backend,\n                  prompt_chars=len(prompt))\n    t0 = time.monotonic()\n    try:\n        out, usage = adapter.chat(prompt)   # usage: dict | None\n        rec.status = \"ok\"\n    except Exception:\n        rec.status = \"error\"\n        rec.latency_ms = int((time.monotonic() - t0) * 1000)\n        return \"\", rec\n    rec.latency_ms = int((time.monotonic() - t0) * 1000)\n    rec.output_chars = len(out)\n    if usage:\n        rec.usage_prompt = usage.get(\"prompt_tokens\")\n        rec.usage_output = usage.get(\"completion_tokens\")\n    else:\n        rec.usage_prompt = estimate_tokens(prompt)\n        rec.usage_output = estimate_tokens(out)\n    return out, rec\n```\n\nThe `adapter.chat()` signature is a stub by design — write your own transport for whatever endpoint you're testing. Mine is twenty lines and returns `(text, usage_or_none)`. That's the whole contract.\n\n**Verify with a fake, before you spend a single token:**\n\n``` python\n# tests/test_meter.py\nfrom ledger.meter import metered_call\n\nclass FakeAdapter:\n    backend = \"free\"\n    def chat(self, prompt):\n        return \"ok\", {\"prompt_tokens\": 11, \"completion_tokens\": 2}\n\nclass NoUsageAdapter(FakeAdapter):\n    def chat(self, prompt):\n        return \"ok\", None\n\ndef test_provider_numbers_win():\n    _, rec = metered_call(FakeAdapter(), \"hello there\", \"draft\", \"m\")\n    assert (rec.usage_prompt, rec.usage_output) == (11, 2)\n\ndef test_estimate_is_still_recorded():\n    _, rec = metered_call(NoUsageAdapter(), \"hello there\", \"draft\", \"m\")\n    assert rec.usage_prompt > 0 and rec.status == \"ok\"\npython -m pytest tests/test_meter.py -q\n```\n\nGreen? Now point the adapter at a real free endpoint and run ten prompts. You should see `usage_prompt` populated or your estimate filling in — either way, no receipts with zeros.\n\nMetering after the fact is accounting. Metering before the call is control. The gate answers: *is the next request even allowed?*\n\nTwo rules earn their keep immediately. A projected-token ceiling, and an error-rate circuit breaker — because a free endpoint that starts 500-ing is burning your retry budget, not your token budget.\n\n``` python\n# ledger/gate.py\nfrom collections import deque\n\nclass Gate:\n    def __init__(self, max_tokens: int, window: int = 20, max_error_rate: float = 0.25):\n        self.max_tokens = max_tokens\n        self.window = deque(maxlen=window)\n        self.max_error_rate = max_error_rate\n        self.spent = 0\n\n    def record(self, rec):\n        if rec.status == \"error\":\n            self.window.append(1)\n        else:\n            self.window.append(0)\n            self.spent += (rec.usage_prompt or 0) + (rec.usage_output or 0)\n\n    def allow(self, projected_prompt_tokens: int) -> tuple[bool, str]:\n        if self.spent + projected_prompt_tokens > self.max_tokens:\n            return False, \"ceiling\"\n        if self.window and sum(self.window) / len(self.window) > self.max_error_rate:\n            return False, \"error_rate\"\n        return True, \"ok\"\n```\n\n**Verify with a table of cases, not vibes:**\n\n``` python\nfrom ledger.gate import Gate\nfrom ledger.receipt import Receipt\n\ndef test_gate_stops_at_ceiling():\n    g = Gate(max_tokens=100)\n    g.record(Receipt(usage_prompt=60, usage_output=30))\n    assert g.allow(20) == (False, \"ceiling\")\n    assert g.allow(5)[0] is True\n\ndef test_gate_trips_on_errors():\n    g = Gate(max_tokens=10_000, window=4, max_error_rate=0.25)\n    for _ in range(2):\n        g.record(Receipt(status=\"error\"))\n    assert g.allow(10) == (False, \"error_rate\")\n```\n\nThe ceiling number is yours to pick — start smaller than you think is polite. The point is that the gate refuses *before* the request leaves your process, so a surprise quota change produces a clean stop, not a half-finished batch.\n\nHere's where most metering tutorials quietly cheat. They count tokens and stop. But an output that fails your parser cost you the same tokens as one that passed.\n\nSo attach a verdict. Not a human opinion — a mechanical check: does it parse, does the test pass, does the field exist?\n\n``` php\n# ledger/accept.py\nimport json\n\ndef judge_json(rec, text: str) -> bool:\n    try:\n        json.loads(text)\n        rec.accepted = True\n    except json.JSONDecodeError:\n        rec.accepted = False\n    return rec.accepted\n```\n\n**Verify both directions:**\n\n``` python\nfrom ledger.receipt import Receipt\nfrom ledger.accept import judge_json\n\ndef test_accepts_valid():\n    r = Receipt()\n    assert judge_json(r, '{\"a\": 1}') is True\n\ndef test_rejects_garbage():\n    r = Receipt()\n    assert judge_json(r, 'Sure! Here is your JSON:') is False\n```\n\nThat second test is the one people skip, and it's the one that shows up in production. A model that answers conversationally instead of emitting JSON hasn't failed loudly — it has failed expensively.\n\nNow the payoff. Read your receipts back, split by backend, and compute the ratio.\n\n``` python\n# ledger/report.py\nimport json\nfrom collections import defaultdict\n\ndef report(path: str) -> dict:\n    buckets = defaultdict(lambda: {\"tokens\": 0, \"accepted\": 0, \"calls\": 0, \"errors\": 0})\n    with open(path) as f:\n        for line in f:\n            r = json.loads(line)\n            b = buckets[r[\"backend\"]]\n            b[\"calls\"] += 1\n            b[\"tokens\"] += (r[\"usage_prompt\"] or 0) + (r[\"usage_output\"] or 0)\n            b[\"errors\"] += r[\"status\"] == \"error\"\n            b[\"accepted\"] += bool(r[\"accepted\"])\n    for name, b in buckets.items():\n        b[\"tokens_per_accepted\"] = round(b[\"tokens\"] / b[\"accepted\"], 1) if b[\"accepted\"] else None\n    return dict(buckets)\npython\npython -c \"from ledger.report import report; import json; print(json.dumps(report('runs/2026-10-26.jsonl'), indent=2))\"\n```\n\n**Verify the report against a hand-built fixture first.** Two receipts, one accepted, 100 tokens total, ratio must be `100.0`. If your fixture disagrees with the function, the function is wrong — not the fixture.\n\nWhat do you do with the ratio? Compare it honestly.\n\n| Situation | Free backend wins? | Why | \n|---|---|---|\n| Prompt tuning, exploratory work | Yes | Cost of a bad call is zero; the ledger tells you the retry tax | \n| Batch job with a hard deadline | Rarely | Error-rate breaker trips more often than you'd like | \n| Pipeline with strict schema output | Depends | If tokens-per-accepted is high, free is not free | \n| Anything user-facing and latency-bound | No | You're measuring the wrong axis entirely | \n\nThat third row is the whole article. A free endpoint at 3,000 tokens per accepted output can lose to a paid one at 400.\n\nThe estimator is not a tokenizer. `len(text) // 4` is a blunt instrument and will mislead you on non-English text, code, or anything with heavy punctuation. Treat estimated numbers as directional only.\n\nProvider `usage` blocks are not standardized. If yours is missing fields, your ledger will silently under-count — add an assertion, not a comment.\n\nFree availability is not an SLA. Both claims I mentioned earlier are claims, not guarantees; the gate exists precisely because the ground can move under you.\n\nAnd the acceptance check is the weakest link. `json.loads` is easy. A semantic grader is not. If you can't define \"accepted\" mechanically, you can't compute the ratio — build that first.\n\nSkip this approach if you're in a regulated path needing audit-grade billing, if your latency budget is under a second end-to-end, or if your volume is high enough that the free tier is a rounding error anyway.\n\nFive stages, maybe ninety minutes, and you end up with a number that survives contact with a design review: tokens per accepted output, split by backend, with the errors counted instead of hidden.\n\nIf standing up the endpoint is the part that keeps holding you back, MonkeyCode's free server option is one way to get a live target for Stage 2 — verify the current terms yourself and let the ledger do the arguing.", "url": "https://wpnews.pro/news/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger", "canonical_source": "https://dev.to/aiio_8140/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger-tutorial-5gi6", "published_at": "2026-10-26 11:45:00+00:00", "updated_at": "2026-10-26 11:46:38.621874+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-infrastructure"], "entities": ["MonkeyCode"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger", "markdown": "https://wpnews.pro/news/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger.md", "text": "https://wpnews.pro/news/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger.txt", "jsonld": "https://wpnews.pro/news/stop-counting-calls-start-counting-accepted-outputs-a-five-stage-token-ledger.jsonld"}}