Every free-tier LLM project I look at has the same blind spot. The dashboard says requests, the bill says zero, and nobody can answer the one question that matters: how many tokens did we burn to get one output we actually kept?
So here is the conclusion up front. Tokens per call is a vanity metric. Tokens per accepted output is the number that decides whether free inference is cheap or just deferred cost. In this tutorial we build the ledger that produces that number — from an empty file to a report you can paste into a design review.
Five stages. Each one has code and a verification step, so you never advance on a stage you haven't seen work. The code is provider-agnostic on purpose: it talks to any chat endpoint behind a thin adapter, so the ledger stays useful no matter which backend you point it at.
One practical note on where to run it. I use MonkeyCode's free model access behind the adapter in Stage 2 so the meter has something real to measure without a card on file, and the free server option to keep the tiny stub endpoint alive while I iterate.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Both of those are availability claims supplied by the project, not benchmarks — quotas, limits, and terms change, so read the current ones on the site before you plan a sprint around them. The ledger below works the same either way.
Why start with the data shape? Because every metering project I've watched die, died the same way: somebody added a field in week three and the historical numbers stopped being comparable.
Freeze the receipt first. Keep it boring and JSON-serializable.
from dataclasses import dataclass, asdict, field
import json, time, uuid
@dataclass
class Receipt:
request_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
stage: str = "draft" # which pipeline step asked
backend: str = "free" # "free" | "paid" — label it, never infer it
model: str = "unknown"
prompt_chars: int = 0
output_chars: int = 0
usage_prompt: int | None = None # provider-reported, None when absent
usage_output: int | None = None
latency_ms: int = 0
status: str = "ok" # ok | error | truncated
accepted: bool | None = None # None = not judged yet
def to_json(self) -> str:
return json.dumps(asdict(self), sort_keys=True)
Two fields carry all the weight. backend because a free run and a paid run are not the same observation and must never be averaged together. accepted because a receipt with no verdict is just a timestamp with extra steps.
Verify:
python -c "from ledger.receipt import Receipt; r=Receipt(); print(r.to_json()); assert r.accepted is None"
If that assertion fails, you've already started guessing. Stop and fix it.
Now the adapter. The trap here is pretending you know the token count. You often don't — many endpoints return a usage block, some don't, and the ones that do disagree about what counts.
So: prefer provider numbers, fall back to an estimate, and mark the estimate as an estimate.
import time
from .receipt import Receipt
def estimate_tokens(text: str) -> int:
return max(1, len(text) // 4)
def metered_call(adapter, prompt: str, stage: str, model: str) -> tuple[str, Receipt]:
rec = Receipt(stage=stage, model=model, backend=adapter.backend,
prompt_chars=len(prompt))
t0 = time.monotonic()
try:
out, usage = adapter.chat(prompt) # usage: dict | None
rec.status = "ok"
except Exception:
rec.status = "error"
rec.latency_ms = int((time.monotonic() - t0) * 1000)
return "", rec
rec.latency_ms = int((time.monotonic() - t0) * 1000)
rec.output_chars = len(out)
if usage:
rec.usage_prompt = usage.get("prompt_tokens")
rec.usage_output = usage.get("completion_tokens")
else:
rec.usage_prompt = estimate_tokens(prompt)
rec.usage_output = estimate_tokens(out)
return out, rec
The adapter.chat() signature is a stub by design — write your own transport for whatever endpoint you're testing. Mine is twenty lines and returns (text, usage_or_none). That's the whole contract.
Verify with a fake, before you spend a single token:
from ledger.meter import metered_call
class FakeAdapter:
backend = "free"
def chat(self, prompt):
return "ok", {"prompt_tokens": 11, "completion_tokens": 2}
class NoUsageAdapter(FakeAdapter):
def chat(self, prompt):
return "ok", None
def test_provider_numbers_win():
_, rec = metered_call(FakeAdapter(), "hello there", "draft", "m")
assert (rec.usage_prompt, rec.usage_output) == (11, 2)
def test_estimate_is_still_recorded():
_, rec = metered_call(NoUsageAdapter(), "hello there", "draft", "m")
assert rec.usage_prompt > 0 and rec.status == "ok"
python -m pytest tests/test_meter.py -q
Green? Now point the adapter at a real free endpoint and run ten prompts. You should see usage_prompt populated or your estimate filling in — either way, no receipts with zeros.
Metering after the fact is accounting. Metering before the call is control. The gate answers: is the next request even allowed?
Two rules earn their keep immediately. A projected-token ceiling, and an error-rate circuit breaker — because a free endpoint that starts 500-ing is burning your retry budget, not your token budget.
from collections import deque
class Gate:
def __init__(self, max_tokens: int, window: int = 20, max_error_rate: float = 0.25):
self.max_tokens = max_tokens
self.window = deque(maxlen=window)
self.max_error_rate = max_error_rate
self.spent = 0
def record(self, rec):
if rec.status == "error":
self.window.append(1)
else:
self.window.append(0)
self.spent += (rec.usage_prompt or 0) + (rec.usage_output or 0)
def allow(self, projected_prompt_tokens: int) -> tuple[bool, str]:
if self.spent + projected_prompt_tokens > self.max_tokens:
return False, "ceiling"
if self.window and sum(self.window) / len(self.window) > self.max_error_rate:
return False, "error_rate"
return True, "ok"
Verify with a table of cases, not vibes:
from ledger.gate import Gate
from ledger.receipt import Receipt
def test_gate_stops_at_ceiling():
g = Gate(max_tokens=100)
g.record(Receipt(usage_prompt=60, usage_output=30))
assert g.allow(20) == (False, "ceiling")
assert g.allow(5)[0] is True
def test_gate_trips_on_errors():
g = Gate(max_tokens=10_000, window=4, max_error_rate=0.25)
for _ in range(2):
g.record(Receipt(status="error"))
assert g.allow(10) == (False, "error_rate")
The ceiling number is yours to pick — start smaller than you think is polite. The point is that the gate refuses before the request leaves your process, so a surprise quota change produces a clean stop, not a half-finished batch.
Here's where most metering tutorials quietly cheat. They count tokens and stop. But an output that fails your parser cost you the same tokens as one that passed.
So attach a verdict. Not a human opinion — a mechanical check: does it parse, does the test pass, does the field exist?
import json
def judge_json(rec, text: str) -> bool:
try:
json.loads(text)
rec.accepted = True
except json.JSONDecodeError:
rec.accepted = False
return rec.accepted
Verify both directions:
from ledger.receipt import Receipt
from ledger.accept import judge_json
def test_accepts_valid():
r = Receipt()
assert judge_json(r, '{"a": 1}') is True
def test_rejects_garbage():
r = Receipt()
assert judge_json(r, 'Sure! Here is your JSON:') is False
That second test is the one people skip, and it's the one that shows up in production. A model that answers conversationally instead of emitting JSON hasn't failed loudly — it has failed expensively.
Now the payoff. Read your receipts back, split by backend, and compute the ratio.
import json
from collections import defaultdict
def report(path: str) -> dict:
buckets = defaultdict(lambda: {"tokens": 0, "accepted": 0, "calls": 0, "errors": 0})
with open(path) as f:
for line in f:
r = json.loads(line)
b = buckets[r["backend"]]
b["calls"] += 1
b["tokens"] += (r["usage_prompt"] or 0) + (r["usage_output"] or 0)
b["errors"] += r["status"] == "error"
b["accepted"] += bool(r["accepted"])
for name, b in buckets.items():
b["tokens_per_accepted"] = round(b["tokens"] / b["accepted"], 1) if b["accepted"] else None
return dict(buckets)
python
python -c "from ledger.report import report; import json; print(json.dumps(report('runs/2026-10-26.jsonl'), indent=2))"
Verify the report against a hand-built fixture first. Two receipts, one accepted, 100 tokens total, ratio must be 100.0. If your fixture disagrees with the function, the function is wrong — not the fixture.
What do you do with the ratio? Compare it honestly.
| Situation | Free backend wins? | Why |
|---|---|---|
| Prompt tuning, exploratory work | Yes | Cost of a bad call is zero; the ledger tells you the retry tax |
| Batch job with a hard deadline | Rarely | Error-rate breaker trips more often than you'd like |
| Pipeline with strict schema output | Depends | If tokens-per-accepted is high, free is not free |
| Anything user-facing and latency-bound | No | You're measuring the wrong axis entirely |
That third row is the whole article. A free endpoint at 3,000 tokens per accepted output can lose to a paid one at 400.
The estimator is not a tokenizer. len(text) // 4 is a blunt instrument and will mislead you on non-English text, code, or anything with heavy punctuation. Treat estimated numbers as directional only.
Provider usage blocks are not standardized. If yours is missing fields, your ledger will silently under-count — add an assertion, not a comment.
Free availability is not an SLA. Both claims I mentioned earlier are claims, not guarantees; the gate exists precisely because the ground can move under you.
And the acceptance check is the weakest link. json.loads is easy. A semantic grader is not. If you can't define "accepted" mechanically, you can't compute the ratio — build that first.
Skip this approach if you're in a regulated path needing audit-grade billing, if your latency budget is under a second end-to-end, or if your volume is high enough that the free tier is a rounding error anyway.
Five stages, maybe ninety minutes, and you end up with a number that survives contact with a design review: tokens per accepted output, split by backend, with the errors counted instead of hidden.
If standing up the endpoint is the part that keeps holding you back, MonkeyCode's free server option is one way to get a live target for Stage 2 — verify the current terms yourself and let the ledger do the arguing.