{"slug": "counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases", "title": "Counting OpenAI Tokens per Request in FastAPI — Without Touching My Use Cases", "summary": "A developer building evorag, a Python retrieval service using OpenAI's text-embedding-3-small with PostgreSQL and pgvector, added per-request token counting and JSON Lines logging to a FastAPI app without leaking observability concerns into its Clean Architecture domain or use-case layers. The approach uses contextvars to carry a request ID and token counts as a side channel between the OpenAI adapter and HTTP middleware, so the use case never learns about billing metrics.", "body_md": "I'm rebuilding **evorag**, a Python retrieval service: it chunks documents, embeds them with OpenAI's `text-embedding-3-small`, stores vectors in PostgreSQL + pgvector, and returns the most relevant chunks for a query. Answer generation is out of scope. The plan is to connect it to my NestJS [rag-service](https://github.com/YusufOztoprak/rag-service), with evorag handling retrieval and rag-service handling the API and generation, but today the two run independently.\n\nThis week's goal was observability. For every request I wanted to know three things:\n\nThe catch: evorag follows Clean Architecture, and I didn't want any of this to leak into the domain or use-case layers.\n\nThe project is laid out like this:\n\n```\nsrc/evorag/\n  domain/          Chunk entity (plain frozen dataclass)\n  ingestion/       IngestDocument use case + ports\n  retrieval/       SearchChunks use case + ports\n  shared/          EmbeddingProvider port\n  observability/   JSON formatter, per-request context\n  adapters/        FastAPI, OpenAI, pgvector\n  main.py          composition root\n```\n\nSource code dependencies point inward only. A use case knows an `EmbeddingProvider` *protocol*, not OpenAI:\n\n``` python\nclass EmbeddingProvider(Protocol):\n    async def embed(self, texts: list[str]) -> list[list[float]]: ...\n```\n\nHere's the problem. The token count is only known inside the OpenAI adapter (`response.usage.total_tokens`). The per-request log line is written in the HTTP middleware. Between them sits the use case.\n\nThe naive fix is to make `embed()` return `(vectors, tokens)` and thread the number up through `IngestDocument.execute()` back to the route. That would change the port, the use case, every fake in my tests, and the use case would suddenly \"know\" about billing metrics. That's exactly what the dependency rule is supposed to prevent.\n\nSo I needed a side channel: something request-scoped that the adapter can write to and the middleware can read from, with the use case in the middle never noticing.\n\nFirst, logs. I stuck to the standard library and wrote a small formatter that emits [JSON Lines](https://jsonlines.org/):\n\n``` python\nclass JsonFormatter(logging.Formatter):\n    def format(self, record: logging.LogRecord) -> str:\n        payload = {\n            \"ts\": datetime.fromtimestamp(record.created, UTC).isoformat(),\n            \"level\": record.levelname,\n            \"logger\": record.name,\n            \"event\": record.getMessage(),\n            \"request_id\": get_request_id(),\n        }\n        payload.update(getattr(record, \"fields\", {}))\n        return json.dumps(payload)\n```\n\nThe message is an event name (`request.completed`, `embedding.created`), and structured data goes in via `extra={\"fields\": {...}}`. Any log tool can then filter by `event` or aggregate `latency_ms` without regex.\n\n`contextvars`\n`contextvars` is the async-safe way to hold \"current request\" state in Python. Each request gets its own value, even when hundreds are interleaved on one event loop.\n\n```\n_request_id: ContextVar[str | None] = ContextVar(\"request_id\", default=None)\n\ndef set_request_id(request_id: str) -> None:\n    _request_id.set(request_id)\n\ndef get_request_id() -> str | None:\n    return _request_id.get()\n```\n\nThe middleware reuses an incoming `X-Request-ID` header if there is one (so once rag-service calls evorag, the same ID can follow a request across both services), otherwise it generates one, and it echoes the ID back in the response:\n\n``` php\nasync def log_requests(request: Request, call_next) -> Response:\n    request_id = request.headers.get(\"x-request-id\") or uuid.uuid4().hex\n    set_request_id(request_id)\n    ...\n    response = await call_next(request)\n    response.headers[\"x-request-id\"] = request_id\n```\n\nBecause the formatter calls `get_request_id()` itself, every log line from anywhere in the app gets the ID automatically. No logger needs to be passed around.\n\n`ContextVar[int]`\nThe obvious approach for tokens would be the same pattern as the request ID: a `ContextVar[int]`, reset to 0 in the middleware, incremented in the adapter with `.set(current + tokens)`, read back in the middleware after `call_next`.\n\nThat looks right, but it doesn't work. A minimal reproduction:\n\n```\ncounter: ContextVar[int] = ContextVar(\"c\", default=0)\nholder: ContextVar[dict | None] = ContextVar(\"h\", default=None)\n\n@app.middleware(\"http\")\nasync def mw(request, call_next):\n    counter.set(0)\n    h = {\"n\": 0}\n    holder.set(h)\n    response = await call_next(request)\n    print(counter.get(), h[\"n\"])\n    return response\n\n@app.get(\"/x\")\nasync def x():\n    counter.set(counter.get() + 42)\n    holder.get()[\"n\"] += 42\n    return {}\n```\n\nOutput:\n\n```\n0 42\n```\n\nThe `int` version reads back **0**. The object version reads back **42**.\n\nWhy: `@app.middleware(\"http\")` is Starlette's `BaseHTTPMiddleware`, and `call_next` runs the rest of the app in a separate task. A new task gets a **copy** of the current context. The endpoint can *read* everything the middleware set (that's why the request ID works), but a `.set()` inside the endpoint only changes the copy. The middleware never sees it.\n\nWhat *does* cross the boundary is a mutable object. The copied context still points to the same object, so mutating it in place is visible on both sides.\n\nSo the fix is to store a small holder object in the ContextVar and mutate it, never re-set it:\n\n```\n@dataclass\nclass RequestMetrics:\n    embedding_tokens: int = 0\n\n_metrics: ContextVar[RequestMetrics | None] = ContextVar(\"metrics\", default=None)\n\ndef start_metrics() -> RequestMetrics:\n    metrics = RequestMetrics()\n    _metrics.set(metrics)\n    return metrics\n\ndef add_embedding_tokens(tokens: int) -> None:\n    metrics = _metrics.get()\n    if metrics is None:   # e.g. called outside a request (scripts, tests)\n        return\n    metrics.embedding_tokens += tokens\n```\n\nThe middleware creates the holder before `call_next` and keeps a reference to it:\n\n```\n    metrics = start_metrics()\n    started = time.perf_counter()\n    response = await call_next(request)\n    latency_ms = round((time.perf_counter() - started) * 1000, 2)\n\n    logger.info(\"request.completed\", extra={\"fields\": {\n        \"method\": request.method,\n        \"path\": request.url.path,\n        \"status\": response.status_code,\n        \"latency_ms\": latency_ms,\n        \"embedding_tokens\": metrics.embedding_tokens,\n    }})\n```\n\nAnd the OpenAI adapter, the only place that knows about tokens, reports them:\n\n```\n        started = time.perf_counter()\n        response = await self._client.embeddings.create(...)\n        latency_ms = round((time.perf_counter() - started) * 1000, 2)\n\n        tokens = response.usage.total_tokens\n        add_embedding_tokens(tokens)\n        logger.info(\"embedding.created\", extra={\"fields\": {\n            \"model\": self._model,\n            \"inputs\": len(texts),\n            \"tokens\": tokens,\n            \"latency_ms\": latency_ms,\n        }})\n```\n\nThe `EmbeddingProvider` port didn't change. `IngestDocument` and `SearchChunks` didn't change. The test fakes didn't change. Observability lives entirely in the outer ring: the HTTP adapter and the OpenAI adapter, both talking to a small `observability` module.\n\nOne `POST /documents` now produces two lines (some fields trimmed):\n\n```\n{\"event\": \"embedding.created\", \"request_id\": \"2a2c1c38...\", \"model\": \"text-embedding-3-small\", \"inputs\": 1, \"tokens\": 18, \"latency_ms\": 390.19}\n{\"event\": \"request.completed\", \"request_id\": \"2a2c1c38...\", \"method\": \"POST\", \"path\": \"/documents\", \"status\": 201, \"latency_ms\": 439.52, \"embedding_tokens\": 18}\n```\n\nMeasuring the OpenAI call separately from the whole request turned out to be the most useful part. In my early measurements the embedding call took **89–97% of total request time**, while evorag's own work (chunking, DB write, serialization) stayed around **50 ms**.\n\nThat changes what I'll work on next. Optimizing my own code is pointless right now; the real levers are around the external call: timeouts, retries, and moving ingestion off the request path (a RabbitMQ consumer is on the roadmap).\n\nThe observability tests use a tiny FastAPI app with the real middleware and a fake endpoint, so no database or API key is needed:\n\n``` python\ndef make_app():\n    app = FastAPI()\n    app.middleware(\"http\")(log_requests)\n\n    @app.get(\"/fake\")\n    async def fake():\n        add_embedding_tokens(30)\n        add_embedding_tokens(12)\n        return {\"ok\": \"yes\"}\n\n    return app\n\ndef test_middleware_sums_embedding_tokens(caplog):\n    caplog.set_level(logging.INFO, logger=\"evorag\")\n    TestClient(make_app()).get(\"/fake\")\n\n    [record] = [r for r in caplog.records if r.getMessage() == \"request.completed\"]\n    assert record.fields[\"embedding_tokens\"] == 42\n```\n\nThis test is also a regression guard for the trap above: if someone \"simplifies\" `RequestMetrics` back into a plain `ContextVar[int]`, it fails with `0 != 42`.\n\nOther tests cover generating a request ID, reusing an incoming `X-Request-ID`, and the formatter with and without extra fields.\n\n`latency_ms` field told me more about where to spend effort than any amount of profiling my own code would have.\nNext up for evorag: timeouts and retries around the OpenAI calls, a RabbitMQ consumer for asynchronous ingestion, and a small evaluation set to calibrate chunk size and the similarity threshold properly.\n\nThe code is on GitHub: [YusufOztoprak/evorag](https://github.com/YusufOztoprak/evorag)\n\n*Originally published on [my blog](https://yusufoztoprak.hashnode.dev/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases).*", "url": "https://wpnews.pro/news/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases", "canonical_source": "https://dev.to/yusufoztoprak/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases-4pce", "published_at": "2026-10-17 13:27:37+00:00", "updated_at": "2026-10-17 13:34:14.506426+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["OpenAI", "FastAPI", "PostgreSQL", "pgvector", "evorag", "NestJS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases", "markdown": "https://wpnews.pro/news/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases.md", "text": "https://wpnews.pro/news/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases.txt", "jsonld": "https://wpnews.pro/news/counting-openai-tokens-per-request-in-fastapi-without-touching-my-use-cases.jsonld"}}