{"slug": "llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant", "title": "LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas", "summary": "A developer at Loop Origin documented an LLM gateway architecture for B2B SaaS that centralizes tenant identification, token budget reservation, exact and semantic caching, task-class routing to the cheapest adequate model, and post-call usage metering. The design uses an atomic Redis/Valkey Lua script to reserve estimated token cost before each call and settle actual usage afterward, so spend is attributed to the correct tenant's ledger. The writeup follows a scenario in which one starter-plan tenant ran a nightly script sending the same 40-page contract through a summarize endpoint 600 times, all on the most expensive default model.", "body_md": "The finance lead forwards you the model provider invoice with one line highlighted. Spend tripled in a month. Revenue did not. When you dig in, a single tenant on the starter plan is running a nightly script that sends the same 40-page contract through your \"summarize\" endpoint 600 times, every request goes to the most expensive model because that was the default in the SDK call, and nothing in your stack knew which tenant any of those tokens belonged to.\n\nI wrote this with help from AI tools and checked every config, code path and claim myself. I run Loop Origin, where we build [custom web applications](https://looporigin.com/services/custom-web-application-development?utm_source=devto&utm_medium=series&utm_campaign=llm-gateway-finops-architecture-b2b-saas-token-quotas) for B2B SaaS teams, and this is the final part of the series: the gateway layer that turns LLM calls from an unbounded cost into a metered, budgeted product feature.\n\n**📚 Series Roadmap — Production AI & SaaS Systems Engineering**\n\n**Part 1:** [Building Production MCP Servers & Tool-Calling AI Agents: Sandboxing, Schemas & Approval Gates](https://dev.to/muhammadalimughal/building-production-mcp-servers-tool-calling-ai-agents-sandboxing-schemas-approval-gates-7nb)\n**Part 2:** [Zero-Downtime Database Migrations in Laravel & Node.js SaaS: Expand-Contract & Lock Timeouts](https://dev.to/muhammadalimughal/zero-downtime-database-migrations-in-laravel-nodejs-saas-expand-contract-lock-timeouts-6lp)\n**Part 3:** [Multi-Tier Caching & Stampede Protection with Redis/Valkey: Singleflight Locks & SWR](https://dev.to/muhammadalimughal/multi-tier-caching-stampede-protection-with-redisvalkey-singleflight-locks-swr-34hk)\n**Part 4:** [SaaS Usage-Based Billing & Metering Architecture: Event Ingestion, Rollups & Stripe Invoicing](https://dev.to/muhammadalimughal/saas-usage-based-billing-metering-architecture-event-ingestion-rollups-stripe-invoicing-m3c)\n**Part 5:** [Production Observability on a Budget: OpenTelemetry Traces, Structured Logs & Tail Sampling](https://dev.to/muhammadalimughal/production-observability-on-a-budget-opentelemetry-traces-structured-logs-tail-sampling-360a)\n**Part 6 (This Post):** LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas\n\n**Direct answer:** An LLM gateway is one internal service that every model call passes through. It identifies the tenant and feature, checks a token budget before the call, serves exact or semantic cache hits, routes each task to the cheapest model that meets its quality bar, and records actual token usage afterwards so costs land on the right tenant's ledger.\n\nIn most SaaS codebases, the first AI feature calls the provider SDK directly from a controller. The second feature copies that code. By the fifth, you have five places that pick a model, five retry policies, no shared cache, and API keys in three services. Nobody can answer the two questions finance will ask: *which tenant spent this* and *what would it cost on a cheaper model*.\n\nA gateway centralizes four decisions that should never be made inside product code:\n\n```\n Laravel API ──┐\n Node workers ─┼─► LLM Gateway ──► quota check (Redis) ──► cache (exact → semantic)\n MCP agents  ──┘       │                                        │ miss\n                       │                                        ▼\n                       │                             router: task class → model tier\n                       │                                        │\n                       └──── usage events (Part 4 meter) ◄── provider A / B / self-hosted\n```\n\nThe gateway can be an open-source proxy you configure or a small service you write. Either way, product code sends a *task*, not a model name: `{ tenant_id, feature: \"contract_summary\", input }`.\n\nReserve an estimated token cost before the call, then settle the real number afterwards. A Lua script keeps the check-and-reserve atomic in Redis or Valkey:\n\n```\n-- KEYS[1] = budget:{tenant}:{yyyy-mm}   ARGV[1] = estimate   ARGV[2] = monthly limit\nlocal used = tonumber(redis.call('GET', KEYS[1]) or '0')\nif used + tonumber(ARGV[1]) > tonumber(ARGV[2]) then\n  return -1\nend\nredis.call('INCRBY', KEYS[1], ARGV[1])\nredis.call('EXPIRE', KEYS[1], 40 * 24 * 3600)\nreturn used + tonumber(ARGV[1])\n// gateway/budget.ts\nexport async function reserve(tenant: Tenant, estimate: number) {\n  const key = `budget:${tenant.id}:${monthKey()}`;\n  const res = await redis.eval(RESERVE_LUA, 1, key, estimate, tenant.plan.monthlyTokens);\n  if (res === -1) throw new QuotaExceeded(tenant.id, tenant.plan.name);\n}\n\nexport async function settle(tenant: Tenant, estimate: number, actual: number) {\n  await redis.incrby(`budget:${tenant.id}:${monthKey()}`, actual - estimate);\n  await meter.emit({ tenant: tenant.id, metric: \"llm_tokens\", value: actual }); // Part 4 pipeline\n}\n```\n\nEstimate input tokens with the provider's tokenizer and add the request's `max_tokens` cap for output. Overestimating slightly is fine because settlement corrects it; what matters is that the 600-request nightly loop hits a wall instead of the invoice.\n\n**Exact cache** first: hash the normalized request (model tier, system prompt version, input, temperature) and look it up. It is cheap and has zero quality risk. In my experience it catches more than teams expect, because batch jobs and retries resend identical inputs.\n\n**Semantic cache** second: embed the input, search a vector index scoped to the same tenant and feature, and return the stored answer only above a strict similarity threshold.\n\n``` js\nconst hit = await vectors.search({\n  namespace: `${tenant.id}:${feature}:${promptVersion}`,\n  vector: await embed(input),\n  topK: 1,\n});\nif (hit && hit.score >= FEATURE_THRESHOLDS[feature]) return hit.payload; // e.g. 0.95\n```\n\nThree rules keep semantic caching from becoming a data-leak or wrong-answer machine:\n\nMost SaaS AI features are not equally hard. Classify them once and map each class to a tier:\n\n| Task class | Example feature | Model tier | Fallback | Cacheable | \n|---|---|---|---|---|\n| Extraction / classification | Tag a support ticket | Small, fast model | Second small model | Exact + semantic | \n| Summarization | Contract summary | Mid-tier model | Small model with shorter output | Exact | \n| Reasoning / generation | Draft a proposal | Frontier model | Mid-tier model + warning flag | Rarely | \n| Agentic tool use (MCP) | Multi-step workflow | Frontier model with tool support | Halt and ask human | No | \n| Embeddings | Search, semantic cache | Embedding model | Queue and retry | N/A | \n\n```\n# gateway/routes.yaml\ncontract_summary:\n  primary: { provider: provider_a, model: mid-tier, max_tokens: 800 }\n  fallback: { provider: provider_b, model: small, max_tokens: 500 }\n  timeout_ms: 20000\n  plans:\n    starter: { tier_override: small }\nticket_classify:\n  primary: { provider: self_hosted, model: small }\n  fallback: { provider: provider_a, model: small }\n```\n\nThe `plans` override is where FinOps meets pricing: starter tenants get the cheaper tier, enterprise tenants get the better one, and the difference is visible on your [pricing page](https://looporigin.com/pricing?utm_source=devto&utm_medium=series&utm_campaign=llm-gateway-finops-architecture-b2b-saas-token-quotas) instead of hidden in your margin.\n\nValidate a cheaper route before switching: replay a sample of real (anonymized) inputs through both tiers and score the outputs with your own rubric. Route changes without an eval are guesses.\n\nRetries are where LLM costs quietly multiply. As Part 3 covered for caches, a retry storm turns one outage into many. In the gateway:\n\nReuse the OpenTelemetry setup from Part 5. Each gateway call becomes a span with low-cardinality attributes:\n\n```\nspan.setAttributes({\n  \"llm.feature\": feature,\n  \"llm.route\": route.name,\n  \"llm.model_tier\": route.tier,\n  \"llm.cache\": cacheResult,          // \"exact\" | \"semantic\" | \"miss\"\n  \"llm.tokens.input\": usage.input,\n  \"llm.tokens.output\": usage.output,\n  \"tenant.plan\": tenant.plan.name,   // plan, not tenant ID, to keep metrics cheap\n});\n```\n\nGenerate metrics from these spans for cost per feature, cache hit rate and fallback rate. Keep the tenant ID in the usage events and the billing ledger, not in metric labels, so dashboards stay affordable.\n\n| Metric | Direct SDK calls | Behind the gateway | \n|---|---|---|\n| Who spent it | Unknown | Tenant + feature on every call | \n| Runaway script | Runs all night | Blocked at the monthly budget | \n| Repeated inputs | Paid every time | Exact cache hit | \n| Model choice | Whatever the SDK default was | Per task class and plan | \n| Provider outage | Feature down | Fallback route + circuit breaker | \n| Billing | Absorbed in margin | Metered and invoiceable (Part 4) | \n\nStart with an existing open-source LLM proxy if it supports your providers, budgets and logging. Write your own thin service when you need tenant-aware quotas tied to your billing data, which most B2B SaaS products eventually do.\n\nA budget check in Redis and an exact cache lookup are usually a few milliseconds. Semantic cache adds an embedding call, so only enable it for features where hit rates justify it.\n\nThere is no universal number. Start strict, around 0.95 cosine similarity, log near-misses, and review them by hand before loosening it per feature.\n\nCount input tokens with the provider's tokenizer library and add the `max_tokens` cap for output. Settle with the real usage numbers the provider returns.\n\nYes, if your metering is accurate. Emit settled token usage into the same pipeline as your other usage events and price it per plan, with clear limits shown in the product.\n\nAgents make many calls per task, so they need the gateway most. Give each agent run its own budget reservation and stop the run, with a human approval gate, when it exceeds it.\n\nThat closes Production AI & SaaS Systems Engineering. Across six parts we sandboxed MCP agents, migrated databases without downtime, protected caches from stampedes, metered usage for billing, made traces affordable and, here, put a budget and a router in front of every model call. The common thread: put one deliberate control point in front of every expensive or risky operation, measure what passes through it, and let product code ask for outcomes instead of hard-coding infrastructure choices. Questions about your own setup are welcome in the comments.", "url": "https://wpnews.pro/news/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant", "canonical_source": "https://dev.to/muhammadalimughal/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant-quotas-17l4", "published_at": "2026-11-02 16:17:19+00:00", "updated_at": "2026-11-02 16:26:55.559519+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops", "ai-tools"], "entities": ["Loop Origin", "Redis", "Valkey", "Laravel", "Node.js", "Stripe"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant", "markdown": "https://wpnews.pro/news/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant.md", "text": "https://wpnews.pro/news/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant.txt", "jsonld": "https://wpnews.pro/news/llm-gateway-finops-architecture-for-b2b-saas-semantic-caching-routing-tenant.jsonld"}}