Building an LLM API Gateway in Node.js

An engineer demonstrates how to build an LLM API gateway in Node.js using LiteLLM, addressing the governance problem of direct API calls to providers like OpenAI and Anthropic. The gateway centralizes routing, cost tracking, and rate limiting, with options including LiteLLM, Portkey, and Cloudflare AI Gateway. The walkthrough uses LiteLLM as a proxy with an OpenAI-compatible API, requiring minimal changes to existing Node.js code.

Your LLM bill just tripled and nobody on the team can say why. Sound familiar? If you're calling OpenAI or Anthropic directly from every service in your stack, you don't have an integration problem, you have a governance problem. This is where an LLM API gateway earns its keep, and setting one up in Node.js takes less time than debugging your next surprise invoice. Here's what actually happens without a gateway. Every service that calls an LLM hardcodes its own API key, its own retry logic, its own model name. Then one day you want to switch from GPT to Claude for cost reasons, and you're grepping through six repos. Or a junior dev's script goes into an infinite retry loop against a rate limited endpoint and your bill spikes overnight. Or, worse, you have zero visibility into which team or feature is actually driving spend. A gateway sits between your app code and the model providers. Every request flows through one chokepoint, which means you get centralized routing, centralized cost tracking, centralized rate limiting, and a single place to swap providers without touching application code. Enterprise model API spend has already climbed past 8.4 billion dollars and keeps climbing, and most of that spend is going through zero centralized control. That's not a scale problem you'll deal with later, it's a scale problem you're already in. The gotcha nobody warns you about: adding a gateway late in a project is way more painful than adding it on day one, because by then every service has its own bespoke calling convention baked in. If you're greenfield, wire this up before you write your first prompt call. You've got three solid options here and picking the wrong one for your situation is an easy footgun. | Gateway | Best for | Deployment | Standout feature | |---|---|---|---| | LiteLLM | Teams who want full control, self hosted | Your own infra Docker, VM, k8s | Massive provider support, drop in OpenAI compatible API | | Portkey | Teams who want guardrails without building them | Cloud or self hosted | Fully open source under Apache 2.0, over 1600 models across 250 plus providers, 40 plus prebuilt guardrails | | Cloudflare AI Gateway | Teams already on Cloudflare's edge | Cloudflare's network | Near zero added latency since it's already at the edge, built in analytics | If you're the kind of team that wants to own every knob and doesn't mind running another service, reach for LiteLLM. If you want guardrails PII detection, prompt injection checks, content moderation without writing that logic yourself, Portkey's prebuilt set is hard to beat, especially now that it's fully open source. If your traffic already runs through Cloudflare, the AI Gateway is the path of least resistance since there's no new infra to stand up. For this walkthrough I'm going with LiteLLM, since it gives you the clearest picture of what's actually happening under the hood, and that understanding transfers even if you end up choosing Portkey or Cloudflare later. LiteLLM runs as a proxy server with an OpenAI compatible API, so your Node code barely changes, you just point it at a different base URL. litellm-config.yaml model list: - model name: gpt-4o litellm params: model: openai/gpt-4o api key: os.environ/OPENAI API KEY - model name: claude-sonnet litellm params: model: anthropic/claude-sonnet-4-5 api key: os.environ/ANTHROPIC API KEY router settings: routing strategy: least-busy fallbacks: - gpt-4o: claude-sonnet general settings: master key: os.environ/LITELLM MASTER KEY spin up the proxy locally Docker keeps this reproducible across environments docker run -d \ -v $ pwd /litellm-config.yaml:/app/config.yaml \ -e OPENAI API KEY=$OPENAI API KEY \ -e ANTHROPIC API KEY=$ANTHROPIC API KEY \ -e LITELLM MASTER KEY=$LITELLM MASTER KEY \ -p 4000:4000 \ ghcr.io/berriai/litellm:main-latest \ --config /app/config.yaml Now your Node.js code calls the proxy instead of the provider directly. If you're already using the OpenAI SDK, this is a one line change to the base URL: python // lib/llm-client.ts import OpenAI from "openai"; const gateway = new OpenAI { apiKey: process.env.LITELLM MASTER KEY, baseURL: "http://localhost:4000/v1", } ; export async function askModel prompt: string { const response = await gateway.chat.completions.create { model: "gpt-4o", messages: { role: "user", content: prompt } , } ; return response.choices 0 .message.content; } That's it. Your application code has no idea it's talking to a proxy instead of OpenAI directly, and swapping providers later is a config change, not a code change. Here's a pattern that bit me early on. If your app answers common questions, "what's your refund policy," "how do I reset my password," you're paying full price to regenerate nearly identical answers over and over. Semantic caching fixes this by checking if a semantically similar prompt has already been answered, instead of requiring an exact string match. Semantic caching alone can cut costs 30 to 50 percent for repetitive workloads, and routing strategies combined with caching cut costs 40 to 70 percent overall. Stack a few optimization techniques together and you're looking at 50 to 80 percent reduction on production workloads. That's not a marginal tweak, that's the difference between a sustainable AI feature and one finance keeps asking you to justify. I cover more of this cost math in my LLM inference cost optimization https://mudassirkhan.me/blog/llm-inference-cost-optimization writeup if you want the full breakdown. python // lib/semantic-cache.ts import { createClient } from "redis"; import OpenAI from "openai"; const redis = createClient { url: process.env.REDIS URL } ; await redis.connect ; const embeddings = new OpenAI { apiKey: process.env.OPENAI API KEY } ; const SIMILARITY THRESHOLD = 0.95; async function getEmbedding text: string : Promise<number { const res = await embeddings.embeddings.create { model: "text-embedding-3-small", input: text, } ; return res.data 0 .embedding; } function cosineSimilarity a: number , b: number : number { const dot = a.reduce sum, val, i = sum + val b i , 0 ; const magA = Math.sqrt a.reduce sum, val = sum + val val, 0 ; const magB = Math.sqrt b.reduce sum, val = sum + val val, 0 ; return dot / magA magB ; } export async function cachedAsk prompt: string, askFn: p: string = Promise<string { const promptEmbedding = await getEmbedding prompt ; const cachedKeys = await redis.keys "semcache: " ; for const key of cachedKeys { const entry = JSON.parse await redis.get key ?? "{}" ; if entry.embedding continue; if cosineSimilarity promptEmbedding, entry.embedding = SIMILARITY THRESHOLD { return entry.response; // cache hit, skip the model call entirely } } const response = await askFn prompt ; await redis.set semcache:${Date.now } , JSON.stringify { embedding: promptEmbedding, response } , { EX: 86400 } ; return response; } Scanning all cached keys with a linear similarity check is fine at small scale, but reach for a vector database Pinecone, Qdrant, or pgvector once you've got more than a few thousand cached entries. The pattern stays identical, only the lookup mechanism changes. Providers go down. Rate limits get hit. Latency spikes happen at the worst possible time. If your gateway doesn't handle this gracefully, one flaky provider takes your whole feature down with it. A circuit breaker watches for repeated failures and, once a threshold is crossed, "opens" and stops sending requests to the failing provider for a cooldown period, instead immediately failing over to a backup. This is exactly what LiteLLM's fallback config handles at the proxy level, but it's worth understanding the pattern so you can reason about it, and so you can add the same protection at the application layer for anything that bypasses the gateway. python // lib/circuit-breaker.ts import CircuitBreaker from "opossum"; import { gateway } from "./llm-client"; async function callPrimary prompt: string { const res = await gateway.chat.completions.create { model: "gpt-4o", messages: { role: "user", content: prompt } , } ; return res.choices 0 .message.content; } const options = { timeout: 8000, // fail fast instead of hanging errorThresholdPercentage: 50, resetTimeout: 30000, // try again after 30 seconds }; const breaker = new CircuitBreaker callPrimary, options ; breaker.fallback async prompt: string = { // circuit is open, reroute to the fallback model instead of failing the request const res = await gateway.chat.completions.create { model: "claude-sonnet", messages: { role: "user", content: prompt } , } ; return res.choices 0 .message.content; } ; breaker.on "open", = console.warn "Circuit opened, primary model degraded" ; breaker.on "close", = console.info "Circuit closed, primary model recovered" ; export async function resilientAsk prompt: string { return breaker.fire prompt ; } Bifrost, one of the newer gateway options, adds just 11 microseconds of overhead per request at 5000 requests per second, worth knowing if you're comparing raw proxy performance, since Python based gateways commonly add hundreds of microseconds by comparison. Node and Go based proxies tend to sit closer to the low end of that range. Routing and caching stop runaway costs at the model layer. But you also want a hard ceiling that stops a single user, tenant, or buggy script from blowing your monthly budget before anyone notices. This is where per user quotas earn their place, right at the middleware layer, before a request ever reaches the gateway. js // middleware/cost-cap.ts import { Request, Response, NextFunction } from "express"; import { createClient } from "redis"; const redis = createClient { url: process.env.REDIS URL } ; await redis.connect ; const DAILY TOKEN BUDGET = 100 000; export async function costCapMiddleware req: Request, res: Response, next: NextFunction { const userId = req.headers "x-user-id" as string; if userId return res.status 401 .json { error: "Missing user identifier" } ; const key = quota:${userId}:${new Date .toISOString .slice 0, 10 } ; const used = parseInt await redis.get key ?? "0", 10 ; if used = DAILY TOKEN BUDGET { return res.status 429 .json { error: "Daily token budget exceeded", resetAt: "midnight UTC", } ; } res.locals.recordUsage = async tokensUsed: number = { await redis.incrBy key, tokensUsed ; await redis.expire key, 86400 ; }; next ; } Wire this in ahead of your route handler, and have the handler call res.locals.recordUsage tokensUsed once it gets the token count back from the model response. Combine this with the gateway's own governance layer this is where Portkey's guardrails or LiteLLM's budget settings do double duty and you've got protection at both the edge and the middleware. I go deeper into the governance side of this in AI governance in production https://mudassirkhan.me/blog/ai-governance-llm-production if quotas alone don't cover your compliance requirements. What is an LLM API gateway? It's a proxy layer that sits between your application and LLM providers OpenAI, Anthropic, and others , handling routing, caching, rate limiting, and cost tracking in one centralized place instead of scattering that logic across every service that calls a model. How do I reduce LLM API costs? Combine semantic caching cuts 30 to 50 percent on repetitive workloads , smart routing between cheaper and more capable models depending on task complexity, and hard per user quotas. Stacked together, these techniques commonly deliver 50 to 80 percent cost reduction versus calling providers directly with no controls. What is the best LLM gateway for Node.js? LiteLLM if you want a self hosted, fully controllable proxy with an OpenAI compatible API. Portkey if you want prebuilt guardrails without building them yourself. Cloudflare AI Gateway if your traffic already runs through Cloudflare's edge and you want the lowest added latency. If you want a deeper look at LLM cost optimization, I cover it in more detail on my site. If you want this wired up on your own site end to end, that is exactly the kind of work I take on. Drop a comment if your setup looks different, curious what routing or caching strategies people are actually running in production.