Building an LLM API Gateway in Node.js

Your LLM bill just tripled and nobody on the team can say why. Sound familiar? If you're calling OpenAI or Anthropic directly from every service in your stack, you don't have an integration problem, you have a governance problem. This is where an LLM API gateway earns its keep, and setting one up in Node.js takes less time than debugging your next surprise invoice.

Here's what actually happens without a gateway. Every service that calls an LLM hardcodes its own API key, its own retry logic, its own model name. Then one day you want to switch from GPT to Claude for cost reasons, and you're grepping through six repos. Or a junior dev's script goes into an infinite retry loop against a rate limited endpoint and your bill spikes overnight. Or, worse, you have zero visibility into which team or feature is actually driving spend.

A gateway sits between your app code and the model providers. Every request flows through one chokepoint, which means you get centralized routing, centralized cost tracking, centralized rate limiting, and a single place to swap providers without touching application code. Enterprise model API spend has already climbed past 8.4 billion dollars and keeps climbing, and most of that spend is going through zero centralized control. That's not a scale problem you'll deal with later, it's a scale problem you're already in.

The gotcha nobody warns you about: adding a gateway late in a project is way more painful than adding it on day one, because by then every service has its own bespoke calling convention baked in. If you're greenfield, wire this up before you write your first prompt call.

You've got three solid options here and picking the wrong one for your situation is an easy footgun.

If you're the kind of team that wants to own every knob and doesn't mind running another service, reach for LiteLLM. If you want guardrails (PII detection, prompt injection checks, content moderation) without writing that logic yourself, Portkey's prebuilt set is hard to beat, especially now that it's fully open source. If your traffic already runs through Cloudflare, the AI Gateway is the path of least resistance since there's no new infra to stand up.

For this walkthrough I'm going with LiteLLM, since it gives you the clearest picture of what's actually happening under the hood, and that understanding transfers even if you end up choosing Portkey or Cloudflare later.

LiteLLM runs as a proxy server with an OpenAI compatible API, so your Node code barely changes, you just point it at a different base URL.

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: least-busy
  fallbacks:
    - gpt-4o: [claude-sonnet]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
docker run -d \
  -v $(pwd)/litellm-config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -e LITELLM_MASTER_KEY=$LITELLM_MASTER_KEY \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Now your Node.js code calls the proxy instead of the provider directly. If you're already using the OpenAI SDK, this is a one line change to the base URL:

// lib/llm-client.ts
import OpenAI from "openai";

const gateway = new OpenAI({
  apiKey: process.env.LITELLM_MASTER_KEY,
  baseURL: "http://localhost:4000/v1",
});

export async function askModel(prompt: string) {
  const response = await gateway.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  return response.choices[0].message.content;
}

That's it. Your application code has no idea it's talking to a proxy instead of OpenAI directly, and swapping providers later is a config change, not a code change.

Here's a pattern that bit me early on. If your app answers common questions, "what's your refund policy," "how do I reset my password," you're paying full price to regenerate nearly identical answers over and over. Semantic caching fixes this by checking if a semantically similar prompt has already been answered, instead of requiring an exact string match.

Semantic caching alone can cut costs 30 to 50 percent for repetitive workloads, and routing strategies combined with caching cut costs 40 to 70 percent overall. Stack a few optimization techniques together and you're looking at 50 to 80 percent reduction on production workloads. That's not a marginal tweak, that's the difference between a sustainable AI feature and one finance keeps asking you to justify. I cover more of this cost math in my LLM inference cost optimization writeup if you want the full breakdown.

// lib/semantic-cache.ts
import { createClient } from "redis";
import OpenAI from "openai";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const embeddings = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const SIMILARITY_THRESHOLD = 0.95;

async function getEmbedding(text: string): Promise<number[]> {
  const res = await embeddings.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return res.data[0].embedding;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

export async function cachedAsk(prompt: string, askFn: (p: string) => Promise<string>) {
  const promptEmbedding = await getEmbedding(prompt);
  const cachedKeys = await redis.keys("semcache:*");

  for (const key of cachedKeys) {
    const entry = JSON.parse((await redis.get(key)) ?? "{}");
    if (!entry.embedding) continue;
    if (cosineSimilarity(promptEmbedding, entry.embedding) >= SIMILARITY_THRESHOLD) {
      return entry.response; // cache hit, skip the model call entirely
    }
  }

  const response = await askFn(prompt);
  await redis.set(
    `semcache:${Date.now()}`,
    JSON.stringify({ embedding: promptEmbedding, response }),
    { EX: 86400 }
  );
  return response;
}

Scanning all cached keys with a linear similarity check is fine at small scale, but reach for a vector database (Pinecone, Qdrant, or pgvector) once you've got more than a few thousand cached entries. The pattern stays identical, only the lookup mechanism changes.

Providers go down. Rate limits get hit. Latency spikes happen at the worst possible time. If your gateway doesn't handle this gracefully, one flaky provider takes your whole feature down with it.

A circuit breaker watches for repeated failures and, once a threshold is crossed, "opens" and stops sending requests to the failing provider for a cooldown period, instead immediately failing over to a backup. This is exactly what LiteLLM's fallback config handles at the proxy level, but it's worth understanding the pattern so you can reason about it, and so you can add the same protection at the application layer for anything that bypasses the gateway.

// lib/circuit-breaker.ts
import CircuitBreaker from "opossum";
import { gateway } from "./llm-client";

async function callPrimary(prompt: string) {
  const res = await gateway.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  return res.choices[0].message.content;
}

const options = {
  timeout: 8000,           // fail fast instead of hanging
  errorThresholdPercentage: 50,
  resetTimeout: 30000,     // try again after 30 seconds
};

const breaker = new CircuitBreaker(callPrimary, options);

breaker.fallback(async (prompt: string) => {
  // circuit is open, reroute to the fallback model instead of failing the request
  const res = await gateway.chat.completions.create({
    model: "claude-sonnet",
    messages: [{ role: "user", content: prompt }],
  });
  return res.choices[0].message.content;
});

breaker.on("open", () => console.warn("Circuit opened, primary model degraded"));
breaker.on("close", () => console.info("Circuit closed, primary model recovered"));

export async function resilientAsk(prompt: string) {
  return breaker.fire(prompt);
}

Bifrost, one of the newer gateway options, adds just 11 microseconds of overhead per request at 5000 requests per second, worth knowing if you're comparing raw proxy performance, since Python based gateways commonly add hundreds of microseconds by comparison. Node and Go based proxies tend to sit closer to the low end of that range.

Routing and caching stop runaway costs at the model layer. But you also want a hard ceiling that stops a single user, tenant, or buggy script from blowing your monthly budget before anyone notices. This is where per user quotas earn their place, right at the middleware layer, before a request ever reaches the gateway.

// middleware/cost-cap.ts
import { Request, Response, NextFunction } from "express";
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const DAILY_TOKEN_BUDGET = 100_000;

export async function costCapMiddleware(req: Request, res: Response, next: NextFunction) {
  const userId = req.headers["x-user-id"] as string;
  if (!userId) return res.status(401).json({ error: "Missing user identifier" });

  const key = `quota:${userId}:${new Date().toISOString().slice(0, 10)}`;
  const used = parseInt((await redis.get(key)) ?? "0", 10);

  if (used >= DAILY_TOKEN_BUDGET) {
    return res.status(429).json({
      error: "Daily token budget exceeded",
      resetAt: "midnight UTC",
    });
  }

  res.locals.recordUsage = async (tokensUsed: number) => {
    await redis.incrBy(key, tokensUsed);
    await redis.expire(key, 86400);
  };

  next();
}

Wire this in ahead of your route handler, and have the handler call res.locals.recordUsage(tokensUsed)

once it gets the token count back from the model response. Combine this with the gateway's own governance layer (this is where Portkey's guardrails or LiteLLM's budget settings do double duty) and you've got protection at both the edge and the middleware. I go deeper into the governance side of this in AI governance in production if quotas alone don't cover your compliance requirements.

What is an LLM API gateway?

It's a proxy layer that sits between your application and LLM providers (OpenAI, Anthropic, and others), handling routing, caching, rate limiting, and cost tracking in one centralized place instead of scattering that logic across every service that calls a model.

How do I reduce LLM API costs?

Combine semantic caching (cuts 30 to 50 percent on repetitive workloads), smart routing between cheaper and more capable models depending on task complexity, and hard per user quotas. Stacked together, these techniques commonly deliver 50 to 80 percent cost reduction versus calling providers directly with no controls.

What is the best LLM gateway for Node.js?

LiteLLM if you want a self hosted, fully controllable proxy with an OpenAI compatible API. Portkey if you want prebuilt guardrails without building them yourself. Cloudflare AI Gateway if your traffic already runs through Cloudflare's edge and you want the lowest added latency.

If you want a deeper look at LLM cost optimization, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different, curious what routing or caching strategies people are actually running in production.