# API Gateway HTTP API + Lambda Function URLs: Building a Prompt‑Caching LLM Proxy, Explained Simply

> Source: <https://dev.to/dineshgowtham/api-gateway-http-api-lambda-function-urls-building-a-prompt-caching-llm-proxy-explained-simply-1m5>
> Published: 2026-10-12 09:36:20+00:00

LLM‑powered features like code completion are often wrapped in heavyweight Lambda functions that add latency and cost. By letting API Gateway’s HTTP API route directly to a lightweight Function URL and adding a tiny in‑process cache, you get sub‑100 ms responses for repeated prompts. This post shows exactly how to wire it together with Node.js 22’s native fetch and diagnostics_channel.

When a user asks an LLM (large language model) the same question twice, the model does the same heavy computation both times. That work costs money and adds time. A **prompt cache** stores the model’s answer for a given prompt so the second request can be answered instantly, without contacting the model again.

**In plain English:** Think of a cache like a sticky note you put on your desk. The first time you look up a phone number you write it down; the next time you need it, you just glance at the note instead of dialing the operator again.

Why keep the cache **inside** the Lambda instead of an external store?

The trade‑off is that the cache disappears when the Lambda container is recycled, but for high‑frequency prompts that’s fine; the next cold start will rebuild the cache quickly.

```
// A very small in‑process cache.
// Keys are SHA‑256 hashes of the prompt string.
const cache = new Map<string, string>();
```

**Key takeaway:** Caching at the function level can shave tens of milliseconds off every repeat request, and you don’t need a separate service to make it happen.

API Gateway offers two flavors: **REST API (v1)** and **HTTP API (v2)**. The latter is lighter, cheaper, and supports **Function URLs** – a direct HTTPS endpoint that invokes a Lambda without the extra integration layer.

Why use a Function URL instead of the classic “Lambda integration”?

Below is a minimal script that creates an HTTP API, a route `/prompt`, and ties it to a Lambda Function URL. It also shows the binary‑support flag that prevents the 6 KB body‑drop bug.

``` js
import {
  ApiGatewayV2Client,
  CreateApiCommand,
  CreateIntegrationCommand,
  CreateRouteCommand,
  UpdateApiCommand,
} from "@aws-sdk/client-api-gateway";

// Replace with your region and Lambda ARN
const REGION = "us-east-1";
const LAMBDA_URL = "https://abcde12345.lambda-url.us-east-1.on.aws";

const client = new ApiGatewayV2Client({ region: REGION });

async function createHttpApi() {
  // 1️⃣ Create the HTTP API
  const apiResult = await client.send(
    new CreateApiCommand({
      Name: "PromptCacheAPI",
      ProtocolType: "HTTP",
      // Enable binary support so bodies >6 KB are not stripped
      // (the list of supported media types must include application/json)
      BinaryMediaTypes: ["application/json"],
    })
  );

  const apiId = apiResult.ApiId!;
  console.log(`API created: ${apiId}`);

  // 2️⃣ Create an integration that points to the Lambda Function URL
  const integrationResult = await client.send(
    new CreateIntegrationCommand({
      ApiId: apiId,
      IntegrationType: "HTTP_PROXY", // direct HTTP call
      IntegrationUri: LAMBDA_URL,
      PayloadFormatVersion: "2.0",
    })
  );

  const integrationId = integrationResult.IntegrationId!;
  console.log(`Integration created: ${integrationId}`);

  // 3️⃣ Create a route that listens on POST /prompt
  await client.send(
    new CreateRouteCommand({
      ApiId: apiId,
      RouteKey: "POST /prompt",
      Target: `integrations/${integrationId}`,
    })
  );

  // 4️⃣ (Optional) Enable CORS – the most common support ticket
  await client.send(
    new UpdateApiCommand({
      ApiId: apiId,
      CorsConfiguration: {
        AllowOrigins: ["*"],
        AllowMethods: ["POST"],
        AllowHeaders: ["Content-Type", "Authorization"],
      },
    })
  );

  console.log(`Route /prompt ready at https://${apiId}.execute-api.${REGION}.amazonaws.com/prompt`);
}

createHttpApi().catch((e) => console.error(e));
```

**What the script does**

`application/json` as binary. This avoids the hidden bug where payloads larger than 6 KB are silently dropped before they reach the Lambda.`HTTP_PROXY` mode means “just forward the request as‑is”).`POST /prompt`) to the integration.
**Tip:** After creating the API, give the API’s execution role permission to invoke the Lambda Function URL (`lambda:InvokeFunctionUrl`) and set the URL’s auth type to `NONE` or `AWS_IAM` depending on your security model.

Now that the front‑door is ready, let’s write the Lambda handler. Node.js 22 ships with the **fetch** API built‑in, so we don’t need any extra HTTP client library.

Why store the cache in a **Map** rather than a plain object?

``` js
import { createHash } from "crypto";
import { fetch } from "node-fetch"; // native in Node 22, import for clarity
import { diagnosticChannel } from "diagnostics_channel";

/**
 * Simple SHA‑256 hash of the prompt. Using a cryptographic hash
 * guarantees a fixed‑length key regardless of prompt size.
 */
function hashPrompt(prompt) {
  return createHash("sha256").update(prompt, "utf8").digest("hex");
}

/**
 * In‑process cache: Map<hash, string>. The value is the complete
 * Server‑Sent Events (SSE) payload we received from Claude.
 */
const cache = new Map();

/**
 * Exponential backoff helper for retrying 429 (Too Many Requests) errors.
 */
async function retryWithBackoff(fn, attempts = 4, delay = 200) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (err.status !== 429) throw err; // only handle throttling
      const wait = delay * 2 ** i;
      console.warn(`Got 429, retrying in ${wait}ms`);
      await new Promise((r) => setTimeout(r, wait));
    }
  }
  throw new Error("Exceeded retry attempts");
}

/**
 * Lambda entry point – receives an APIGatewayProxyEvent (HTTP API payload)
 * and returns a ProxyResult (HTTP response).
 */
export async function handler(event) {
  // -------------------------------------------------
  // 1️⃣ Parse and validate incoming JSON
  // -------------------------------------------------
  let body;
  try {
    body = JSON.parse(event.body);
    if (typeof body.prompt !== "string") throw new Error("Missing prompt");
  } catch (e) {
    return { statusCode: 400, body: JSON.stringify({ error: "Invalid JSON" }) };
  }

  const prompt = body.prompt.trim();
  const key = hashPrompt(prompt);

  // -------------------------------------------------
  // 2️⃣ Check the in‑process cache
  // -------------------------------------------------
  if (cache.has(key)) {
    diagnosticChannel.publish("promptCache.hit", { key });
    return {
      statusCode: 200,
      headers: { "Content-Type": "text/event-stream" },
      body: cache.get(key),
    };
  }

  diagnosticChannel.publish("promptCache.miss", { key });

  // -------------------------------------------------
  // 3️⃣ Call Claude via Bedrock using native fetch
  // -------------------------------------------------
  const region = process.env.AWS_REGION || "us-east-1";
  const bedrockUrl = `https://bedrock-runtime.${region}.amazonaws.com/model/anthropic.claude-v2/invoke`;

  const requestBody = JSON.stringify({
    prompt,
    max_tokens_to_sample: 256,
    temperature: 0.7,
    stream: true, // ask Bedrock to stream SSE
  });

  // The retry wrapper only retries on 429.
  const response = await retryWithBackoff(() =>
    fetch(bedrockUrl, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        // The SDK would normally sign this request; for brevity we assume
        // the Lambda execution role has the needed Bedrock permissions.
      },
      body: requestBody,
    })
  );

  if (!response.ok) {
    const errText = await response.text();
    console.error(`Bedrock error ${response.status}: ${errText}`);
    return { statusCode: 502, body: JSON.stringify({ error: "Upstream failure" }) };
  }

  // -------------------------------------------------
  // 4️⃣ Pipe the streaming SSE back to the client
  // -------------------------------------------------
  // We accumulate the stream into a string so we can cache it.
  let ssePayload = "";
  const decoder = new TextDecoder();

  // The fetch response body is a Node.js ReadableStream.
  for await (const chunk of response.body) {
    const text = decoder.decode(chunk);
    ssePayload += text; // store for caching
    // Immediately forward to the API Gateway client.
    // API Gateway HTTP API expects the full response body at the end,
    // but we can still build it incrementally.
  }

  // Save to cache for future identical prompts.
  cache.set(key, ssePayload);
  diagnosticChannel.publish("promptCache.store", { key });

  // Return as Server‑Sent Events (SSE) content type.
  return {
    statusCode: 200,
    headers: { "Content-Type": "text/event-stream" },
    body: ssePayload,
  };
}
```

**Explanation of key lines**

| Line | Reason | 
|---|---|
| `hashPrompt` | Turns any length prompt into a short, fixed‑size key. | 
| `cache.has(key)` | Quick lookup to decide if we can skip the external call. | 
| `retryWithBackoff` | Handles the common Bedrock throttling response ( `429` ) by waiting longer each attempt. | 
| `fetch(..., { stream: true })` | Asks Bedrock to stream its answer as SSE, which we then forward. | 
| `diagnosticChannel.publish` | Emits lightweight events that other tools can listen to (covered later). | 
| `cache.set(key, ssePayload)` | Stores the whole SSE payload for the next identical request. | 

**In plain English:** The handler first looks in its own notebook (the Map). If the answer is already written down, it reads it out immediately. If not, it asks Claude, writes the spoken answer into the notebook, and then hands the answer back to the caller.

`BinaryMediaTypes` to include `POST` and `Content-Type` header, otherwise browsers block the request.
Claude (the Anthropic model) can return a **stream** of tokens using Server‑Sent Events (SSE). Streaming lets the client start displaying text as soon as the first token arrives, which feels snappier than waiting for the whole response.

Why use **native fetch** instead of the AWS SDK’s `InvokeModelCommand`?

`Accept: text/event-stream` easily.`for await … of`.

``` js
async function streamClaude(prompt) {
  const url = `https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke`;
  const res = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt, stream: true }),
  });

  // Guard against non‑200 responses early.
  if (!res.ok) throw new Error(`Bedrock returned ${res.status}`);

  // The body is an async iterator of Uint8Array chunks.
  const decoder = new TextDecoder();
  for await (const chunk of res.body) {
    const text = decoder.decode(chunk);
    // Each chunk may contain one or more SSE lines like:
    //   data: {"completion":"Hello"}
    console.log("Received chunk:", text);
    // Forward the raw text to the HTTP client (or accumulate).
  }
}
```

**Key points**

`res.body` implements the `chunk` is a `Uint8Array`; converting with `TextDecoder` gives the human‑readable SSE line.`stream: true`, Bedrock never buffers the whole answer; it pushes each token as soon as it’s generated.
**Analogy:** Think of a live sports broadcast. Instead of waiting for the entire game to finish before you hear the result, the commentator describes the play as it happens. Streaming works the same way for LLM output.

Bedrock enforces a request‑per‑second quota per account. When you exceed it, the service returns HTTP 429. The helper `retryWithBackoff` from the previous section automatically retries with increasing delays (200 ms → 400 ms → 800 ms …). This pattern prevents your Lambda from failing outright while still respecting the service limits.

**diagnostics_channel** is a lightweight Node.js feature that lets you emit custom events without pulling in a full‑blown tracing library. It works well inside Lambda because it adds virtually no latency and doesn’t need extra IAM permissions.

Why instrument the cache with diagnostics?

``` js
import { diagnosticChannel } from "diagnostics_channel";

// Listener that logs cache activity – you could replace this with a
// structured logger or a metric exporter.
diagnosticChannel.subscribe("promptCache.hit", (info) => {
  console.info(`[Cache] HIT for ${info.key}`);
});

diagnosticChannel.subscribe("promptCache.miss", (info) => {
  console.info(`[Cache] MISS for ${info.key}`);
});

diagnosticChannel.subscribe("promptCache.store", (info) => {
  console.info(`[Cache] STORED result for ${info.key}`);
});
```

**Where to place the subscription?**

Put the listener code in a separate module that the Lambda imports early (e.g., `observability.js`). Because the subscription runs at module load time, it executes once per container, not per invocation.

**Tip:** If you enable **API Gateway access logs**, you’ll see the same request IDs printed by both the API layer and your diagnostics events, making it easy to correlate end‑to‑end latency.

When the Lambda runs, the `console.info` statements appear in the function’s CloudWatch log stream. You can then create a **Metric Filter** that turns lines like `[Cache] HIT*` into a custom CloudWatch metric (`PromptCacheHit`). Plotting `Hit` vs. `Miss` over time shows you how effective the cache is.

*The core ideas that let you shrink LLM proxy latency.*

By combining these pieces, you can build a lean LLM proxy that feels fast, stays cheap, and remains easy to monitor. Happy coding!

**Transparency notice**

This article was written with the help of an AI system — [Groq](https://groq.com) (GPT OSS 120B).

**Published:** 2026-10-12 · **Primary focus:** APIGateway

All code blocks are intended to be correct and runnable, but please verify them

against the official docs for the tools mentioned before using in production.

*Find an error? Drop a comment — corrections are always welcome.*