API Gateway HTTP API + Lambda Function URLs: Building a Prompt‑Caching LLM Proxy, Explained Simply

A developer outlined a lightweight LLM proxy architecture that routes API Gateway HTTP APIs directly to Lambda Function URLs and adds a small in-process prompt cache, claiming sub-100 ms responses for repeated prompts. The setup uses Node.js 22's native fetch and diagnostics_channel, with SHA-256 prompt hashes stored in an in-memory Map that is rebuilt on cold starts, and enables binary media support to avoid a 6 KB body-drop bug.

LLM‑powered features like code completion are often wrapped in heavyweight Lambda functions that add latency and cost. By letting API Gateway’s HTTP API route directly to a lightweight Function URL and adding a tiny in‑process cache, you get sub‑100 ms responses for repeated prompts. This post shows exactly how to wire it together with Node.js 22’s native fetch and diagnostics channel. When a user asks an LLM large language model the same question twice, the model does the same heavy computation both times. That work costs money and adds time. A prompt cache stores the model’s answer for a given prompt so the second request can be answered instantly, without contacting the model again. In plain English: Think of a cache like a sticky note you put on your desk. The first time you look up a phone number you write it down; the next time you need it, you just glance at the note instead of dialing the operator again. Why keep the cache inside the Lambda instead of an external store? The trade‑off is that the cache disappears when the Lambda container is recycled, but for high‑frequency prompts that’s fine; the next cold start will rebuild the cache quickly. // A very small in‑process cache. // Keys are SHA‑256 hashes of the prompt string. const cache = new Map<string, string ; Key takeaway: Caching at the function level can shave tens of milliseconds off every repeat request, and you don’t need a separate service to make it happen. API Gateway offers two flavors: REST API v1 and HTTP API v2 . The latter is lighter, cheaper, and supports Function URLs – a direct HTTPS endpoint that invokes a Lambda without the extra integration layer. Why use a Function URL instead of the classic “Lambda integration”? Below is a minimal script that creates an HTTP API, a route /prompt , and ties it to a Lambda Function URL. It also shows the binary‑support flag that prevents the 6 KB body‑drop bug. js import { ApiGatewayV2Client, CreateApiCommand, CreateIntegrationCommand, CreateRouteCommand, UpdateApiCommand, } from "@aws-sdk/client-api-gateway"; // Replace with your region and Lambda ARN const REGION = "us-east-1"; const LAMBDA URL = "https://abcde12345.lambda-url.us-east-1.on.aws"; const client = new ApiGatewayV2Client { region: REGION } ; async function createHttpApi { // 1️⃣ Create the HTTP API const apiResult = await client.send new CreateApiCommand { Name: "PromptCacheAPI", ProtocolType: "HTTP", // Enable binary support so bodies 6 KB are not stripped // the list of supported media types must include application/json BinaryMediaTypes: "application/json" , } ; const apiId = apiResult.ApiId ; console.log API created: ${apiId} ; // 2️⃣ Create an integration that points to the Lambda Function URL const integrationResult = await client.send new CreateIntegrationCommand { ApiId: apiId, IntegrationType: "HTTP PROXY", // direct HTTP call IntegrationUri: LAMBDA URL, PayloadFormatVersion: "2.0", } ; const integrationId = integrationResult.IntegrationId ; console.log Integration created: ${integrationId} ; // 3️⃣ Create a route that listens on POST /prompt await client.send new CreateRouteCommand { ApiId: apiId, RouteKey: "POST /prompt", Target: integrations/${integrationId} , } ; // 4️⃣ Optional Enable CORS – the most common support ticket await client.send new UpdateApiCommand { ApiId: apiId, CorsConfiguration: { AllowOrigins: " " , AllowMethods: "POST" , AllowHeaders: "Content-Type", "Authorization" , }, } ; console.log Route /prompt ready at https://${apiId}.execute-api.${REGION}.amazonaws.com/prompt ; } createHttpApi .catch e = console.error e ; What the script does application/json as binary. This avoids the hidden bug where payloads larger than 6 KB are silently dropped before they reach the Lambda. HTTP PROXY mode means “just forward the request as‑is” . POST /prompt to the integration. Tip: After creating the API, give the API’s execution role permission to invoke the Lambda Function URL lambda:InvokeFunctionUrl and set the URL’s auth type to NONE or AWS IAM depending on your security model. Now that the front‑door is ready, let’s write the Lambda handler. Node.js 22 ships with the fetch API built‑in, so we don’t need any extra HTTP client library. Why store the cache in a Map rather than a plain object? js import { createHash } from "crypto"; import { fetch } from "node-fetch"; // native in Node 22, import for clarity import { diagnosticChannel } from "diagnostics channel"; / Simple SHA‑256 hash of the prompt. Using a cryptographic hash guarantees a fixed‑length key regardless of prompt size. / function hashPrompt prompt { return createHash "sha256" .update prompt, "utf8" .digest "hex" ; } / In‑process cache: Map<hash, string . The value is the complete Server‑Sent Events SSE payload we received from Claude. / const cache = new Map ; / Exponential backoff helper for retrying 429 Too Many Requests errors. / async function retryWithBackoff fn, attempts = 4, delay = 200 { for let i = 0; i < attempts; i++ { try { return await fn ; } catch err { if err.status == 429 throw err; // only handle throttling const wait = delay 2 i; console.warn Got 429, retrying in ${wait}ms ; await new Promise r = setTimeout r, wait ; } } throw new Error "Exceeded retry attempts" ; } / Lambda entry point – receives an APIGatewayProxyEvent HTTP API payload and returns a ProxyResult HTTP response . / export async function handler event { // ------------------------------------------------- // 1️⃣ Parse and validate incoming JSON // ------------------------------------------------- let body; try { body = JSON.parse event.body ; if typeof body.prompt == "string" throw new Error "Missing prompt" ; } catch e { return { statusCode: 400, body: JSON.stringify { error: "Invalid JSON" } }; } const prompt = body.prompt.trim ; const key = hashPrompt prompt ; // ------------------------------------------------- // 2️⃣ Check the in‑process cache // ------------------------------------------------- if cache.has key { diagnosticChannel.publish "promptCache.hit", { key } ; return { statusCode: 200, headers: { "Content-Type": "text/event-stream" }, body: cache.get key , }; } diagnosticChannel.publish "promptCache.miss", { key } ; // ------------------------------------------------- // 3️⃣ Call Claude via Bedrock using native fetch // ------------------------------------------------- const region = process.env.AWS REGION || "us-east-1"; const bedrockUrl = https://bedrock-runtime.${region}.amazonaws.com/model/anthropic.claude-v2/invoke ; const requestBody = JSON.stringify { prompt, max tokens to sample: 256, temperature: 0.7, stream: true, // ask Bedrock to stream SSE } ; // The retry wrapper only retries on 429. const response = await retryWithBackoff = fetch bedrockUrl, { method: "POST", headers: { "Content-Type": "application/json", // The SDK would normally sign this request; for brevity we assume // the Lambda execution role has the needed Bedrock permissions. }, body: requestBody, } ; if response.ok { const errText = await response.text ; console.error Bedrock error ${response.status}: ${errText} ; return { statusCode: 502, body: JSON.stringify { error: "Upstream failure" } }; } // ------------------------------------------------- // 4️⃣ Pipe the streaming SSE back to the client // ------------------------------------------------- // We accumulate the stream into a string so we can cache it. let ssePayload = ""; const decoder = new TextDecoder ; // The fetch response body is a Node.js ReadableStream. for await const chunk of response.body { const text = decoder.decode chunk ; ssePayload += text; // store for caching // Immediately forward to the API Gateway client. // API Gateway HTTP API expects the full response body at the end, // but we can still build it incrementally. } // Save to cache for future identical prompts. cache.set key, ssePayload ; diagnosticChannel.publish "promptCache.store", { key } ; // Return as Server‑Sent Events SSE content type. return { statusCode: 200, headers: { "Content-Type": "text/event-stream" }, body: ssePayload, }; } Explanation of key lines | Line | Reason | |---|---| | hashPrompt | Turns any length prompt into a short, fixed‑size key. | | cache.has key | Quick lookup to decide if we can skip the external call. | | retryWithBackoff | Handles the common Bedrock throttling response 429 by waiting longer each attempt. | | fetch ..., { stream: true } | Asks Bedrock to stream its answer as SSE, which we then forward. | | diagnosticChannel.publish | Emits lightweight events that other tools can listen to covered later . | | cache.set key, ssePayload | Stores the whole SSE payload for the next identical request. | In plain English: The handler first looks in its own notebook the Map . If the answer is already written down, it reads it out immediately. If not, it asks Claude, writes the spoken answer into the notebook, and then hands the answer back to the caller. BinaryMediaTypes to include POST and Content-Type header, otherwise browsers block the request. Claude the Anthropic model can return a stream of tokens using Server‑Sent Events SSE . Streaming lets the client start displaying text as soon as the first token arrives, which feels snappier than waiting for the whole response. Why use native fetch instead of the AWS SDK’s InvokeModelCommand ? Accept: text/event-stream easily. for await … of . js async function streamClaude prompt { const url = https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke ; const res = await fetch url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify { prompt, stream: true } , } ; // Guard against non‑200 responses early. if res.ok throw new Error Bedrock returned ${res.status} ; // The body is an async iterator of Uint8Array chunks. const decoder = new TextDecoder ; for await const chunk of res.body { const text = decoder.decode chunk ; // Each chunk may contain one or more SSE lines like: // data: {"completion":"Hello"} console.log "Received chunk:", text ; // Forward the raw text to the HTTP client or accumulate . } } Key points res.body implements the chunk is a Uint8Array ; converting with TextDecoder gives the human‑readable SSE line. stream: true , Bedrock never buffers the whole answer; it pushes each token as soon as it’s generated. Analogy: Think of a live sports broadcast. Instead of waiting for the entire game to finish before you hear the result, the commentator describes the play as it happens. Streaming works the same way for LLM output. Bedrock enforces a request‑per‑second quota per account. When you exceed it, the service returns HTTP 429. The helper retryWithBackoff from the previous section automatically retries with increasing delays 200 ms → 400 ms → 800 ms … . This pattern prevents your Lambda from failing outright while still respecting the service limits. diagnostics channel is a lightweight Node.js feature that lets you emit custom events without pulling in a full‑blown tracing library. It works well inside Lambda because it adds virtually no latency and doesn’t need extra IAM permissions. Why instrument the cache with diagnostics? js import { diagnosticChannel } from "diagnostics channel"; // Listener that logs cache activity – you could replace this with a // structured logger or a metric exporter. diagnosticChannel.subscribe "promptCache.hit", info = { console.info Cache HIT for ${info.key} ; } ; diagnosticChannel.subscribe "promptCache.miss", info = { console.info Cache MISS for ${info.key} ; } ; diagnosticChannel.subscribe "promptCache.store", info = { console.info Cache STORED result for ${info.key} ; } ; Where to place the subscription? Put the listener code in a separate module that the Lambda imports early e.g., observability.js . Because the subscription runs at module load time, it executes once per container, not per invocation. Tip: If you enable API Gateway access logs , you’ll see the same request IDs printed by both the API layer and your diagnostics events, making it easy to correlate end‑to‑end latency. When the Lambda runs, the console.info statements appear in the function’s CloudWatch log stream. You can then create a Metric Filter that turns lines like Cache HIT into a custom CloudWatch metric PromptCacheHit . Plotting Hit vs. Miss over time shows you how effective the cache is. The core ideas that let you shrink LLM proxy latency. By combining these pieces, you can build a lean LLM proxy that feels fast, stays cheap, and remains easy to monitor. Happy coding Transparency notice This article was written with the help of an AI system — Groq https://groq.com GPT OSS 120B . Published: 2026-10-12 · Primary focus: APIGateway All code blocks are intended to be correct and runnable, but please verify them against the official docs for the tools mentioned before using in production. Find an error? Drop a comment — corrections are always welcome.