Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

A new AWS technical post details how to build a self-service LLM gateway that puts LiteLLM on AWS Fargate in front of Amazon Bedrock, exposing foundation models such as Claude Sonnet and Claude Haiku through a single OpenAI-compatible API with per-user keys. The pattern uses Amazon Bedrock Application Inference Profiles for per-model quota isolation, Amazon RDS for PostgreSQL to store API key metadata and spend tracking, Amazon ElastiCache for Valkey for response caching and cross-task rate limiting, and Amazon CloudWatch to capture requests per minute (RPM) and tokens per minute (TPM) metrics. The post says the setup gives platform teams latency and error visibility, finance teams per-team and per-project inference cost attribution, and security teams consistent access controls, while noting Amazon Bedrock Guardrails are not covered and should be added before production deployment.

Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ Building a self-service LLM gateway with Amazon Bedrock and LiteLLM Developers want AI coding assistants with minimal setup. With a large language model LLM gateway and LiteLLM, they can quickly access foundation models FMs available on Amazon Bedrock while you gain visibility and usage control. Platform teams need to monitor latency and errors, finance teams need to attribute inference costs to teams and projects, and security teams need consistent access controls. An LLM gateway can help address these needs by providing a centrally managed access point to the models that power coding assistants. In this post, we show how to build a self-service LLM gateway using Amazon Bedrock Application Inference Profiles with LiteLLM https://github.com/BerriAI/litellm running on AWS Fargate to abstract Amazon Bedrock https://aws.amazon.com/bedrock/ behind an OpenAI-compatible API. With this pattern, developers can access foundation models through a single API with a dedicated key while you gain per-user cost controls and model performance observability. By the end, you will understand the pattern well enough to build your own version from the illustrative snippets and AWS Cloud Development Kit AWS CDK constructs. This post uses Claude Sonnet and Claude Haiku models for example purposes. For a list of models and availability by AWS Region, see Supported models by AWS Region in Amazon Bedrock https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html . This LLM gateway pattern follows a straightforward request flow from the developer, through the LiteLLM layer, to Amazon Bedrock and the underlying FM. The following diagram illustrates the production topology behind that flow. For simplicity, we don’t cover the addition of Amazon Bedrock Guardrails, such as content filtering and grounding, but you would normally include appropriate responsible AI controls before deploying to production. A developer’s OpenAI-compatible client reaches the LLM gateway through Amazon Route 53, Amazon CloudFront, and an Application Load Balancer ALB . It authenticates to LiteLLM on Fargate with a per-user API key. Behind that, the request draws on Amazon Relational Database Service Amazon RDS PostgreSQL for virtual-key and spend state. It also interacts with Amazon ElastiCache for Valkey as a cache, and AWS Secrets Manager for secrets. LiteLLM then ultimately invokes Amazon Bedrock Application Inference Profiles using a scoped AWS Identity and Access Management IAM task role. Amazon CloudWatch captures requests per minute RPM and tokens per minute TPM metrics and the quota alarm. The architecture consists of these core components: 1. LiteLLM on AWS Fargate : LiteLLM provides a proxy that exposes an OpenAI-compatible API. We run it on an AWS Fargate task with target-tracking Auto Scaling on CPU utilization. Developers point their tools at this endpoint and don’t interact with Amazon Bedrock directly. 2. Self-service portal : LiteLLM includes a lightweight portal for admin and developer self-service. This web app can integrate with your corporate single sign-on SSO . Using the portal, operations engineers set the spend policy while developers self-provision API keys within those parameters. 3. Persistent and cache storage : LiteLLM uses Amazon ElastiCache for Valkey for non-persistent response caching. It also coordinates rate-limiting across AWS Fargate tasks. A separate Amazon RDS PostgreSQL database is the database of record for persistent state such as API key metadata and spend tracking. 4. Amazon Bedrock Application Inference Profiles : Each model such as Claude Sonnet and Claude Haiku gets its own inference profile, which you manage using the AWS Cloud Development Kit AWS CDK . With this, you get per-model quota isolation and a clean abstraction layer between LiteLLM’s configuration and the model versioning of Amazon Bedrock. 5. Langfuse : Open source LLM observability for prompt-level telemetry, latency tracking, and debugging. It complements the infrastructure-level metrics of Amazon CloudWatch with application-level insight. By using an OpenAI-compatible protocol, developers can configure their tools, such as Claude Code, Continue, and Cursor, with a single base URL and API key. Developers then operate at a level above the Amazon Bedrock backend and can swap models, add fallbacks, or migrate model providers without touching developer configurations. Prerequisites You will need an AWS account, Amazon Bedrock model access, and appropriate IAM permissions to deploy this project. You will also need LiteLLM. The LiteLLM AI Gateway component is open source and suitable for a small pilot with some limitations. LiteLLM documents https://docs.litellm.ai/docs/enterprise the AI Gateway as free for up to five users with a license required beyond that. LiteLLM Enterprise is a scalable, production-minded licensed version that includes the open source gateway plus single sign-on, audit logs, delegated admin roles, and support. While the free tier supports virtual keys, the self-service key issuance described in this post requires the Enterprise version. End-to-end configuration: Left to right We walk through each configuration layer from the developer’s machine to the foundation model. Step 1: Developer IDE configuration for example, Claude Code Developers configure their coding assistant to point to the LLM gateway. This is the only configuration a developer needs to make to their tool: With that configuration, each developer gets their own API key from the self-service portal and drops it into their tool configuration. Step 2: LiteLLM gateway configuration Deploy LiteLLM to a container service following the LiteLLM documentation https://docs.litellm.ai/docs/proxy/docker quick start . In this example we are using AWS Fargate. This will include the portal, which you can open at http://localhost:4000/ui during local testing. The LiteLLM gateway is where you will map and enforce model names to Amazon Bedrock inference profile ARNs with per-key spend limits: Set on the Fargate task from AWS Secrets Manager: Note: Don’t hardcode the master key or database credentials in config.yaml . LiteLLM resolves os.environ/VAR NAME references at load time, so set LITELLM MASTER KEY and DATABASE URL as environment variables on the Fargate task, sourced from AWS Secrets Manager. Set the Langfuse credentials the same way in Step 5. When a developer calls claude-sonnet-4-20250514 , for example, LiteLLM resolves it to the inference profile ARN. It then authenticates to Amazon Bedrock using the AWS Fargate task’s IAM role and streams the response back in OpenAI format. Step 3: Bedrock inference profiles CDK With Amazon Bedrock Application Inference Profiles https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles.html you can create named endpoints for specific models, so you get quota isolation and fine-grained IAM control. Choosing cross-Region inference CRIS : Geographic or Global Amazon Bedrock CRIS avoids the overhead of implementing cross-Region request routing yourself. Other benefits include: - Access to a broader pool of Regional capacity. - Automatic routing of requests to a supported Region with available capacity. - Broader model selection, because some models are available only through an inference profile. For this proxy pattern, we recommend Global cross-Region inference when the selected models support it and organizational policies permit inference processing across supported AWS commercial Regions worldwide. Geographic cross-Region inference keeps processing within a defined geography. For example, a US inference profile routes requests only to supported US Regions. Global cross-Region inference provides access to a broader pool of Regional capacity with potential pricing benefits compared with Geographic cross-Region inference. You can verify model availability, applicable service quotas, and model-specific pricing through current AWS documentation and AWS pricing pages. CDK construct Important: For Global CRIS, the copyFrom ARN must reference the system-defined inference profile prefixed with global. , not the foundation-model ARN directly. Using a foundation-model ARN creates a single-Region profile without cross-Region routing. Additionally, the IAM policy must include the resource ARNs the invocation actually touches: the application inference profile and the underlying foundation model in any Region it might route to Region-wildcarded . Scoping only to the application-inference-profile ARN returns not authorized to perform: bedrock:InvokeModel on resource: arn:aws:bedrock:us-east-1::foundation-model/... at invoke time. This is scoped least-privilege: it names the exact models in use, not bedrock: or . Step 4: Bedrock invokes the foundation model The inference profile handles model resolution and cross-Region routing transparently without additional configuration. When LiteLLM calls the inference profile ARN, Amazon Bedrock routes to available capacity across configured AWS Regions and returns the response. Step 5: Langfuse observability LiteLLM integrates with Langfuse through logging callbacks, providing visibility into model invocations without requiring changes to developer tool configurations. To enable the integration, merge the following entries into the existing litellm settings block from the Step 2 configuration, preserving existing callback entries: Configure the Langfuse public key, secret key, and deployment endpoint as environment variables on the Fargate task. Store the credentials in AWS Secrets Manager: The endpoint must match the selected Langfuse Cloud region or self-hosted deployment. Choose the hosting option based on data governance and operational requirements. Before enabling prompt and response logging, review the content captured by the integration and configure appropriate redaction, retention, and access controls. Langfuse traces can include prompts, responses, model parameters, token usage, and latency, depending on the integration and logging configuration. This information helps investigate unexpected model responses and complements the operational metrics collected in Amazon CloudWatch. Implementing cost controls Cost controls are an important dimension to a successful AI strategy. Without cost controls, heavy usage of extended thinking features can lead to unexpectedly high charges, which we observed in our early pilots. The spend limits configured in Step 2 max budget: 250 , budget duration: 7d enforce a rolling 7-day window per API key, provided per-model model info costs are set refer to the Step 2 note . When a user reaches their limit, LiteLLM returns an HTTP 429 code with a clear message explaining that their budget has been exceeded. There’s no silent degradation or surprise bill. Developers and engineering teams can check current spend and reset timing through any of the following: - Daily spend : Query GET /user/daily/activity https://docs.litellm.ai/docs/proxy/cost tracking daily-spend-breakdown-api with a start date and end date range to get per-day metrics: spend, prompt tokens, completion tokens, and API requests, broken down by model, provider, and API key. - Per user spend : Query GET /user/info?user id=… https://docs.litellm.ai/docs/proxy/cost tracking total-spend-per-user for a summary of a user’s tot