{"slug": "self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock", "title": "Self-hosted LiteLLM gateway for Codex on AWS ECS with Bedrock", "summary": "A self-hosted LiteLLM gateway for OpenAI's Codex on AWS ECS with Bedrock provides per-team virtual keys, spend caps, and CloudWatch logging, with a reference CDK stack in the guidance-codex repo under the feat/enterprise-gateway-readiness branch. The gateway routes model calls to Bedrock models like Claude 3.5 Sonnet and GPT-4o, costs an estimated $150–300/month, and is overkill for small teams or those not mixing providers.", "body_md": "# Self-hosted LiteLLM gateway for Codex on AWS ECS with Bedrock\n\n## What this actually gets you\n\nCodex keeps doing what it does best — reading your repo, proposing edits, running tests inside its sandbox. The only difference: every model call hits `/v1/responses`\n\non your LiteLLM endpoint instead of OpenAI directly. LiteLLM validates the request, checks the caller's virtual key against your policy, then assumes its ECS task role to invoke the Bedrock model ([Claude](/en/tags/claude/) 3.5 Sonnet, GPT-4o, whatever you've approved). The response streams back through the same path.\n\nYou get:\n\n- Per-team or per-project virtual keys with hard spend caps\n- Request/response logging to CloudWatch for compliance\n- WAF rules in front of the ALB for IP allowlists or geo-blocking\n- RDS PostgreSQL backing LiteLLM's usage and budget tables\n- Secrets Manager + KMS for key storage — no plaintext creds in task definitions\n\n## Deploy the stack\n\nThe reference implementation lives in the `guidance-codex`\n\nrepo under the `feat/enterprise-gateway-readiness`\n\nbranch. Clone it and follow the LiteLLM quickstart — it provisions everything via CDK.\n\n```\ngit clone -b feat/enterprise-gateway-readiness \\\n  https://github.com/openai-on-aws/guidance-codex.git\ncd guidance-codex\n# Bootstrap CDK if you haven't in this account/region\ncdk bootstrap\n# Deploy the gateway stack (ALB → Fargate → RDS → Bedrock)\ncdk deploy LiteLLMGatewayStack\n```\n\nThe stack outputs the gateway URL and a sample virtual key. Save both.\n\n## Configure Codex to use your gateway\n\nOn each developer machine, set two environment variables before launching Codex:\n\n```\nexport OPENAI_BASE_URL=https://<gateway-alb-dns>/v1\nexport OPENAI_API_KEY=sk-litellm-<your-virtual-key>\n```\n\nThen run `codex`\n\nnormally. The CLI will POST to your `/v1/responses`\n\nendpoint, and you'll see the traffic in CloudWatch Logs under the Fargate task's log group.\n\n## Validate the loop works end-to-end\n\n1. **Streaming** — ask Codex to \"write a quicksort in Python with comments.\" You should see tokens appear incrementally in the terminal.\n\n2. **Function calling** — have it \"list all `.py`\n\nfiles in this repo and summarize each.\" Codex will emit a `list_files`\n\ntool call, run it locally, then send the result back through the gateway for the next turn.\n\n3. **Budget enforcement** — exceed the virtual key's daily limit; the gateway returns a 429 with a clear error body.\n\n## When this is overkill\n\n**Small team, high trust**— IAM Identity Center with direct Bedrock access is simpler. No gateway to maintain.** Don't want to run infra**— Portkey's managed gateway handles auth, budgets, and observability with a SaaS control plane. You trade operational burden for a monthly bill.**Only need OpenAI models**— LiteLLM adds latency (one extra hop). If you're not routing to Bedrock or mixing providers, a plain API key works fine.\n\n## Worth the effort?\n\nFor orgs that need audit trails, hard spend controls, and zero long-lived credentials on laptops — yes. The CDK stack is production-grade (multi-AZ Fargate, RDS with deletion protection, WAF managed rules). Expect ~$150–300/month for the gateway infra at modest traffic, plus Bedrock model costs. The repo's quickstart gets you to a working endpoint in under an hour.\n\n[Next Claude Fable 5.1 just cracked a 370-year-old royalist cipher →](/en/threads/8733/)", "url": "https://wpnews.pro/news/self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock", "canonical_source": "https://promptcube3.com/en/threads/8763/", "published_at": "2026-09-03 22:50:12+00:00", "updated_at": "2026-09-03 23:22:33.969943+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["LiteLLM", "OpenAI Codex", "AWS ECS", "Amazon Bedrock", "Claude 3.5 Sonnet", "GPT-4o", "CloudWatch", "Portkey"], "alternates": {"html": "https://wpnews.pro/news/self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock", "markdown": "https://wpnews.pro/news/self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock.md", "text": "https://wpnews.pro/news/self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock.txt", "jsonld": "https://wpnews.pro/news/self-hosted-litellm-gateway-for-codex-on-aws-ecs-with-bedrock.jsonld"}}