{"slug": "set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock", "title": "Set up OpenAI ChatGPT Codex with LiteLLM on Amazon ECS and Amazon Bedrock", "summary": "Amazon Web Services (AWS) published a guide for deploying OpenAI ChatGPT Codex with the open-source LiteLLM gateway on Amazon Elastic Container Service (ECS) and Amazon Bedrock, enabling centralized enterprise controls for generative AI coding agents. The architecture routes Codex requests through LiteLLM for authentication, budgets, rate limits, and telemetry, with Amazon RDS for PostgreSQL for state and AWS Secrets Manager for key storage. The guide includes a five-step request flow and notes when direct AWS IAM Identity Center access or a managed gateway like Portkey may be preferable.", "body_md": "[Artificial Intelligence](/blogs/machine-learning/)\n\n# Set up OpenAI ChatGPT Codex with LiteLLM on Amazon ECS and Amazon Bedrock\n\nOpenAI ChatGPT Codex with LiteLLM can provide centralized enterprise controls for generative AI coding agents. These agents help developers understand repositories, write code, run tests, and complete multi-step engineering tasks. As organizations move from individual experimentation to managed adoption, teams need a consistent way to control model access and attribute consumption. They must also apply budgets and rate limits, and observe the model-access path.\n\nOpenAI Codex (Codex) runs its task loop on the developer workstation. It reads local files and runs approved tools under its local sandbox and approval settings. Customers can still route model inference through infrastructure in the customer’s AWS account.\n\nIn this post, we walk you through deploying a customer-operated LiteLLM gateway on Amazon Elastic Container Service (Amazon ECS). We show how to connect it to an OpenAI model on Amazon Bedrock, and configure Codex to use the gateway’s Responses API. We also show how to validate semantic continuation, streaming, and function calling. Finally, we explain when direct AWS IAM Identity Center access or a managed gateway such as Portkey can be a better fit.\n\nThe complete implementation is available in the [guidance-codex repository](https://github.com/openai-on-aws/guidance-codex/tree/feat/enterprise-gateway-readiness). For the primary path in this post, follow the [LiteLLM on AWS quickstart](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/QUICKSTART_LLM_GATEWAY_LITELLM.md).\n\n## Solution overview\n\nThe following architecture places LiteLLM between Codex and Amazon Bedrock. LiteLLM becomes the shared control point for model authentication, routing, budgets, rate limits, and gateway telemetry. Codex retains responsibility for the local task and tool-execution loop.\n\nThe following diagram shows the end-to-end request flow from the developer workstation through the gateway to Amazon Bedrock and back. Look for the five numbered steps that trace a single model turn through the infrastructure.\n\nThe request flow proceeds in five steps:\n\n- Codex sends the current task context and available tool definitions to the gateway’s\n`/v1/responses`\n\nendpoint. - The Application Load Balancer and AWS WAF apply network and web-layer controls before forwarding the request to LiteLLM on AWS Fargate.\n- LiteLLM authenticates the caller, checks the configured model and consumption policy, and uses its ECS task role to invoke the approved model on Amazon Bedrock.\n- Amazon Bedrock returns text or a function call through LiteLLM.\n- If the model requests a tool, Codex runs it locally under its sandbox and approval policy. Codex sends the tool result through LiteLLM on the next Responses request, and the loop continues.\n\nThe reference deployment also uses:\n\n- Amazon Relational Database Service (Amazon RDS) for PostgreSQL for LiteLLM state, usage, and budget data.\n- AWS Secrets Manager and AWS Key Management Service (AWS KMS) for gateway and scoped-key storage.\n- Amazon CloudWatch logs, CloudWatch Container Insights, alarms, and deployment health.\n- Amazon Elastic Container Registry (Amazon ECR) for an immutable gateway image.\n- Optional AWS WAF managed protections and source-IP rate limiting.\n\nThis separation matters. The gateway does not receive a general-purpose shell in the AWS account, and it doesn’t replace Codex’s local approvals. It governs each model turn.\n\n## Why use LiteLLM for this pattern\n\n[LiteLLM](https://docs.litellm.ai/docs/simple_proxy) is an open source AI gateway that provides model routing, virtual keys, budgets, rate limits, and usage telemetry behind API-compatible endpoints. In this pattern, the customer operates LiteLLM and its supporting infrastructure inside their AWS account.\n\nDirect access to Amazon Bedrock is the lowest-complexity option when native AWS identity, AWS Identity and Access Management (IAM) policies, and AWS CloudTrail logs meet the customer’s requirements. A gateway becomes useful when the system team needs additional controls that are consistent across developers, teams, or model providers.\n\nLiteLLM is a useful customer-operated option when you need to:\n\n- Allow only approved model aliases.\n- Issue user- or team-scoped gateway keys.\n- Apply hard budgets and requests-per-minute or tokens-per-minute limits.\n- Centralize routing and fallback policy.\n- Retain gateway-level identity when the upstream model uses a shared ECS task role.\n- Operate the gateway, database, networking, logs, and upgrade process in your own AWS account.\n\nThe operational responsibility is the main trade-off. Your team owns gateway availability, database lifecycle, version upgrades, incident response, and capacity planning.\n\n## Deploy a LiteLLM gateway for Codex on Amazon ECS\n\nThis section walks you through deploying the LiteLLM gateway, connecting it to Amazon Bedrock, and configuring Codex to send requests through the gateway.\n\n### Prerequisites\n\nFor this walkthrough, you need:\n\n- An AWS account and permissions to create a virtual private cloud (VPC), Amazon ECS, Elastic Load Balancing, Amazon RDS, Amazon ECR, AWS WAF, IAM, AWS KMS, Secrets Manager, and CloudWatch resources.\n- Access to the selected OpenAI model on Amazon Bedrock in the deployment AWS Region.\n- AWS Command Line Interface (AWS CLI) version 2 with an authenticated profile.\n- Docker with Buildx.\n- Codex CLI.\n- Python 3.\n- For an HTTPS deployment, a public Amazon Route 53 hosted zone or an existing AWS Certificate Manager (ACM) certificate in the same Region.\n\nThe walkthrough was validated in the US East (N. Virginia) Region (us-east-1) with the gateway alias `openai.gpt-5.5`\n\n, which maps to `bedrock_mantle/openai.gpt-5.5`\n\nin the LiteLLM configuration. Model availability varies by account and Region.\n\nCost note:This solution creates billable resources, including an Application Load Balancer, Fargate tasks, Amazon RDS, AWS WAF, logs, and model inference. A sample VPC can also create networking charges. Review current pricing for your Region and follow the cleanup section after the walkthrough.\n\n### Deploy LiteLLM on Amazon ECS\n\nClone the repository and create a local deployment environment file:\n\nThe real `.env.deploy`\n\nfile is ignored by Git. Set the intended AWS profile, Regions, source CIDR, and DNS or certificate values. The following excerpt shows the production-oriented settings:\n\nUse an existing certificate by setting `ALB_CERTIFICATE_ARN`\n\ninstead of `ROUTE53_HOSTED_ZONE_ID`\n\n. For a customer landing zone, also provide an existing VPC and separate public, private application, and private database subnets as described in the production deployment guide.\n\nRun the read-only preflight:\n\nThe preflight verifies AWS CLI v2, AWS identity, Docker, immutable image references, Region consistency, CIDR restrictions, TLS inputs, local documentation links, and AWS CloudFormation syntax when cfn-lint is available.\n\nBuild the reviewed LiteLLM image and push it to Amazon ECR:\n\nThe build uses a digest-pinned LiteLLM base image and records the resulting ECR digest in a local ignored state file. CloudFormation receives the immutable digest, rather than a mutable image tag. Internally, the helper creates or reuses an immutable Amazon ECR repository, signs in to Amazon ECR, runs `docker buildx build --push`\n\n, and resolves the pushed image digest.\n\nCreate a non-executed CloudFormation change set:\n\nThis calls `aws cloudformation deploy --no-execute-changeset`\n\nfor the networking or gateway template. It creates a reviewable change set but does not execute it.\n\nReview the change set, then deploy:\n\nThe deploy helper runs `aws cloudformation deploy`\n\nfor the networking stack and then the LiteLLM gateway stack. The status helper runs `aws cloudformation describe-stacks`\n\nand prints the stack state and outputs.\n\nThe ECS service uses deployment circuit-breaker rollback and Application Load Balancer health checks. The reference template also configures target-tracking autoscaling, encrypted logs and data, RDS backups, ALB access logs, and operational alarms.\n\nFor customer environments, keep `ENABLE_TLS=true`\n\n, use a trusted DNS name and an ACM certificate, and restrict the Application Load Balancer to approved corporate or VPN CIDRs. Place ECS tasks and Amazon RDS in private subnets. Don’t expose the ECS task port 4000 or PostgreSQL port 5432 publicly.\n\n### Create a scoped gateway identity\n\nDon’t distribute the LiteLLM master key to developers. Configure a user or team identity and policy in the ignored deployment file:\n\nProvision the key:\n\nThe helper resolves the master credential inside a child process, calls the LiteLLM `/key/generate`\n\nAPI with the configured model, budget, and rate policies, and writes the generated key directly to a KMS-encrypted Secrets Manager secret. It doesn’t place either credential in command arguments or print them to the terminal.\n\nFor enterprise rollout, grant each developer profile permission to read only its assigned scoped-key secret and decrypt it with the stack KMS key. Use separate secret paths and IAM policies for teams or environments.\n\n### Configure Codex\n\nGenerate the provider block:\n\nThe helper reads the deployed gateway endpoint from CloudFormation and prints the following Codex provider configuration. It doesn’t write to the user configuration automatically.\n\nAdd the output to the user-level ~/.codex/`config.toml`\n\n. Provider and authentication settings belong in user-level configuration. Codex ignores them in project-local .codex/`config.toml`\n\nfiles.\n\nThe generated configuration has this shape:\n\nCodex runs the authentication command without standard input and reads the bearer token from its standard output. The helper retrieves the current secret using the named AWS profile, so the token is not stored in `config.toml`\n\n.\n\nIn the LiteLLM administration UI, **Models + Endpoints** shows the stable aliases available to developers and their upstream Amazon Bedrock mappings. This provides a quick visual check that developers see the gateway alias rather than coupling their Codex configuration directly to a provider-specific model ID.\n\nThe following figure shows the Models + Endpoints page with two model aliases configured. Confirm that your gateway aliases appear in this list before proceeding to the Codex configuration.\n\n### Test Codex through LiteLLM\n\nStart a new interactive Codex session and use `/status`\n\nto verify that `litellm-gateway`\n\nis the selected provider. For a repeatable non-interactive test, first run a minimal smoke request:\n\nThe command should exit successfully and print `LITELLM_GATEWAY_OK`\n\n. Next, exercise the agent loop with a task that requires both model inference and a local tool:\n\nCodex sends the task and tool definitions through LiteLLM. If the model asks to read the file, Codex runs that command locally and returns the tool result through the same gateway.\n\nOpen **Logs** in the LiteLLM administration UI, select **Request Logs**, and filter to the test window. Verify:\n\n- the requests have a\n`Success`\n\nstatus. - the key alias identifies the dedicated walkthrough or developer key.\n- the model resolves to the expected Amazon Bedrock mapping.\n- token count, request duration, and cost are populated.\n- the tool-using task creates multiple rows as Codex sends tool results in subsequent Responses requests.\n\nA `403`\n\nresponse before a row appears in LiteLLM commonly indicates an upstream network or AWS WAF block. A `401`\n\nresponse indicates missing or invalid gateway authentication. Don’t publish prompts, responses, raw keys, full request IDs, or real employee identities in screenshots.\n\nThe following figure shows the LiteLLM **Request Logs** page after several successful Codex requests have passed through the gateway. Use this page to confirm that requests are reaching Amazon Bedrock and to troubleshoot errors by status code.\n\n### Validate the Responses API contract\n\nA successful text prompt does not prove that an agent workflow is compatible. Codex relies on more than a basic chat-completions response. Run the included strict probe:\n\nThe helper resolves the scoped key into a child-process environment and runs deployment/scripts/`validate-responses-contract.py`\n\nagainst the deployed gateway.\n\nThe probe verifies:\n\n- required Responses object fields and output shape.\n- semantic continuation with\n`previous_response_id`\n\n. - server-sent event streaming with a completed terminal response.\n- a forced function-tool call with a call ID.\n\nThe continuation test records a unique test marker in the first response and verifies that it can be recalled in a follow-up response. This catches gateways that accept `previous_response_id`\n\nsyntactically but do not preserve the prior response state.\n\nThe live deployment passed the complete contract. CloudFormation completed successfully, the ECS service reached its desired task count, the deployment rollout completed, the ALB target was healthy, and the encrypted PostgreSQL database was available and not publicly accessible.\n\nThe following figure shows the end-to-end deployment validation dashboard captured from the live walkthrough stack. Look for the overall pass/fail status and the four detail panels that confirm infrastructure health, API contract compliance, data-tier configuration, and security posture.\n\nThe validation script is a compatibility gate, not a load test. Before production, also test concurrent agent sessions, long-running streams, request cancellation, key revocation, failure recovery, and expected peak traffic.\n\n### Operationalize LiteLLM for enterprise use\n\nThe successful `codex exec`\n\nand contract tests show that Codex can use the gateway, but compatibility is only the starting point. Before onboarding developers, define how the gateway will control model access, attribute usage, enforce consumption policy, and provide an operational record for each model request.\n\nBegin with the model surface that developers are allowed to use. Publish stable gateway aliases only for models approved by the organization, and pin each upstream mapping in deployment/litellm/`litellm_config.yaml`\n\n. Rebuild the gateway image and promote the same Amazon ECR digest through environments. Developers can continue to configure a stable alias while the system team retains control over the Amazon Bedrock model behind it.\n\nModel policy becomes more useful when every request is attributable. Issue separate scoped keys for users, teams, or workloads instead of distributing the LiteLLM master key. Amazon Bedrock sees the LiteLLM ECS task role on the upstream request, so carry the originating identity in the scoped key, LiteLLM records, and exported telemetry. This preserves developer or team attribution at the gateway and allows request IDs to be correlated with AWS service logs without recording credentials.\n\nThe same scoped identity can enforce consumption policy. Set `CODEX_KEY_MAX_BUDGET`\n\n, `CODEX_KEY_BUDGET_DURATION`\n\n, `CODEX_KEY_TPM_LIMIT`\n\n, and `CODEX_KEY_RPM_LIMIT`\n\nwhen provisioning a key. Validate those controls with a disposable identity by crossing a configured threshold and confirming that LiteLLM rejects the next request. This tests the policy itself rather than only confirming that the settings were accepted.\n\nOperating the gateway also requires a view of the complete request path. AWS infrastructure signals show whether the service is healthy, while LiteLLM records show who used which model and how much capacity or budget the request consumed. Monitor ECS desired and running task counts, deployment rollback, Application Load Balancer target health and latency, Amazon RDS health, LiteLLM requests and rejections, token usage, spend, secret access, IAM changes, and AWS WAF blocks.\n\nBefore developer onboarding, decide whether prompts and responses can be logged. Treat this content as potentially sensitive customer data and define redaction, encryption, access controls, and retention accordingly. Production deployments should enable Amazon Bedrock Guardrails on the model-access path for content filtering, denied-topic detection, and grounding checks. Gateway-level controls (budgets, rate limits, and routing) complement but do not replace responsible AI safeguards applied at the model layer.\n\nThe following figure shows the LiteLLM Usage page, which aggregates request and token metrics across all developers using the gateway. Use this view to monitor adoption, identify cost anomalies, and isolate traffic by user or time range before adjusting budgets or rate limits.\n\nThe validation window in Figure 5 includes successful traffic and a failed request, demonstrating that the dashboard exposes both normal activity and errors. A tool-using task can produce multiple request-log rows because Codex returns local tool results to the model in subsequent Responses requests. The result makes the execution boundary visible in operational data: LiteLLM governs and records model requests, while Codex executes tools on the developer workstation.\n\n### Consider two alternative access paths\n\nLiteLLM is the primary walkthrough because it makes the additional gateway controls visible. It isn’t the right fit for every customer.\n\n#### Direct access with IAM Identity Center\n\nUse the built-in `amazon-bedrock`\n\nCodex provider when AWS native identity and audit controls are sufficient:\n\nThe developer signs in with a named IAM Identity Center profile:\n\nThis path removes the gateway, database, and associated operations. It retains the developer’s AWS session identity in AWS CloudTrail logs, but it doesn’t add a centralized gateway-level hard budget or routing policy.\n\nThe repository’s [IAM Identity Center quickstart](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/QUICKSTART_NATIVE_AWS_ACCESS.md) includes CloudFormation and helper commands to create an isolated group and permission set, assign the group to an AWS account, print the client configuration, and validate the signed-in profile against Amazon Bedrock. Use this direct path as the baseline before adding a gateway.\n\n#### Managed or hybrid access with Portkey\n\nPortkey can be useful when the customer wants a managed control plane, centralized routing and policy, or a supported hybrid data plane without operating the LiteLLM reference stack.\n\nFor a Codex evaluation, configure a Portkey workspace key, an Amazon Bedrock Model Catalog provider, and the Responses wire protocol:\n\nRun the same strict Responses probe before promotion. In particular, validate semantic `previous_response_id`\n\ncontinuation, streaming, function calls, the exact Amazon Bedrock route, and the IAM role and external-ID design. Product documentation or an authentication response alone is not evidence that the intended model path satisfies the complete Codex contract.\n\nChoose Portkey when the managed or hybrid operating model is more important than running the gateway entirely in your AWS account. Review vendor licensing, data handling, Regional availability, failure modes, and support boundaries as part of the architecture decision.\n\n### Clean up\n\nPreview what CloudFormation will delete and retain:\n\nThis reads the stack and lists resources and retention behavior without deleting anything.\n\nDelete the gateway only after confirming its exact stack name:\n\nThe helper checks Amazon RDS deletion protection, calls `aws cloudformation delete-stack`\n\n, and waits for gateway-stack deletion to complete.\n\nFor a sample networking stack that is not shared, opt in and confirm it separately:\n\nThis additionally deletes the sample networking stack after the gateway stack. Don’t use this option for shared networking.\n\nThe reference stack creates a final Amazon RDS snapshot and retains KMS keys, Secrets Manager secrets, the CloudWatch log group, and the ALB access-log bucket. ECR images and provisioned scoped-key secrets are also outside stack deletion. Review and remove retained resources according to your data retention policy.\n\n## Conclusion\n\nRouting Codex through LiteLLM provides a customer-operated control point for model selection, scoped identity, budgets, rate limits, and telemetry while preserving Codex’s local task and tool-execution model. The important production gate isn’t whether a gateway can return text. It’s whether the gateway preserves the Responses behaviors that an agent workflow needs and whether the system team can operate the added infrastructure reliably.\n\nStart with direct IAM Identity Center access when native AWS controls meet the requirement. Add LiteLLM when customer-operated gateway policy justifies the operational work. Evaluate Portkey when a managed or hybrid operating model is the better organizational fit, and apply the same contract tests to every path.\n\n## Resources\n\n[Guidance for Codex on AWS implementation](https://github.com/openai-on-aws/guidance-codex/tree/feat/enterprise-gateway-readiness)[OpenAI on Amazon Bedrock](/bedrock/openai/)[Amazon Bedrock documentation](https://docs.aws.amazon.com/bedrock/)[LiteLLM on AWS quickstart](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/QUICKSTART_LLM_GATEWAY_LITELLM.md)[IAM Identity Center quickstart](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/QUICKSTART_NATIVE_AWS_ACCESS.md)[Portkey evaluation quickstart](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/QUICKSTART_LLM_GATEWAY_PORTKEY.md)[Production deployment guidance](https://github.com/openai-on-aws/guidance-codex/blob/feat/enterprise-gateway-readiness/docs/PRODUCTION_DEPLOYMENT.md)[Codex custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers)[Use Codex with Amazon Bedrock](https://learn.chatgpt.com/docs/amazon-bedrock)[Amazon ECS on AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html)[AWS IAM Identity Center documentation](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html)[LiteLLM proxy documentation](https://docs.litellm.ai/docs/simple_proxy)[Portkey Codex integration](https://docs.portkey.ai/docs/integrations/libraries/codex)[AWS Pricing Calculator](https://calculator.aws/)", "url": "https://wpnews.pro/news/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock/", "published_at": "2026-09-03 16:10:39+00:00", "updated_at": "2026-09-03 16:24:29.342886+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-infrastructure", "ai-tools"], "entities": ["Amazon Web Services", "OpenAI", "Codex", "LiteLLM", "Amazon Elastic Container Service", "Amazon Bedrock", "Amazon RDS for PostgreSQL", "AWS Secrets Manager"], "alternates": {"html": "https://wpnews.pro/news/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock", "markdown": "https://wpnews.pro/news/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock.md", "text": "https://wpnews.pro/news/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock.txt", "jsonld": "https://wpnews.pro/news/set-up-openai-chatgpt-codex-with-litellm-on-amazon-ecs-and-amazon-bedrock.jsonld"}}