In today’s multi-cloud AI landscape, organizations often find themselves juggling multiple LLM providers — OpenAI, Anthropic Claude via Azure AI Foundry, Cohere for embeddings, and more. Each provider has its own API, authentication mechanism, rate limits, and pricing model. Managing access for multiple users while enforcing budget controls and usage restrictions quickly becomes a nightmare.
Enter LiteLLM: a unified proxy that acts as a single gateway to 100+ LLM providers. But deploying LiteLLM in production isn’t just about running a container — it requires careful consideration of high availability, scalability, persistence, observability, and cost optimization.
In this article, I’ll walk you through our production-grade deployment of LiteLLM on AWS EKS, managed via ArgoCD for GitOps. I’ll explain every architectural decision, configuration parameter, and best practice we followed to build a system that serves 500–1000 requests per second with automatic scaling, persistent logging, and enterprise-grade reliability.
Our data science team Lab Enablement (LENA) group faced several challenges:
1. Multi-Provider Chaos: We used Azure AI Foundry for Claude models and Cohere v3 for embeddings. Each required separate authentication, SDKs, and management.
2. No Central Budget Control: Azure AI Foundry doesn’t provide granular budget restrictions. Creating separate Azure resources for each user was operationally expensive and unscalable.
3. User Management Nightmare: Provisioning API keys, tracking usage, and revoking access required manual intervention.
4. Cost Visibility Gap: No unified view of spending across providers made it impossible to optimize costs or forecast budgets.
**LiteLLM solved all of these problems **by providing:
Our deployment consists of four core components orchestrated on AWS EKS:
Kubernetes-Native: Leverages EKS for orchestration, scaling, and self-healing
GitOps-Driven: ArgoCD ensures declarative state, audit trails, and rollback capability
Highly Available: Pod anti-affinity, HPA, and multiple replicas prevent single points of failure
Cost-Optimized: Redis caching and batch database writes reduce infrastructure load
Secure: Internal ALB, VPC-only access, TLS termination, no public exposure
Why ghcr.io/berriai/litellm-database:v1.86.2?
We chose the database-enabled image because:
Deployment Configuration
apiVersion: apps/v1kind: Deploymentmetadata: name: litellm namespace: litellm-ns labels: app: litellmspec: replicas: 1 # HPA will scale this dynamically revisionHistoryLimit: 10 selector: matchLabels: app: litellm strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 # Zero-downtime deployments template: metadata: labels: app: litellm spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: litellm topologyKey: kubernetes.io/hostname containers: - name: litellm image: ghcr.io/berriai/litellm-database:v1.86.2 imagePullPolicy: IfNotPresent args: - "--config" - "/etc/litellm/config.yaml" - "--port" - "4000" - "--num_workers" - "1" # Single worker per pod for predictable resource usage - recomended ports: - name: http containerPort: 4000 env: - name: POSTGRES_USER valueFrom: configMapKeyRef: name: litellm-pg-config key: POSTGRES_USER - name: POSTGRES_DB valueFrom: configMapKeyRef: name: litellm-pg-config key: POSTGRES_DB - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: litellm-pg-secret key: POSTGRES_PASSWORD - name: DATABASE_URL value: "postgresql://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@litellm-pg-svc.litellm-ns.svc.cluster.local:5432/$(POSTGRES_DB)" - name: MASTER_KEY valueFrom: secretKeyRef: name: litellm-secrets key: MASTER_KEY - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: litellm-redis-secret key: REDIS_PASSWORD - name: LITELLM_MODE value: "PRODUCTION" - name: LITELLM_LOG value: "ERROR" # Log only errors to reduce noise - name: USE_PRISMA_MIGRATE value: "True" # Auto-apply database migrations on startup - name: DOCS_URL value: "/docs" - name: ROOT_REDIRECT_URL value: "/ui" # Redirect root to admin UI volumeMounts: - name: config mountPath: /etc/litellm readOnly: true resources: requests: cpu: "1500m" memory: "3Gi" limits: cpu: "1500m" memory: "3Gi" livenessProbe: httpGet: path: /health/liveliness port: 4000 initialDelaySeconds: 60 # Allow time for Prisma migration periodSeconds: 30 failureThreshold: 5 # Tolerates 2.5 minutes of failures readinessProbe: httpGet: path: /health/readiness port: 4000 initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 5 volumes: - name: config configMap: name: litellm-config items: - key: config.yaml path: config.yaml
Key Design Decisions
affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: litellm topologyKey: kubernetes.io/hostname
Why? If one EKS node fails, we want LiteLLM pods on other nodes to keep serving traffic. The preferred (soft) constraint means Kubernetes will try to spread pods but won’t block scheduling if impossible.
Alternative Considered: ‘requiredDuringSchedulingIgnoredDuringExecution` (hard constraint) was too strict for our small cluster — it would prevent scaling if only one node had capacity.
2. Resource Limits = Requests (No Burstability)
resources: requests: cpu: "1500m" memory: "3Gi" limits: cpu: "1500m" memory: "3Gi"
Why?
- Health Check Tuning
livenessProbe: httpGet: path: /health/liveliness port: 4000 initialDelaySeconds: 60 # Critical: Allow Prisma migration to complete periodSeconds: 30 failureThreshold: 5readinessProbe: httpGet: path: /health/readiness port: 4000 initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 5
Why 60s initial delay for liveness?
Why 5 failure thresholds?
Lesson Learned: Initially, we set “initialDelaySeconds: 30”, which caused 40% of deployments to fail during migrations. Increasing to 60s achieved 100% success rate.
apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: litellm-hpa namespace: litellm-nsspec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: litellm minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 behavior: scaleUp: stabilizationWindowSeconds: 120 policies: - type: Pods value: 1 periodSeconds: 120 selectPolicy: Max scaleDown: stabilizationWindowSeconds: 120 policies: - type: Pods value: 1 periodSeconds: 120 selectPolicy: Max
Why These Thresholds?
At 60% CPU, we observed response latencies starting to creep above 200ms. Scaling at 60% keeps p95 latency under 150ms.
**2. Memory: 80% Utilization **Memory usage is more predictable:
Scaling at 80% (2.4 Gi) provides comfortable headroom.
3. Stabilization Windows: 120 Seconds
Scale-Up: Prevents “flapping” during brief traffic spikes (e.g., batch inference jobs). HPA waits 2 minutes before adding a pod, filtering out transient load.
Scale-Down: Avoids premature pod removal. If traffic drops temporarily, HPA waits 2 minutes before scaling down, preventing a scale-up/down cycle.
Real-World Impact: During a traffic spike from 20 → 80 rps:
apiVersion: v1kind: ConfigMapmetadata: name: litellm-config namespace: litellm-nsdata: config.yaml: | general_settings: master_key: os.environ/MASTER_KEY database_url: os.environ/DATABASE_URL store_model_in_db: true use_redis_transaction_buffer: true block_robots: true # Prevent search engine indexing background_health_checks: true health_check_interval: 300 # Check model endpoints every 5 minutes maximum_spend_logs_retention_period: "60d" maximum_spend_logs_retention_interval: "1d" maximum_spend_logs_cleanup_cron: "0 4 * * *" # Daily cleanup at 4 AM user_api_key_cache_ttl: 600 # Cache API key lookups for 10 minutes database_connection_pool_limit: 20 database_connection_timeout: 60 proxy_batch_write_at: 30 # Batch database writes every 30 seconds litellm_settings: json_logs: true # Structured logging for Grafana/CloudWatch cache: true set_verbose: false # Disable verbose logs (reduces noise) request_timeout: 600 # 10 minutes for long-running model requests cache_params: type: redis host: redis-service port: 6379 password: os.environ/REDIS_PASSWORD ttl: 600 # Cache responses for 10 minutes namespace: "litellm.caching" router_settings: routing_strategy: simple-shuffle redis_host: redis-service redis_password: os.environ/REDIS_PASSWORD redis_port: 6379 cooldown_time: 10 # Retry failed models after 10 seconds
- Database Connection Pooling: 20 Connections
database_connection_pool_limit: 20
PostgreSQL’s default max_connections is 100. With 5 HPA replicas max: 5 pods × 20 connections = 100 total. This prevents connection exhaustion while avoiding the memory overhead of increasing PostgreSQL’s connection limit.
- Batch Writing: 30-Second Intervals
proxy_batch_write_at: 30
Groups 30 seconds of request logs into a single transaction, reducing PostgreSQL CPU by 60% (from 3,000 transactions/min to 2 transactions/min).
Trade-off: up to 30 seconds of logs lost if a pod crashes, which is acceptable for analytics workloads.
- Caching Strategy: 10-Minute TTL
user_api_key_cache_ttl: 600 # API key validationcache_params.ttl: 600 # Response caching
API Key: Reduces database load from 50 QPS (cold cache) to ~0.008 QPS (warm cache) for 5 unique users.
- Spend Log Retention: 60 Days
maximum_spend_logs_retention_period: "60d"maximum_spend_logs_cleanup_cron: "0 4 * * *"
Retains 2 months of spend data for financial auditing. Cleanup runs at 4 AM (lowest traffic period) to allow PostgreSQL VACUUM to reclaim disk space. At 50 rps, 60 days of logs consume ~129 GB (80% of our 100 Gi EBS volume).
- Routing Strategy: Simple Shuffle
routing_strategy: simple-shuffle
Randomly distributes requests across model deployments. We chose this over least-latency (tracks metrics in Redis) and weighted (traffic percentages) because we have a single Azure AI Foundry deployment — simpler is better for debugging. Switch to least-latency when adding multi-region redundancy.
apiVersion: apps/v1kind: StatefulSetmetadata: name: litellm-pg-statefulset namespace: litellm-nsspec: serviceName: litellm-pg-headless-svc replicas: 1 selector: matchLabels: app: litellm-pg template: metadata: labels: app: litellm-pg spec: containers: - name: postgres image: postgres:17 imagePullPolicy: IfNotPresent ports: - name: postgres containerPort: 5432 volumeMounts: - name: litellm-pg-data mountPath: /var/lib/postgresql/data env: - name: POSTGRES_USER valueFrom: configMapKeyRef: name: litellm-pg-config key: POSTGRES_USER - name: POSTGRES_DB valueFrom: configMapKeyRef: name: litellm-pg-config key: POSTGRES_DB - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: litellm-pg-secret key: POSTGRES_PASSWORD - name: PGDATA value: /var/lib/postgresql/data/pgdata resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "2000m" memory: "8Gi" readinessProbe: exec: command: - /bin/sh - -c - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: exec: command: - /bin/sh - -c - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" initialDelaySeconds: 30 periodSeconds: 10 volumeClaimTemplates: - metadata: name: litellm-pg-data spec: accessModes: - ReadWriteOnce storageClassName: ebs-sc resources: requests: storage: 100Gi
PostgreSQL 17 delivers 20% faster bulk INSERT’s (critical for batch writes), better JSONB indexing for metadata storage, and incremental VACUUM that reduces write amplification by 30%.
RDS vs Self-Managed: RDS db.t3.medium costs $50/month vs. $12/month self-managed on EKS. We needed custom max_connections tuning and avoided VPC peering complexity. Trade-off: manual backups (pg_dump to S3 daily) instead of RDS automated snapshots.
StatefulSets provide stable DNS names, ordered deployment (critical for single-replica databases), and persistent volume binding that survives pod restarts. We use a headless service (ClusterIP: None) for direct pod connections, eliminating load balancer overhead.
Resource Allocation: Wide Gap Between Requests and Limits
resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "2000m" memory: "8Gi"
PostgreSQL is idle 90% of the time due to 30-second batch writes, justifying low requests (250m CPU fits on smaller nodes). High limits accommodate burst capacity during batch writes (1.5 CPU, 3 Gi) and daily VACUUM operations (up to 6 Gi memory). Measured performance: P50 CPU at 150m, P95 at 1,200m during writes.
Setting requests = limits would waste EKS node capacity during idle periods.
apiVersion: apps/v1kind: Deploymentmetadata: name: litellm-redis namespace: litellm-nsspec: replicas: 1 selector: matchLabels: app: litellm-redis template: metadata: labels: app: litellm-redis spec: containers: - name: redis image: redis:8-alpine imagePullPolicy: IfNotPresent command: ["sh", "-c"] args: - 'exec redis-server --requirepass "$REDIS_PASSWORD" --appendonly yes' ports: - name: redis containerPort: 6379 env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: litellm-redis-secret key: REDIS_PASSWORD - name: REDISCLI_AUTH valueFrom: secretKeyRef: name: litellm-redis-secret key: REDIS_PASSWORD volumeMounts: - name: litellm-redis-pvc mountPath: /data resources: requests: cpu: "200m" memory: "512Mi" limits: cpu: "1000m" memory: "2Gi" readinessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 5 periodSeconds: 10 timeoutSeconds: 3 livenessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 15 periodSeconds: 20 timeoutSeconds: 3 volumes: - name: litellm-redis-pvc persistentVolumeClaim: claimName: litellm-redis-pvc
LiteLLM makes Redis essential for three critical functions:
1. Response Caching
Identical prompts (same model, same input) return cached responses instead of hitting the upstream LLM provider. This is crucial for:
Our 10-minute TTL achieved a 40% cache hit rate.
2. API Key Validation Caching
Every LiteLLM request validates the user’s API key against PostgreSQL. Without caching, this means 50 database queries per second. Redis caches API key lookups for 10 minutes, reducing database load by 99.98% (from 50 QPS to ~0.01 QPS).
3. Router State Management**
LiteLLM tracks model endpoint health (which deployments are responding, which are failing) in Redis. This enables:
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: litellm-ingress namespace: litellm-ns annotations: alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:01234567890:certificate/a1639c-6c2c-4e60-9211-59cd0a alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}, {"HTTPS":443}]' alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/healthcheck-path: /health/readiness alb.ingress.kubernetes.io/healthcheck-interval-seconds: "30" alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "10" alb.ingress.kubernetes.io/healthy-threshold-count: "2" alb.ingress.kubernetes.io/unhealthy-threshold-count: "3"spec: ingressClassName: alb rules: - host: litellm.data-lab.io http: paths: - path: / pathType: Prefix backend: service: name: litellm-svc port: number: 4000
Creates a VPC-internal ALB (no public IP). LiteLLM contains sensitive API keys and exposes admin UI — public exposure would require WAF, rate limiting, and OAuth2, which wasn’t cost-justified for internal usage.
Target Type: IP
alb.ingress.kubernetes.io/target-type: ip
Routes traffic directly to pod IPs, eliminating kube-proxy overhead (~5ms latency reduction) and enabling faster health checks. IP targets are free; instance targets charge per registration.
Health Check Configuration
alb.ingress.kubernetes.io/healthcheck-path: /health/readinessalb.ingress.kubernetes.io/healthcheck-interval-seconds: "30"
Uses /health/readiness (pod can serve traffic) vs. /health/liveliness (pod process running). Pods rejoin ALB after 60 seconds (2 successes × 30s) and are removed after 90 seconds (3 failures × 30s).
Lesson Learned: 10-second intervals caused false negatives during brief database blips. 30-second intervals tolerate transient issues.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: litellm namespace: argocdspec: project: litellm source: repoURL: https://gitlab.com/devops1952445/argocd-playground.git targetRevision: main # Production branch path: src/apps/litellm destination: server: https://kubernetes.default.svc namespace: litellm-ns syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true
Why ArgoCD for LiteLLM?
-
Declarative State: Git is source of truth, not
kubectl applycommands -
Audit Trail: Every change tracked via Git commits (who, what, when, why)
-
Rollback:
git revertinstantly rolls back deployments -
Multi-Environment: Same manifests for dev/staging/prod with Kustomize overlays
-
Self-Healing: If someone
kubectl deletea pod, ArgoCD recreates it within 3 minutes
Auto-Sync with Prune and Self-Heal
syncPolicy:automated:prune: true # Delete resources removed from GitselfHeal: true # Revert manual kubectl changes
Prune: If we delete litellm_redis_deployment.yml from Git, ArgoCD deletes the Redis deployment from the cluster.
Self-Heal: If an engineer runs kubectl scale deployment/litellm — replicas=10, ArgoCD reverts to replicas: 1 within 3 minutes.
Is This Too Aggressive?
Concern: Self-heal might revert emergency hotfixes during incidents.
Mitigation:
-
Emergency Override: ArgoCD sync during incidents:
kubectl patch app litellm -n argocd -p ‘{“spec”:{“syncPolicy”:{“automated”:null}}}’ -
Sync Wave: Critical resources (database, Redis) have
argocd.argoproj.io/sync-wave: “0”, LiteLLM pods have wave”1". Database won’t self-heal during LiteLLM rollouts. -
Notification: ArgoCD webhooks to Slack alert on any manual changes, prompting engineers to commit fixes to Git.
Real-World Incident: An engineer increased PostgreSQL memory limits to debug OOM. Self-heal reverted the change, OOM recurred. We now:
Deploying a production-grade LiteLLM proxy on AWS EKS requires more than just running a container. It demands thoughtful decisions about:
Our deployment serves 500–1000 requests/second across 5 HPA replicas, with p95 latency under 150ms and 99.9% uptime over 1months. The architecture scales to 200 rps without infrastructure changes.
Key Takeaways:
-
Test HPA behavior under load before production
-
Enable Prisma migrations to avoid schema mismatch errors
-
Monitor database connections to prevent exhaustion
-
Use structured JSON logs for easier debugging
-
Cache aggressively if workloads are repetitive
If you’re deploying LiteLLM in your organization, start with this architecture as a foundation. Adjust resource limits based on your load testing, and remember: GitOps with ArgoCD is your safety net — every change is reversible, audited, and reproducible.
You can also get the complete source code from my public gitlab project,https://gitlab.com/KnowledgeSharing1/lite-llm-hpa.git
Production LiteLLM on AWS EKS: High Availability with GitOps was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.