{"slug": "production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo", "title": "Production-Grade GitOps on AWS: Verified Release Promotion Across EKS Environments with Argo CD and Kargo", "summary": "A developer detailed a production-grade GitOps architecture on AWS that uses separate EKS clusters per environment, ECR as the registry, OIDC federation instead of static credentials, and Kargo to automate release promotion between environments. The setup includes automated verification gates backed by real metrics before promotion, and emphasizes promoting image digests rather than tags to ensure byte-identical artifacts reach production.", "body_md": "There's a moment every platform team hits: Argo CD is syncing beautifully, deployments are declarative, and yet moving a release from dev to staging to production still means a human editing a values file and hoping. The deployment problem is solved; the *promotion* problem isn't.\n\nIn this article, I want to walk through how we close that gap on AWS: separate EKS clusters per environment, ECR as the registry, OIDC federation everywhere instead of static credentials, ApplicationSets instead of copy-pasted manifests, and — critically — automated verification gates backed by real metrics before anything gets promoted.\n\nThis is the architecture my team runs variations of in production, and the reasoning behind each decision.\n\nBefore touching any tool, it's worth being precise about what \"promotion\" means in a GitOps world.\n\nArgo CD's job is narrow and it does it extremely well: make the cluster match what Git says. It does not decide *when* a new version should move from dev to staging to production. That decision — historically made by a human editing a values file, or by a brittle CI job running `sed`\n\nagainst your repo — is the promotion problem.\n\nKargo exists to own exactly that gap. It watches your artifact sources (ECR, Git, Helm repos), models each environment as a **Stage**, packages new artifact versions as **Freight**, and promotes that Freight between stages by writing commits to your GitOps repo. Argo CD then does what it always does: reconcile.\n\nThe result is a clean separation of concerns:\n\nEvery environment's state is a commit. Every promotion is auditable. Rollback is `git revert`\n\n. That's the whole philosophy.\n\nThe single biggest gap between tutorial GitOps and production GitOps is environment isolation.\n\nNamespaces on a shared cluster do not give you blast-radius isolation. A misbehaving controller, a noisy neighbor exhausting node resources, a cluster upgrade gone wrong — all of these take dev and prod down together. In a real AWS setup:\n\nThat last point deserves emphasis: **the artifact that reaches production must be byte-identical to the one validated in staging**. Rebuilding \"the same\" image per environment silently invalidates everything your pipeline verified. Promote digests, not tags.\n\nIf your GitHub Actions workflow authenticates to AWS with an `AWS_ACCESS_KEY_ID`\n\nstored in repository secrets, that's a standing credential waiting to leak. GitHub's OIDC provider lets each workflow run exchange a short-lived, cryptographically verifiable token for a scoped IAM role session:\n\n```\npermissions:\n  id-token: write\n  contents: read\n\nsteps:\n  - uses: aws-actions/configure-aws-credentials@v4\n    with:\n      role-to-assume: arn:aws:iam::<SHARED_ACCT>:role/gha-ecr-push-frontend\n      aws-region: eu-west-1\n\n  - uses: aws-actions/amazon-ecr-login@v2\n\n  - name: Build, tag, push\n    run: |\n      IMAGE=$ECR_REGISTRY/frontend\n      docker build -t $IMAGE:$VERSION .\n      docker push $IMAGE:$VERSION\n```\n\nThe IAM role's trust policy pins the exact repository and branch (`repo:digitalzone/frontend:ref:refs/heads/main`\n\n), so a compromised fork or a rogue workflow in another repo simply cannot assume it. Scope one role per service, with push rights to only that service's ECR repository.\n\nTwo additions that cost little and pay off in audit season:\n\nSemantic tags (`v1.4.2`\n\n) remain useful for humans, but your promotion machinery should carry the **digest** end to end.\n\nI follow a polyrepo model for application code — each microservice owns its repo, its CI, and its release cadence — with a **single GitOps repo** as the deployment control plane:\n\n```\nplatform-gitops/\n├── charts/                  # One Helm chart per service\n│   └── frontend/\n├── envs/\n│   ├── dev/frontend/values.yaml\n│   ├── staging/frontend/values.yaml\n│   └── prod/frontend/values.yaml\n├── argocd/\n│   └── applicationsets/\n└── kargo/\n    └── frontend/            # Warehouse, Stages, PromotionTasks\n```\n\nA rule I hold firm on: **environments are directories, not branches**. Long-lived `dev`\n\n/`staging`\n\n/`prod`\n\nbranches turn every promotion into a merge with drift and conflict potential. A single `main`\n\nbranch with per-environment values directories means the diff between environments is always visible in one `git diff`\n\n— which is exactly what you want at 2 a.m. during an incident.\n\nInstead of hand-maintaining an Argo CD Application per service per environment (that's N×M YAML files that will drift), generate them with an **ApplicationSet**:\n\n```\napiVersion: argoproj.io/v1alpha1\nkind: ApplicationSet\nmetadata:\n  name: services\nspec:\n  generators:\n    - matrix:\n        generators:\n          - git:\n              repoURL: https://github.com/digitalzone/platform-gitops\n              directories:\n                - path: envs/*/*\n          - list:\n              elements:\n                - env: dev\n                  cluster: https://dev-cluster-endpoint\n                - env: staging\n                  cluster: https://staging-cluster-endpoint\n                - env: prod\n                  cluster: https://prod-cluster-endpoint\n  template:\n    metadata:\n      name: '{{path.basename}}-{{env}}'\n    spec:\n      project: platform\n      source:\n        repoURL: https://github.com/digitalzone/platform-gitops\n        path: charts/{{path.basename}}\n        helm:\n          valueFiles:\n            - ../../envs/{{env}}/{{path.basename}}/values.yaml\n      destination:\n        server: '{{cluster}}'\n        namespace: '{{path.basename}}'\n      syncPolicy:\n        automated:\n          prune: true\n          selfHeal: true\n```\n\nAdding a new service to every environment becomes: add a chart, add three values files, done. No new Application YAML, ever.\n\nOne deliberate asymmetry: I keep `selfHeal`\n\nand `prune`\n\nfully automated in dev and staging, but in production I pair automated sync with **sync windows** and require promotion (not sync) to be the gated step. The cluster should always converge on Git — the control point is what gets *into* Git.\n\nKargo's **Warehouse** polls ECR for new images. On EKS, don't feed it credentials — bind its controller service account to an IAM role via **IRSA** (or EKS Pod Identity) with read-only ECR permissions:\n\n```\napiVersion: kargo.akuity.io/v1alpha1\nkind: Warehouse\nmetadata:\n  name: frontend\n  namespace: kargo-platform\nspec:\n  subscriptions:\n    - image:\n        repoURL: <ACCT>.dkr.ecr.eu-west-1.amazonaws.com/frontend\n        semverConstraint: \">=1.0.0\"\n        strictSemvers: true\n```\n\nEach environment is a **Stage**. Dev subscribes directly to the warehouse; staging subscribes to dev's verified freight; prod subscribes to staging's:\n\n```\napiVersion: kargo.akuity.io/v1alpha1\nkind: Stage\nmetadata:\n  name: staging\n  namespace: kargo-platform\nspec:\n  requestedFreight:\n    - origin:\n        kind: Warehouse\n        name: frontend\n      sources:\n        stages: [dev]        # Only freight verified in dev is eligible\n  promotionTemplate:\n    spec:\n      steps:\n        - uses: git-clone\n        - uses: yaml-update\n          config:\n            path: envs/staging/frontend/values.yaml\n            updates:\n              - key: image.tag\n                value: ${{ imageFrom(\"...frontend\").Tag }}\n        - uses: git-commit\n        - uses: git-push\n        - uses: argocd-update\n```\n\nThat `sources.stages: [dev]`\n\nline is the promotion graph. An image physically cannot reach staging without having passed dev, and cannot reach prod without passing staging. The pipeline topology is declarative and enforced — not a convention someone can forget under deadline pressure.\n\nAutomated promotion without automated verification is just automated blast radius. This is where most write-ups wave their hands, and where the real engineering lives.\n\nKargo integrates with Argo Rollouts' **AnalysisTemplates**, which means each stage can run metric-backed verification against your observability stack before its freight becomes eligible for the next stage. We run the LGTM stack, so verification queries Prometheus directly:\n\n```\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n  name: frontend-health\nspec:\n  metrics:\n    - name: error-rate\n      interval: 1m\n      count: 10\n      failureLimit: 1\n      provider:\n        prometheus:\n          address: http://prometheus.monitoring:9090\n          query: |\n            sum(rate(http_requests_total{app=\"frontend\",status=~\"5..\"}[5m]))\n            /\n            sum(rate(http_requests_total{app=\"frontend\"}[5m]))\n      successCondition: result[0] < 0.01\n    - name: p99-latency\n      interval: 1m\n      count: 10\n      provider:\n        prometheus:\n          address: http://prometheus.monitoring:9090\n          query: |\n            histogram_quantile(0.99,\n              sum(rate(http_request_duration_seconds_bucket{app=\"frontend\"}[5m])) by (le))\n      successCondition: result[0] < 0.5\n```\n\nA release now soaks in dev, gets measured against real SLO-shaped queries for ten minutes, and only then becomes promotable. The same pattern in staging, ideally alongside synthetic traffic or smoke tests, gives you two independent metric-verified gates before production.\n\nFor the prod stage itself, pair the promotion with **Argo Rollouts canary steps** — shift 10% of traffic behind the ALB, re-run the same analysis against the canary subset, then progress. Verification and progressive delivery share templates, which keeps your definition of \"healthy\" in exactly one place.\n\nThe final gate — staging to prod — stays **manual by policy** in our setup: a human clicks approve in Kargo's UI (or via the CLI, from Slack). Not because the automation isn't trusted, but because production timing is a business decision. During a regional sales event on our ticketing platform, \"verified and ready\" and \"deploy right now\" are very different statements.\n\n`v1.4.2`\n\nto ECR via OIDC.`envs/dev/frontend/values.yaml`\n\n, Argo CD syncs the dev cluster.`envs/prod/frontend/values.yaml`\n\n; Argo CD executes a canary rollout with in-flight analysis; full traffic shift on success.Mean time from merge to verified-in-staging: about twenty-five minutes, with zero human involvement. Mean human effort per production release: one approval click.\n\n**Promote digests, pin everything.** Tags are for humans; digests are for machines. Immutable ECR tags plus digest-based promotion eliminates an entire category of \"but it worked in staging\" incidents.\n\n**Verification queries are product code.** Treat AnalysisTemplates with the same review rigor as application code. A query with a subtly wrong label selector is a green light that means nothing.\n\n**Keep the emergency path inside GitOps.** When you need to ship a hotfix at 3 a.m., the answer is Kargo's manual promotion of a specific freight — not `kubectl edit`\n\n. If your break-glass procedure bypasses Git, your audit trail is fiction precisely when you'll need it most.\n\n**Budget for ECR lifecycle policies early.** A warehouse polling a repository with ten thousand untagged images is slow and expensive. Expire untagged images aggressively; keep the last N releases per service.\n\n**Don't over-gate.** Every manual approval you add is a place where releases queue and context evaporates. Gate production. Let metrics gate everything else.\n\nThe tooling here — Actions, Argo CD, Helm, Kargo — is the same stack you'll find in a weekend tutorial. What separates a demo from a platform is everything around it: account-level isolation, federated identity end to end, generated rather than hand-written manifests, and promotion gates that measure reality instead of assuming it.\n\nGit as the source of truth is the principle. Verified, auditable, boring promotions are the payoff.\n\n*How is your team handling environment promotion today — Git commits, CI scripts, or still a human with kubectl? I'd genuinely like to hear what's working (and what isn't) in the comments.*", "url": "https://wpnews.pro/news/production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo", "canonical_source": "https://dev.to/aws-builders/production-grade-gitops-on-aws-verified-release-promotion-across-eks-environments-with-argo-cd-and-1204", "published_at": "2026-08-25 17:54:44+00:00", "updated_at": "2026-08-25 18:14:51.222676+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "ai-infrastructure"], "entities": ["AWS", "EKS", "ECR", "Argo CD", "Kargo", "GitHub Actions", "OIDC", "IAM"], "alternates": {"html": "https://wpnews.pro/news/production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo", "markdown": "https://wpnews.pro/news/production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo.md", "text": "https://wpnews.pro/news/production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo.txt", "jsonld": "https://wpnews.pro/news/production-grade-gitops-on-aws-verified-release-promotion-across-eks-with-argo.jsonld"}}