{"slug": "the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran", "title": "🌟 The CI Gate Rejected the Terraform Change—but the LLM Still Ran", "summary": "A developer built an AI-powered Terraform review agent combining Terrascan, GitHub Actions, AWS Lambda, and Gemini, but discovered that the LLM was invoked even when a deterministic policy should have rejected the change. To enforce the correct control boundary, the developer used AgentInspect to add a deterministic CI contract that ensures the LLM path executes zero times for blocked changes.", "body_md": "How I added a deterministic trace contract to an AI Terraform reviewer so rejected infrastructure changes stop before the model is invoked.\n\nI recently built an [AI-powered Terraform review agent](https://github.com/Pravesh-Sudha/ai-devops-agent/tree/main/terraform-review-agent) that combines Terrascan, GitHub Actions, AWS Lambda and Gemini.\n\nThe workflow is straightforward:\n\nThe project worked. A risky change produced `REJECT`\n\n, and GitHub Actions failed the pull request as expected.\n\nBut while reviewing the execution path, I noticed an important problem: **the correct final verdict did not prove that the control was enforced at the correct boundary.**\n\nThe pipeline rejected the change—but the LLM had already run.\n\nThat distinction matters in DevOps. A policy that says “do not continue” should stop the next action. It should not merely ask the next action to agree that the request should have been stopped.\n\nThis article shows how I used AgentInspect to make that path visible and add a deterministic CI contract around it.\n\nDisclosure:I tested AgentInspect independently in the workflow described here. The maintainer reviewed the AgentInspect commands for technical accuracy; the conclusions are my own. AgentInspect did not replace Terrascan, GitHub Actions, AWS controls or the application’s security policy.\n\nMy Terraform review project uses a practical serverless workflow:\n\n```\nTerraform pull request\n        ↓\nGitHub Actions\n        ↓\nTerrascan JSON report\n        ↓\nAWS Lambda\n        ↓\nGemini review\n        ↓\nAPPROVE | APPROVE_WITH_CHANGES | REJECT\n```\n\nThe Lambda function extracts relevant Terrascan findings and sends a bounded structure to Gemini. The prompt contains explicit decision rules:\n\nThe [GitHub Actions workflow](https://github.com/Pravesh-Sudha/ai-devops-agent/blob/main/.github/workflows/main.yml) then reads the returned verdict and exits with status `1`\n\nwhen the model returns `REJECT`\n\n.\n\nAt first, this looked like a security gate. The result was correct and the PR was blocked.\n\nThe problem was where the authority lived.\n\nIn the original [Lambda implementation](https://github.com/Pravesh-Sudha/ai-devops-agent/blob/main/terraform-review-agent/lambda/lambda_function.py), the sequence was effectively:\n\n```\nfindings = extract_relevant_findings(results)\nprompt = build_prompt(findings)\n\nai_review = call_gemini(prompt)\nverdict = extract_verdict(ai_review)\n```\n\nThe risk thresholds were described inside the prompt. They were not evaluated as a deterministic control before the provider call.\n\nThis creates three different concerns:\n\nThe third concern is the easiest one to miss in ordinary CI output.\n\nImagine a fixture containing a HIGH-severity public-access violation. A conventional test might assert:\n\n```\nexpect(result.verdict).toBe(\"REJECT\");\n```\n\nThat assertion passes whether the pipeline rejects before Gemini or calls Gemini and then accepts its rejection.\n\nThose paths are not equivalent:\n\n```\nDesired\nTerrascan → deterministic policy → REJECT → stop\n\nOriginal\nTerrascan → Gemini → parse response → REJECT → stop\n```\n\nThe final value is the same. The control boundary is different.\n\nFor the blocked test case, I wanted to assert a stronger invariant:\n\nTerrascan and the deterministic policy check must execute, and the LLM path must execute zero times.\n\nThat is an execution contract, not an answer-quality evaluation.\n\nThe existing Lambda is written in Python, while AgentInspect is a TypeScript-first toolkit. I did not pretend that it could automatically instrument the Python function.\n\nInstead, I added a small Node.js evidence runner at the CI boundary. It wraps the operations the pipeline owns: reading the scan, evaluating the policy and, only when appropriate, invoking the existing Lambda.\n\nFor this test, I used AgentInspect `6.17.4`\n\nand pinned the version so the CI behavior would not move underneath the experiment:\n\n```\nnpm install agent-inspect@6.17.4\n```\n\nThe simplified runner looks like this:\n\n``` js\nimport { inspectRun, step } from \"agent-inspect\";\n\nconst traceDir = \".agent-inspect/terraform-review\";\n\nfunction evaluatePolicy(report) {\n  const violations = report.violations ?? [];\n  const severities = violations.map((item) =>\n    String(item.severity ?? \"\").toUpperCase()\n  );\n\n  const highOrCritical = severities.some(\n    (severity) => severity === \"HIGH\" || severity === \"CRITICAL\"\n  );\n  const mediumCount = severities.filter(\n    (severity) => severity === \"MEDIUM\"\n  ).length;\n\n  if (highOrCritical || mediumCount >= 4) {\n    return { verdict: \"REJECT\", reason: \"risk-threshold\" };\n  }\n\n  if (mediumCount >= 1) {\n    return {\n      verdict: \"APPROVE_WITH_CHANGES\",\n      reason: \"medium-findings\"\n    };\n  }\n\n  return { verdict: \"APPROVE\", reason: \"low-or-info-only\" };\n}\n\nconst result = await inspectRun(\n  \"terraform-ai-review\",\n  async () => {\n    const report = await step.tool(\"terrascan\", () =>\n      readTerrascanReport(\"terrascan_report.json\")\n    );\n\n    const policy = await step.tool(\"evaluate_policy\", () =>\n      evaluatePolicy(report)\n    );\n\n    if (policy.verdict === \"REJECT\") {\n      return policy;\n    }\n\n    const review = await step.llm(\"gemini-2.5-flash\", () =>\n      invokeTerraformReviewLambda(report)\n    );\n\n    return {\n      ...review,\n      verdict: policy.verdict\n    };\n  },\n  {\n    traceDir,\n    silent: process.env.CI === \"true\",\n    metadata: {\n      workflow: \"terraform-ai-review\",\n      fixture: \"high-severity-public-access\"\n    }\n  }\n);\n\nconsole.log(result);\n```\n\nThis example deliberately keeps the policy small. The HTTPS rule needs its own structured input or a stable mapping from specific Terrascan findings. I would not implement it by searching arbitrary free text and call that deterministic.\n\nThe important change is architectural: **code owns the risk threshold and final CI authority; the model can provide explanation and remediation only after the deterministic gate allows that path.**\n\nI ran a controlled fixture representing a HIGH-severity finding and inspected the local trace:\n\n```\nnpx agent-inspect list --dir .agent-inspect/terraform-review\nnpx agent-inspect view <run-id> --dir .agent-inspect/terraform-review\n```\n\nBefore the short-circuit, the execution contained an LLM step:\n\n```\nterraform-ai-review\n├─ tool:terrascan              success\n└─ llm:gemini-2.5-flash       success\n```\n\nAfter moving policy enforcement ahead of the provider call, the rejected path became:\n\n```\nterraform-ai-review\n├─ tool:terrascan              success\n└─ tool:evaluate_policy        success\n```\n\nThe absence of the LLM step was now visible, but I did not want reviewers to verify it manually on every pull request. The next step was turning it into a deterministic check.\n\nAgentInspect supports deterministic checks over retained local traces. For this case, I used a JSON check configuration:\n\n```\n{\n  \"checks\": {\n    \"tool\": {\n      \"required\": [\"terrascan\", \"evaluate_policy\"]\n    },\n    \"llm\": {\n      \"maxCalls\": 0\n    }\n  }\n}\n```\n\nThen CI checks the trace produced by the blocked fixture:\n\n```\nnpx agent-inspect check .agent-inspect/terraform-review \\\n  --config blocked-path.check.json \\\n  --require-completed \\\n  --json\n```\n\nThe broken path fails because the model call count is greater than zero. The fixed path passes because:\n\n`terrascan`\n\nappeared.`evaluate_policy`\n\nappeared.This is the exact behavior I wanted from CI. It does not ask another model whether the trace looks safe. It evaluates a small, reproducible structural contract and returns a deterministic exit code.\n\nA CI failure without usable evidence usually starts another debugging cycle. To retain a bounded report, I added an artifact step:\n\n```\n- name: Run blocked-path regression\n  run: node scripts/run-blocked-policy-case.mjs\n\n- name: Check blocked-path contract\n  run: |\n    npx agent-inspect check .agent-inspect/terraform-review \\\n      --config blocked-path.check.json \\\n      --require-completed \\\n      --json > trace-contract-result.json\n\n- name: Build safe trace artifacts\n  if: always()\n  run: |\n    npx agent-inspect artifacts .agent-inspect/terraform-review \\\n      --output-dir ./agent-inspect-artifacts \\\n      --github-summary \"$GITHUB_STEP_SUMMARY\"\n\n- name: Upload review evidence\n  if: always()\n  uses: actions/upload-artifact@v4\n  with:\n    name: terraform-agent-trace\n    path: |\n      agent-inspect-artifacts/\n      trace-contract-result.json\n    retention-days: 14\n```\n\nAgentInspect creates the local report; GitHub Actions owns the upload and retention. For real pipeline data, I would still review the exact derived artifact before sharing it outside the repository. Redaction and safety scans are safeguards, not compliance certification.\n\nMy first implementation treated the final verdict as proof that the gate worked. It was only proof that the workflow ended with the expected string.\n\nThe stronger DevOps questions are:\n\nThis is where execution traces are useful. They provide evidence about the path rather than only the answer.\n\nThere is also a useful separation of responsibilities:\n\n| Layer | Responsibility |\n|---|---|\n| Terrascan | Detect infrastructure findings |\n| Deterministic policy code | Enforce explicit risk thresholds |\n| Gemini | Explain findings and suggest remediation on allowed paths |\n| AgentInspect | Record and check the execution path |\n| GitHub Actions | Enforce the build result and retain reviewed artifacts |\n\nAgentInspect did not make the Terraform deployment secure. It helped me verify whether my own control flow matched the policy I claimed to enforce.\n\nA green trace contract is narrow evidence. It does not prove that every Terraform rule is correct or that the deployed infrastructure is safe.\n\nThis workflow still needs:\n\nIt also does not prove that Gemini’s remediation advice is good. That requires separate evaluation.\n\nThe contract proves one specific invariant: **when deterministic policy rejects a Terraform change, the model path does not run.**\n\nAI can add useful context to DevOps workflows, but it should not own controls that can be expressed clearly in code.\n\nMy Terraform review agent already produced the expected rejection. AgentInspect showed me that the route to that answer was weaker than the answer itself suggested.\n\nMoving the risk threshold ahead of the LLM call gave the workflow a cleaner authority boundary. Adding a trace contract made that boundary reviewable in CI.\n\nThe lesson is simple:\n\nDo not test only what your agent returned. Test which actions it was allowed to take before returning it.", "url": "https://wpnews.pro/news/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran", "canonical_source": "https://dev.to/pravesh_sudha_3c2b0c2b5e0/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran-3hfg", "published_at": "2026-09-02 16:57:13+00:00", "updated_at": "2026-09-02 17:24:10.316602+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "ai-safety"], "entities": ["Terrascan", "GitHub Actions", "AWS Lambda", "Gemini", "AgentInspect", "Pravesh-Sudha"], "alternates": {"html": "https://wpnews.pro/news/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran", "markdown": "https://wpnews.pro/news/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran.md", "text": "https://wpnews.pro/news/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran.txt", "jsonld": "https://wpnews.pro/news/the-ci-gate-rejected-the-terraform-change-but-the-llm-still-ran.jsonld"}}