🌟 The CI Gate Rejected the Terraform Change—but the LLM Still Ran

A developer built an AI-powered Terraform review agent combining Terrascan, GitHub Actions, AWS Lambda, and Gemini, but discovered that the LLM was invoked even when a deterministic policy should have rejected the change. To enforce the correct control boundary, the developer used AgentInspect to add a deterministic CI contract that ensures the LLM path executes zero times for blocked changes.

How I added a deterministic trace contract to an AI Terraform reviewer so rejected infrastructure changes stop before the model is invoked. I recently built an AI-powered Terraform review agent https://github.com/Pravesh-Sudha/ai-devops-agent/tree/main/terraform-review-agent that combines Terrascan, GitHub Actions, AWS Lambda and Gemini. The workflow is straightforward: The project worked. A risky change produced REJECT , and GitHub Actions failed the pull request as expected. But while reviewing the execution path, I noticed an important problem: the correct final verdict did not prove that the control was enforced at the correct boundary. The pipeline rejected the change—but the LLM had already run. That distinction matters in DevOps. A policy that says “do not continue” should stop the next action. It should not merely ask the next action to agree that the request should have been stopped. This article shows how I used AgentInspect to make that path visible and add a deterministic CI contract around it. Disclosure:I tested AgentInspect independently in the workflow described here. The maintainer reviewed the AgentInspect commands for technical accuracy; the conclusions are my own. AgentInspect did not replace Terrascan, GitHub Actions, AWS controls or the application’s security policy. My Terraform review project uses a practical serverless workflow: Terraform pull request ↓ GitHub Actions ↓ Terrascan JSON report ↓ AWS Lambda ↓ Gemini review ↓ APPROVE | APPROVE WITH CHANGES | REJECT The Lambda function extracts relevant Terrascan findings and sends a bounded structure to Gemini. The prompt contains explicit decision rules: The GitHub Actions workflow https://github.com/Pravesh-Sudha/ai-devops-agent/blob/main/.github/workflows/main.yml then reads the returned verdict and exits with status 1 when the model returns REJECT . At first, this looked like a security gate. The result was correct and the PR was blocked. The problem was where the authority lived. In the original Lambda implementation https://github.com/Pravesh-Sudha/ai-devops-agent/blob/main/terraform-review-agent/lambda/lambda function.py , the sequence was effectively: findings = extract relevant findings results prompt = build prompt findings ai review = call gemini prompt verdict = extract verdict ai review The risk thresholds were described inside the prompt. They were not evaluated as a deterministic control before the provider call. This creates three different concerns: The third concern is the easiest one to miss in ordinary CI output. Imagine a fixture containing a HIGH-severity public-access violation. A conventional test might assert: expect result.verdict .toBe "REJECT" ; That assertion passes whether the pipeline rejects before Gemini or calls Gemini and then accepts its rejection. Those paths are not equivalent: Desired Terrascan → deterministic policy → REJECT → stop Original Terrascan → Gemini → parse response → REJECT → stop The final value is the same. The control boundary is different. For the blocked test case, I wanted to assert a stronger invariant: Terrascan and the deterministic policy check must execute, and the LLM path must execute zero times. That is an execution contract, not an answer-quality evaluation. The existing Lambda is written in Python, while AgentInspect is a TypeScript-first toolkit. I did not pretend that it could automatically instrument the Python function. Instead, I added a small Node.js evidence runner at the CI boundary. It wraps the operations the pipeline owns: reading the scan, evaluating the policy and, only when appropriate, invoking the existing Lambda. For this test, I used AgentInspect 6.17.4 and pinned the version so the CI behavior would not move underneath the experiment: npm install agent-inspect@6.17.4 The simplified runner looks like this: js import { inspectRun, step } from "agent-inspect"; const traceDir = ".agent-inspect/terraform-review"; function evaluatePolicy report { const violations = report.violations ?? ; const severities = violations.map item = String item.severity ?? "" .toUpperCase ; const highOrCritical = severities.some severity = severity === "HIGH" || severity === "CRITICAL" ; const mediumCount = severities.filter severity = severity === "MEDIUM" .length; if highOrCritical || mediumCount = 4 { return { verdict: "REJECT", reason: "risk-threshold" }; } if mediumCount = 1 { return { verdict: "APPROVE WITH CHANGES", reason: "medium-findings" }; } return { verdict: "APPROVE", reason: "low-or-info-only" }; } const result = await inspectRun "terraform-ai-review", async = { const report = await step.tool "terrascan", = readTerrascanReport "terrascan report.json" ; const policy = await step.tool "evaluate policy", = evaluatePolicy report ; if policy.verdict === "REJECT" { return policy; } const review = await step.llm "gemini-2.5-flash", = invokeTerraformReviewLambda report ; return { ...review, verdict: policy.verdict }; }, { traceDir, silent: process.env.CI === "true", metadata: { workflow: "terraform-ai-review", fixture: "high-severity-public-access" } } ; console.log result ; This example deliberately keeps the policy small. The HTTPS rule needs its own structured input or a stable mapping from specific Terrascan findings. I would not implement it by searching arbitrary free text and call that deterministic. The important change is architectural: code owns the risk threshold and final CI authority; the model can provide explanation and remediation only after the deterministic gate allows that path. I ran a controlled fixture representing a HIGH-severity finding and inspected the local trace: npx agent-inspect list --dir .agent-inspect/terraform-review npx agent-inspect view <run-id --dir .agent-inspect/terraform-review Before the short-circuit, the execution contained an LLM step: terraform-ai-review ├─ tool:terrascan success └─ llm:gemini-2.5-flash success After moving policy enforcement ahead of the provider call, the rejected path became: terraform-ai-review ├─ tool:terrascan success └─ tool:evaluate policy success The absence of the LLM step was now visible, but I did not want reviewers to verify it manually on every pull request. The next step was turning it into a deterministic check. AgentInspect supports deterministic checks over retained local traces. For this case, I used a JSON check configuration: { "checks": { "tool": { "required": "terrascan", "evaluate policy" }, "llm": { "maxCalls": 0 } } } Then CI checks the trace produced by the blocked fixture: npx agent-inspect check .agent-inspect/terraform-review \ --config blocked-path.check.json \ --require-completed \ --json The broken path fails because the model call count is greater than zero. The fixed path passes because: terrascan appeared. evaluate policy appeared.This is the exact behavior I wanted from CI. It does not ask another model whether the trace looks safe. It evaluates a small, reproducible structural contract and returns a deterministic exit code. A CI failure without usable evidence usually starts another debugging cycle. To retain a bounded report, I added an artifact step: - name: Run blocked-path regression run: node scripts/run-blocked-policy-case.mjs - name: Check blocked-path contract run: | npx agent-inspect check .agent-inspect/terraform-review \ --config blocked-path.check.json \ --require-completed \ --json trace-contract-result.json - name: Build safe trace artifacts if: always run: | npx agent-inspect artifacts .agent-inspect/terraform-review \ --output-dir ./agent-inspect-artifacts \ --github-summary "$GITHUB STEP SUMMARY" - name: Upload review evidence if: always uses: actions/upload-artifact@v4 with: name: terraform-agent-trace path: | agent-inspect-artifacts/ trace-contract-result.json retention-days: 14 AgentInspect creates the local report; GitHub Actions owns the upload and retention. For real pipeline data, I would still review the exact derived artifact before sharing it outside the repository. Redaction and safety scans are safeguards, not compliance certification. My first implementation treated the final verdict as proof that the gate worked. It was only proof that the workflow ended with the expected string. The stronger DevOps questions are: This is where execution traces are useful. They provide evidence about the path rather than only the answer. There is also a useful separation of responsibilities: | Layer | Responsibility | |---|---| | Terrascan | Detect infrastructure findings | | Deterministic policy code | Enforce explicit risk thresholds | | Gemini | Explain findings and suggest remediation on allowed paths | | AgentInspect | Record and check the execution path | | GitHub Actions | Enforce the build result and retain reviewed artifacts | AgentInspect did not make the Terraform deployment secure. It helped me verify whether my own control flow matched the policy I claimed to enforce. A green trace contract is narrow evidence. It does not prove that every Terraform rule is correct or that the deployed infrastructure is safe. This workflow still needs: It also does not prove that Gemini’s remediation advice is good. That requires separate evaluation. The contract proves one specific invariant: when deterministic policy rejects a Terraform change, the model path does not run. AI can add useful context to DevOps workflows, but it should not own controls that can be expressed clearly in code. My Terraform review agent already produced the expected rejection. AgentInspect showed me that the route to that answer was weaker than the answer itself suggested. Moving the risk threshold ahead of the LLM call gave the workflow a cleaner authority boundary. Adding a trace contract made that boundary reviewable in CI. The lesson is simple: Do not test only what your agent returned. Test which actions it was allowed to take before returning it.