{"slug": "building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi", "title": "Building Autonomous AI Coding Agent Pipelines: TDD Evidence, Guardrails, and Multi-Agent Coordination", "summary": "A developer detailed an architecture for autonomous AI coding pipelines that replaces trust-based code generation with evidence-based verification, using Test-Driven Development as a control-flow mechanism. The design wraps an LLM in a Red-Green-Refactor lifecycle where a FeatureAgent generates a failing test, implements code to pass it, and feeds test-runner output back into the model for automatic correction, supplemented by LLM-based review guardrails and multi-agent coordination.", "body_md": "*Originally published on [tamiz.pro](https://tamiz.pro/insights/ai-coding-agent-pipelines-tdd-guardrails).*\n\nThe current state of AI-assisted development is largely defined by autocomplete. Tools like GitHub Copilot and Cursor have revolutionized the speed at which we write code, but they remain passive assistants. They suggest; we decide. They complete; we verify. However, the next frontier of software engineering is not just faster typing—it is autonomous agency. We are moving from \"AI that helps me write code\" to \"AI that verifies and ships features.\"\n\nBuilding a pipeline where an AI agent does not just generate tokens, but actually engineers a feature end-to-end, requires a fundamental shift in architecture. It demands moving away from trust-based integration (\"I hope this works\") to evidence-based integration (\"Here is the test that proves it works\"). This deep dive explores the architectural patterns required to build these systems: Test-Driven Development (TDD) as a verification engine, LLM-based review guardrails to prevent hallucination-induced bugs, and multi-agent coordination to handle complex system design.\n\n\"Vibe coding\"—writing software based on intuition and rapid AI generation without rigorous verification—works well for prototypes. It fails catastrophically in production. The core problem is that Large Language Models (LLMs) are probabilistic, not deterministic. They do not understand state; they predict the next likely token. An agent that simply writes a `UserService` and a `UserRepository` is operating in a vacuum. It has no feedback loop.\n\nTo build a production-grade pipeline, we must close the loop. The agent must not stop after code generation. It must stop only when the evidence is irrefutable. This evidence comes from three sources:\n\nTraditionally, Test-Driven Development is a human discipline. It forces you to write the test before the implementation, ensuring the code is actually testable. For AI agents, TDD is not a discipline; it is a control flow mechanism.\n\nIn an autonomous pipeline, the \"Test\" is the acceptance criterion. If the agent cannot write a failing test that clearly describes the requirement, it does not understand the requirement.\n\nImagine a `CodeAgent` class. It does not have a single `generate` method. It has a `refactor`, `implement`, and `verify` lifecycle.\n\n``` python\nclass FeatureAgent:\n    def __init__(self, llm: BaseLLM, file_system: LocalFS, test_runner: ShellRunner):\n        self.llm = llm\n        self.fs = file_system\n        self.test_runner = test_runner\n\n    def execute_tdd_cycle(self, requirement: str):\n        \"\"\"\n        Executes a TDD cycle: Red -> Green -> Refactor.\n        \"\"\"\n        # 1. RED: Generate a failing test\n        test_code = self._generate_test(requirement)\n        self.fs.write_file(\"tests/test_feature.py\", test_code)\n\n        # Verify it actually fails (sanity check)\n        result = self.test_runner.run(\"pytest tests/test_feature.py\")\n        if not result.has_failures():\n            raise RuntimeError(\"Test passed immediately. Requirement is trivial or test is flawed.\")\n\n        # 2. GREEN: Implement the code to pass the test\n        implementation = self._generate_implementation(requirement, test_code)\n        self.fs.write_file(\"src/feature.py\", implementation)\n\n        # 3. VERIFY\n        result = self.test_runner.run(\"pytest tests/test_feature.py\")\n        if not result.all_passed():\n            # Feed error back into LLM for correction\n            return self._auto_fix(result.output)\n\n        # 4. REFACTOR (Optional)\n        return True\n```\n\nThe critical component here is the `test_runner`. The agent is not guessing that the code works. It is executing it. This shifts the architecture from a \"Black Box\" (input prompt -> output code) to a \"Reinforcement Loop\" (input prompt -> output code -> environment feedback -> error correction).\n\nEven if the tests pass, the code might be bad. It might use deprecated APIs, ignore security best practices, or violate architectural patterns (e.g., a controller talking to the database). This is where the \"Guardrail\" agent comes in.\n\nIn this pipeline, we treat the LLM as a reviewer, not a coder. We use a distinct prompt context that focuses on critique. This is often called the \"LLM-as-a-Judge\" pattern.\n\nThe Critic agent should be stateless and fast. It reads the diff generated by the TDD agent and outputs a structured JSON verdict.\n\n```\n{\n  \"severity\": \"high\",\n  \"issue\": \"SQL Injection vulnerability\",\n  \"location\": \"src/db.py:42\",\n  \"suggestion\": \"Use parameterized queries instead of string interpolation.\"\n}\n```\n\nBy forcing the Critic to output structured data, we make the pipeline deterministic. The main orchestrator can then parse this JSON. If `severity` is \"high\", the pipeline halts and feeds the suggestion back to the TDD agent. This creates a \"Constitutional AI\" constraint, where the agent is bound by rules defined in the Critic's prompt.\n\nA single agent working in a monolithic prompt context will quickly hit the limits of its context window and become confused. Complex features (e.g., \"Add a Stripe billing system\") require separation of concerns.\n\nWe move to a **Supervisor-Worker** architecture.\n\nYou might think: \"Let the LLM decide which agent to call next.\" This is a recipe for infinite loops and unpredictable costs. The LLM should not be the conductor; it should be the player. The conductor is your code.\n\n``` js\nasync function runPipeline() {\n    // 1. Plan\n    const plan = await architectAgent.plan(featureDescription);\n\n    // 2. Execute Tasks Sequentially or in Parallel\n    for (const task of plan.tasks) {\n        if (task.dependsOn.length === 0) {\n            await workerAgent.execute(task);\n        } else {\n            // Check dependencies are met\n            if (task.dependsOn.every(id => completedTasks.has(id))) {\n                await workerAgent.execute(task);\n            }\n        }\n    }\n\n    // 3. Final Review\n    const diff = git.getDiff();\n    const review = await criticAgent.review(diff);\n\n    if (review.blockers > 0) {\n        // Trigger a fix loop\n        await workerAgent.fix(review.issues);\n    }\n}\n```\n\nLet's look at how this actually works in a real-world scenario. We are adding a \"Forgot Password\" endpoint to a Node.js/Express app.\n\n**Prompt:** \"Design a system for password reset tokens. Consider expiration (24h), security (SHA-256 hashing), and database schema changes.\"\n\n**Output (Architect Agent):**\n\n```\n{\n  \"tasks\": [\n    { \"id\": \"db\", \"desc\": \"Add 'reset_token_hash' and 'reset_expires_at' to users table\" },\n    { \"id\": \"logic\", \"desc\": \"Create generateToken service\" },\n    { \"id\": \"api\", \"desc\": \"Create /auth/forgot-password endpoint\" }\n  ]\n}\n```\n\n**Prompt:** \"Write a test that ensures the database migration adds the new columns. Then write the migration file.\"\n\nThe worker writes `tests/migration.test.ts`:\n\n``` js\ntest('db schema update', async () => {\n  const schema = await getSchema();\n  expect(schema.tables.users.columns).toContain('reset_token_hash');\n});\n```\n\nIt runs the test. It fails (Red). It writes the migration. It runs the test. It passes (Green).\n\n**Prompt:** \"Implement the token generation logic based on the test in `tests/token.test.ts`.\"\n\nThe worker generates the logic, ensuring the token is hashed with SHA-256 before storage.\n\n**Prompt:** \"Review the diff for the last two commits. Check for: 1. Is the token stored in plain text? 2. Is the expiration time respected in the query? 3. Are SQL injections possible?\"\n\nThe Critic analyzes the code. It finds that the `SELECT` query in the API endpoint uses string interpolation for the expiration check. It flags this as a `high` severity issue. \n\nThe orchestrator sees the `high` severity flag. It takes the Critic's suggestion: \"Use parameterized queries.\" It passes this back to the API Worker. The Worker updates the code to use `?` placeholders. The Critic re-reviews. It passes. The pipeline commits the changes. \n\nBuilding these pipelines is not plug-and-play. There are specific failure modes that will break your automation.\n\nAs the agent makes more changes, the context window fills with previous code. The agent begins to \"forget\" the requirements from the beginning of the prompt.\n\nThe agent writes `import { RedisClient } from 'redis'` but the project uses `ioredis`. \n\n`npm install` or `pip install` check, or better yet, the Critic must have access to the `package.json` / `requirements.txt` and verify imports against the actual dependencies.\nThe agent fails the test, tries to fix it, fails again, and gets stuck in a cycle.\n\n`HumanInterventionRequired` exception. Giving an AI agent write access to your codebase and execution privileges is terrifying.\n\n`main`. The final commit should be a PR generated by the pipeline, reviewed by a human, and merged manually (or via strict CI checks). We are currently in the early days of this. The \"Agentic SWE\" (Software Engineering) pipeline is the bridge to fully autonomous development. However, the bottleneck is not the LLM's intelligence; it is the LLM's reliability.\n\nBy wrapping the LLM in a TDD loop, we are essentially using the compiler and the test runner as our guardrails. The LLM is the \"brain,\" but the test runner is the \"safety net.\"\n\nFor now, do not try to build a general-purpose agent that can do everything. Build specialized agents that do one thing well, and coordinate them with a deterministic orchestrator. This is the path to software that is not just written by AI, but *verified* by it.\n\nFor the TDD Worker, a 16k-32k context window is usually sufficient to hold the requirement, the current file, and the error output. For the Critic, you need enough space to hold the entire diff plus the project's style guide. 32k-128k is recommended for complex reviews.\n\nYes. Using a different model (or even the same model with a temperature of 0.0 and a highly specific \"judge\" persona) prevents the agent from \"rubber-stamping\" its own work. A second opinion is always more valuable than a confirmation bias.\n\nThe orchestrator must decompose the task. The TDD agent should not be asked to rewrite the whole backend. It should be asked to \"Update the `User` model to support soft deletes\" and then \"Update the `UserService` to respect soft deletes\". This keeps the context small and the diffs reviewable. \n\n*For more insights on AI engineering and developer tooling, check out [Tamiz's Insights](https://tamiz.pro/insights).*\n\nWhile a single agent can handle focused tasks, real-world codebases demand specialization. The most effective autonomous pipelines employ **multi-agent architectures** where distinct agents own different concerns.\n\n```\n┌─────────────────────────────────────────────────────────┐\n│                    Orchestrator Agent                     │\n│  (Task decomposition, state management, conflict res.)   │\n└──────────────┬──────────────────────────────────────────┘\n               │\n    ┌──────────┼──────────────┬──────────────────┐\n    ▼          ▼              ▼                  ▼\n┌────────┐ ┌────────┐  ┌───────────┐    ┌────────────┐\n│Coder   │ │Tester  │  │Reviewer   │    │Doc Writer  │\n│Agent   │ │Agent   │  │Agent      │    │Agent       │\n└────────┘ └────────┘  └───────────┘    └────────────┘\n```\n\nEach agent receives a **narrow system prompt** with domain-specific context, not the full codebase. The orchestrator manages the DAG of dependencies:\n\n``` python\n# orchestrator.py\nfrom dataclasses import dataclass, field\nfrom typing import Literal\nfrom enum import Enum\n\nclass AgentRole(Enum):\n    Coder = \"coder\"\n    Tester = \"tester\"\n    Reviewer = \"reviewer\"\n    DocWriter = \"doc_writer\"\n\n@dataclass\nclass TaskNode:\n    id: str\n    role: AgentRole\n    description: \"str\"\n    dependencies: list[str] = field(default_factory=list)\n    status: Literal[\"pending\", \"in_progress\", \"completed\", \"failed\"] = \"pending\"\n    output: str | None = None\n\nclass Orchestrator:\n    def __init__(self, agents: dict[AgentRole, \"BaseAgent\"]):\n        self.agents = agents\n        self.task_graph: dict[str, TaskNode] = {}\n        self.execution_order: list[str] = []\n\n    def add_task(self, task: TaskNode) -> None:\n        self.task_graph[task.id] = task\n\n    def topological_sort(self) -> list[str]:\n        \"\"\"Resolve execution order respecting dependencies.\"\"\"\n        visited = set()\n        temp_visited = set()\n        order = []\n\n        def visit(node_id: str):\n            if node_id in temp_visited:\n                raise ValueError(f\"Circular dependency detected at {node_id}\")\n            if node_id in visited:\n                return\n            temp_visited.add(node_id)\n            node = self.task_graph[node_id]\n            for dep in node.dependencies:\n                visit(dep)\n            temp_visited.remove(node_id)\n            visited.add(node_id)\n            order.append(node_id)\n\n        for node_id in self.task_graph:\n            visit(node_id)\n        return order\n\n    async def execute(self) -> dict[str, str]:\n        \"\"\"Execute tasks in dependency order.\"\"\"\n        results = {}\n        for task_id in self.topological_sort():\n            node = self.task_graph[task_id]\n            node.status = \"in_progress\"\n\n            # Gather context from dependencies\n            dep_context = {\n                dep_id: self.task_graph[dep_id].output\n                for dep_id in node.dependencies\n                if self.task_graph[dep_id].output\n            }\n\n            agent = self.agents[node.role]\n            try:\n                output = await agent.execute(\n                    task_description=node.description,\n                    context=dep_context\n                )\n                node.output = output\n                node.status = \"completed\"\n                results[task_id] = output\n            except Exception as e:\n                node.status = \"failed\"\n                results[task_id] = f\"FAILED: {e}\"\n                # Propagate failure to dependent tasks\n                self._mark_dependents_failed(task_id)\n\n        return results\n\n    def _mark_dependents_failed(self, failed_id: str) -> None:\n        for node in self.task_graph.values():\n            if failed_id in node.dependencies and node.status == \"pending\":\n                node.status = \"failed\"\n                node.output = f\"SKIPPED: dependency {failed_id} failed\"\n```\n\nAgents don't share memory directly—they exchange **structured artifacts**. This prevents context pollution and makes the pipeline auditable:\n\n``` python\n# agent_protocol.py\nfrom pydantic import BaseModel, Field\nfrom typing import Any\n\nclass Artifact(BaseModel):\n    \"\"\"Standardized output format between agents.\"\"\"\n    task_id: str\n    author_role: str\n    summary: str = Field(..., description=\"One-sentence summary of what was done\")\n    files_modified: list[str] = Field(default_factory=list)\n    tests_added: list[str] = Field(default_factory=list)\n    evidence: dict[str, Any] = Field(default_factory=dict)\n    confidence: float = Field(..., ge=0.0, le=1.0)\n    follow_up_needed: bool = False\n    follow_up_reason: str | None = None\n\nclass ReviewArtifact(Artifact):\n    \"\"\"Specialized artifact from the reviewer agent.\"\"\"\n    verdict: Literal[\"approved\", \"changes_requested\", \"blocked\"]\n    critical_issues: list[str] = Field(default_factory=list)\n    suggestions: list[str] = Field(default_factory=list)\n    security_concerns: list[str] = Field(default_factory=list)\n```\n\nThe **Tester agent** receives the Coder's artifact and produces its own:\n\n``` python\n# tester_agent.py\nclass TesterAgent(BaseAgent):\n    def __init__(self, model: str = \"gpt-4\", test_runner: TestRunner | None = None):\n        super().__init__(model=model)\n        self.test_runner = test_runner or LocalTestRunner()\n\n    async def execute(self, task_description: \"str, context: dict[str, str]) -> Artifact:\"\n        # Extract the code changes from the coder's output\n        code_context = context.get(\"coder_output\", \"\")\n\n        # Generate tests if none exist\n        test_plan = await self._generate_test_plan(task_description, code_context)\n\n        # Write test files\n        test_files = self._write_tests(test_plan)\n\n        # Execute the test suite\n        results = await self.test_runner.run(\n            test_files=test_files,\n            coverage_threshold=80.0\n        )\n\n        return Artifact(\n            task_id=self.current_task_id,\n            author_role=\"tester\",\n            summary=f\"Generated {len(test_files)} test files, {results.passed} passed, {results.failed} failed\",\n            files_modified=[f.path for f in test_files],\n            tests_added=[t.name for t in test_files],\n            evidence={\n                \"coverage_percent\": results.coverage,\n                \"pass_rate\": results.pass_rate,\n                \"failed_tests\": [f\"{t.name}: {t.error}\" for t in results.failures],\n            },\n            confidence=min(results.pass_rate / 100.0, 1.0),\n            follow_up_needed=results.failed > 0,\n            follow_up_reason=f\"{results.failed} tests failing\" if results.failed > 0 else None,\n        )\n\n    async def _generate_test_plan(self, description: \"str, code: str) -> TestPlan:\"\n        prompt = f\"\"\"\nGiven this task description: \"\"\n{description}\n\nAnd this implementation:\n{code}\n\nGenerate a comprehensive test plan covering:\n1. Happy path scenarios\n2. Edge cases (null, empty, boundary values)\n3. Error handling paths\n4. Concurrency/race conditions if applicable\n\nReturn as a structured test plan.\n\"\"\"\n        response = await self._call_model(prompt)\n        return TestPlan.from_response(response)\n```\n\nWhen multiple agents modify the same file (e.g., Coder adds a method, DocWriter updates the docstring), a **merge coordinator** resolves conflicts:\n\n``` python\n# merge_coordinator.py\nfrom dataclasses import dataclass\nfrom pathlib import Path\n\n@dataclass\nclass EditRegion:\n    file_path: str\n    start_line: int\n    end_line: int\n    agent_role: str\n    content: str\n\nclass MergeCoordinator:\n    \"\"\"Detect and resolve overlapping edits from multiple agents.\"\"\"\n\n    def detect_conflicts(self, edits: list[EditRegion]) -> list[tuple[EditRegion, EditRegion]]:\n        conflicts = []\n        for i, edit_a in enumerate(edits):\n            for j, edit_b in enumerate(edits):\n                if i >= j:\n                    continue\n                if self._overlaps(edit_a, edit_b):\n                    conflicts.append((edit_a, edit_b))\n        return conflicts\n\n    def _overlaps(self, a: EditRegion, b: EditRegion) -> bool:\n        if a.file_path != b.file_path:\n            return False\n        return a.start_line <= b.end_line and b.start_line <= a.end_line\n\n    def resolve(self, file_path: str, edits: list[EditRegion]) -> str:\n        \"\"\"\n        Strategy: merge non-overlapping regions in order.\n        For overlapping regions, use the agent with higher priority.\n        \"\"\"\n        priority = {\"coder\": 3, \"tester\": 2, \"doc_writer\": 1, \"reviewer\": 0}\n        sorted_edits = sorted(\n            [e for e in edits if e.file_path == file_path],\n            key=lambda e: (e.start_line, -priority.get(e.agent_role, 0))\n        )\n\n        original = Path(file_path).read_text().splitlines()\n        result = list(original)\n\n        # Apply edits from bottom to top to preserve line numbers\n        for edit in reversed(sorted_edits):\n            result[edit.start_line - 1 : edit.end_line] = edit.content.splitlines()\n\n        return \"\\n\".join(result)\n```\n\nAutonomous agents are powerful but dangerous. Without guardrails, an agent might delete production code, introduce security vulnerabilities, or enter infinite loops. Guardrails operate at multiple levels:\n\nBefore any agent touches the codebase, validate the intent:\n\n``` python\n# guardrails/pre_execution.py\nfrom pydantic import BaseModel, field_validator\nfrom typing import Literal\n\nclass TaskRequest(BaseModel):\n    description: str\n    files_affected: list[str]\n    estimated_complexity: Literal[\"trivial\", \"simple\", \"moderate\", \"complex\", \"architectural\"]\n\n    @field_validator(\"files_affected\")\n    @classmethod\n    def validate_file_paths(cls, files: list[str]) -> list[str]:\n        \"\"\"Reject paths that touch sensitive areas.\"\"\"\n        forbidden_patterns = [\n            \"**/secrets/**\",\n            \"**/.env*\",\n            \"**/credentials/**\",\n            \"**/migrations/production/**\",\n            \"**/*.key\",\n            \"**/*.pem\",\n        ]\n        from pathlib import Path\n        for f in files:\n            for pattern in forbidden_patterns:\n                if Path(f).match(pattern):\n                    raise ValueError(f\"Agent cannot modify sensitive file: {f}\")\n        return files\n\n    @field_validator(\"estimated_complexity\")\n    @classmethod\n    def validate_complexity(cls, complexity: str) -> str:\n        \"\"\"Architectural changes require human approval.\"\"\"\n        if complexity == \"architectural\":\n            raise ValueError(\n                \"Architectural changes require human review before execution. \"\n                \"Break this into smaller tasks or request approval.\"\n            )\n        return complexity\n```\n\nMonitor agent behavior during execution and halt on anomalies:\n\n``` python\n# guardrails/circuit_breaker.py\nimport time\nfrom collections import deque\nfrom dataclasses import dataclass\n\n@dataclass\nclass CircuitState:\n    failures: int = 0\n    last_failure_time: float = 0\n    is_open: bool = False\n    opened_at: float = 0\n\nclass CircuitBreaker:\n    \"\"\"Halt agent execution when failure patterns are detected.\"\"\"\n\n    def __init__(\n        self,\n        failure_threshold: int = 3,\n        recovery_timeout: float = 60.0,\n        max_context_window: int = 50000,\n        max_tool_calls_per_task: int = 20,\n    ):\n        self.failure_threshold = failure_threshold\n        self.recovery_timeout = recovery_timeout\n        self.max_context_window = max_context_window\n        self.max_tool_calls_per_task = max_tool_calls_per_task\n        self.states: dict[str, CircuitState] = {}\n        self.call_counts: dict[str, int] = {}\n        self.history: deque[dict] = deque(maxlen=100)\n\n    def record_tool_call(self, agent_id: str, tool_name: str, context_size: int) -> None:\n        self.call_counts[agent_id] = self.call_counts.get(agent_id, 0) + 1\n        self.history.append({\n            \"agent_id\": agent_id,\n            \"tool\": tool_name,\n            \"context_size\": context_size,\n            \"timestamp\": time.time(),\n        })\n\n    def check(self, agent_id: str) -> tuple[bool, str | None]:\n        \"\"\"Returns (can_proceed, reason_if_blocked).\"\"\"\n        # Check tool call limit\n        calls = self.call_counts.get(agent_id, 0)\n        if calls >= self.max_tool_calls_per_task:\n            return False, f\"Agent {agent_id} exceeded {self.max_tool_calls_per_task} tool calls\"\n\n        # Check context window\n        if self.history:\n            recent = list(self.history)[-10:]\n            avg_context = sum(e[\"context_size\"] for e in recent) / len(recent)\n            if avg_context > self.max_context_window:\n                return False, f\"Context window growing unbounded (avg: {avg_context})\"\n\n        # Check circuit breaker state\n        state = self.states.get(agent_id)\n        if state and state.is_open:\n            if time.time() - state.opened_at > self.recovery_timeout:\n                state.is_open = False\n                state.failures = 0\n                return True, None\n            return False, f\"Circuit breaker open for {agent_id}, retry after {self.recovery_timeout}s\"\n\n        return True, None\n\n    def record_failure(self, agent_id: str) -> None:\n        state = self.states.setdefault(agent_id, CircuitState())\n        state.failures += 1\n        state.last_failure_time = time.time()\n        if state.failures >= self.failure_threshold:\n            state.is_open = True\n            state.opened_at = time.time()\n\n    def record_success(self, agent_id: str) -> None:\n        state = self.states.get(agent_id)\n        if state:\n            state.failures = max(0, state.failures - 1)\n```\n\nAfter an agent completes a task, independently verify the result:\n\n``` python\n# guardrails/post_execution.py\nimport subprocess\nfrom pathlib import Path\nfrom dataclasses import dataclass\n\n@dataclass\nclass VerificationResult:\n    passed: bool\n    checks_run: list[str]\n    failures: list[str]\n    metrics: dict[str, float]\n\nclass PostExecutionVerifier:\n    \"\"\"Independent verification of agent output.\"\"\"\n\n    def __init__(self, project_root: Path):\n        self.project_root = project_root\n\n    def verify(self, changed_files: list[str], tests_to_run: list[str] | None = None) -> VerificationResult:\n        checks = []\n        failures = []\n        metrics = {}\n\n        # 1. Type checking\n        type_result = self._run_type_check()\n        checks.append(\"type_check\")\n        if not type_result.passed:\n            failures.append(f\"Type errors: {type_result.output[:500]}\")\n\n        # 2. Linting\n        lint_result = self._run_linter(changed_files)\n        checks.append(\"lint\")\n        if not lint_result.passed:\n            failures.append(f\"Lint errors: {lint_result.output[:500]}\")\n\n        # 3. Unit tests\n        test_result = self._run_tests(tests_to_run)\n        checks.append(\"unit_tests\")\n        metrics[\"tests_passed\"] = test_result.passed\n        metrics[\"tests_failed\"] = test_result.failed\n        if test_result.failed > 0:\n            failures.append(f\"{test_result.failed} tests failing\")\n\n        # 4. Security scan\n        security_result = self._run_security_scan(changed_files)\n        checks.append(\"security_scan\")\n        metrics[\"security_issues\"] = security_result.critical_count\n        if security_result.critical_count > 0:\n            failures.append(f\"{security_result.critical_count} critical security issues\")\n\n        # 5. Diff size check (prevent massive refactors)\n        diff_size = self._get_diff_size(changed_files)\n        checks.append(\"diff_size\")\n        metrics[\"lines_changed\"] = diff_size\n        if diff_size > 500:\n            failures.append(f\"Diff too large ({diff_size} lines). Break into smaller tasks.\")\n\n        return VerificationResult(\n            passed=len(failures) == 0,\n            checks_run=checks,\n            failures=failures,\n            metrics=metrics,\n        )\n\n    def _run_type_check(self) -> subprocess.CompletedProcess:\n        return subprocess.run(\n            [\"mypy\", \"--strict\", \".\"],\n            capture_output=True, text=True, cwd=self.project_root, timeout=120\n        )\n\n    def _run_linter(self, files: list[str]) -> subprocess.CompletedProcess:\n        return subprocess.run(\n            [\"ruff\", \"check\", *files],\n            capture_output=True, text=True, cwd=self.project_root, timeout=60\n        )\n\n    def _run_tests(self, tests: list[str] | None) -> TestResult:\n        cmd = [\"pytest\", \"-q\", \"--tb=short\"]\n        if tests:\n            cmd.extend(tests)\n        result = subprocess.run(cmd, capture_output=True, text=True, cwd=self.project_root, timeout=300)\n        return self._parse_test_output(result.stdout)\n\n    def _run_security_scan(self, files: list[str]) -> SecurityResult:\n        return subprocess.run(\n            [\"bandit\", \"-r\", *files, \"-f\", \"json\"],\n            capture_output=True, text=True, cwd=self.project_root, timeout=120\n        )\n\n    def _get_diff_size(self, files: list[str]) -> int:\n        result = subprocess.run(\n            [\"git\", \"diff\", \"--stat\", *files],\n            capture_output=True, text=True, cwd=self.project_root\n        )\n        # Parse \"X insertions(+), Y deletions(-)\"\n        import re\n        match = re.search(r\"(\\d+) insertions?\\(\\+\\), (\\d+) deletions?\\(-\\)\", result.stdout)\n        if match:\n            return int(match.group(1)) + int(match.group(2))\n        return 0\n```\n\nNot every change should proceed autonomously. Define **escalation policies**:\n\n``` python\n# guardrails/escalation.py\nfrom enum import Enum\n\nclass RiskLevel(Enum):\n    LOW = \"low\"          # Auto-approve\n    MEDIUM = \"medium\"    # Auto-approve with notification\n    HIGH = \"high\"        # Require human approval\n    CRITICAL = \"critical\" # Block entirely\n\nclass EscalationPolicy:\n    \"\"\"Determine when to escalate to human review.\"\"\"\n\n    def assess(self, verification: VerificationResult, task_metadata: dict) -> RiskLevel:\n        # Critical: security issues or test failures\n        if verification.metrics.get(\"security_issues\", 0) > 0:\n            return RiskLevel.CRITICAL\n        if verification.metrics.get(\"tests_failed\", 0) > 0:\n            return RiskLevel.HIGH\n\n        # High: large diffs or changes to critical files\n        if verification.metrics.get(\"lines_changed\", 0) > 200:\n            return RiskLevel.HIGH\n        if any(f.startswith(\"auth/\") or f.startswith(\"payments/\") for f in task_metadata.get(\"files\", [])):\n            return RiskLevel.HIGH\n\n        # Medium: moderate changes\n        if verification.metrics.get(\"lines_changed\", 0) > 50:\n            return RiskLevel.MEDIUM\n\n        return RiskLevel.LOW\n```\n\nThe strongest guarantee an AI agent can provide is **evidence**—concrete, reproducible proof that the code works. This section covers building a TDD-first pipeline where tests are generated, executed, and archived as evidence.\n\n```\n┌──────────────┐     ┌──────────────┐     ┌──────────────┐\n│  1. SPEC     │────▶│  2. TESTS    │────▶│  3. RED      │\n│  (Intent)    │     │  (Generate)  │     │  (Verify     │\n│              │     │              │     │   failure)   │\n└──────────────┘     └──────────────┘     └──────┬───────┘\n                                                  │\n┌──────────────┐     ┌──────────────┐     ┌──────▼───────┐\n│  6. EVIDENCE │◀────│  5. GREEN    │◀────│  4. CODE     │\n│  (Archive)   │     │  (All pass)  │     │  (Implement) │\n└──────────────┘     └──────────────┘     └──────────────┘\npython\n# tdd_pipeline.py\nfrom dataclasses import dataclass\nfrom pathlib import Path\nimport json\nimport hashlib\nfrom datetime import datetime\n\n@dataclass\nclass TDDEvidence:\n    \"\"\"Immutable record of TDD execution.\"\"\"\n    task_id: str\n    timestamp: str\n    spec_hash: str\n    test_files: list[str]\n    implementation_files: list[str]\n    red_phase_output: str       # Tests failing before implementation\n    green_phase_output: str     # Tests passing after implementation\n    coverage: float\n    test_count: int\n    pass_count: int\n    verification_hash: str      # SHA256 of all evidence combined\n\nclass TDDPipeline:\n    def __init__(self, project_root: Path, evidence_dir: Path):\n        self.project_root = project_root\n        self.evidence_dir = evidence_dir\n        self.evidence_dir.mkdir(parents=True, exist_ok=True)\n\n    async def execute(self, spec: str, target_files: list[str]) -> TDDEvidence:\n        # Phase 1: Generate tests from spec\n        test_files = await self._generate_tests(spec)\n\n        # Phase 2: Verify tests fail (RED)\n        red_output = self._run_tests(test_files)\n        assert not red_output.all_passed, \"Tests should fail before implementation (RED phase)\"\n\n        # Phase 3: Generate implementation\n        impl_files = await self._generate_implementation(spec, test_files)\n\n        # Phase 4: Verify tests pass (GREEN)\n        green_output = self._run_tests(test_files)\n        assert green_output.all_passed, \"Tests should pass after implementation (GREEN phase)\"\n\n        # Phase 5: Archive evidence\n        evidence = TDDEvidence(\n            task_id=hashlib.sha256(spec.encode()).hexdigest()[:12],\n            timestamp=datetime.utcnow().isoformat(),\n            spec_hash=hashlib.sha256(spec.encode()).hexdigest(),\n            test_files=[str(f.relative_to(self.project_root)) for f in test_files],\n            implementation_files=[str(f.relative_to(self.project_root)) for f in impl_files],\n            red_phase_output=red_output.raw_output,\n            green_phase_output=green_output.raw_output,\n            coverage=green_output.coverage,\n            test_count=green_output.total_tests,\n            pass_count=green_output.passed_tests,\n            verification_hash=\"\",  # Computed below\n        )\n\n        # Compute verification hash\n        evidence_payload = json.dumps({\n            \"spec_hash\": evidence.spec_hash,\n            \"test_files\": evidence.test_files,\n            \"red_output\": evidence.red_phase_output,\n            \"green_output\": evidence.green_phase_output,\n            \"coverage\": evidence.coverage,\n        }, sort_keys=True)\n        evidence.verification_hash = hashlib.sha256(evidence_payload.encode()).hexdigest()\n\n        # Persist evidence\n        self._archive_evidence(evidence)\n\n        return evidence\n\n    async def _generate_tests(self, spec: str) -> list[Path]:\n        \"\"\"Generate test files from specification.\"\"\"\n        # Implementation would call LLM with test-generation prompt\n        # Returns list of created test file paths\n        ...\n\n    async def _generate_implementation(self, spec: str, test_files: list[Path]) -> list[Path]:\n        \"\"\"Generate implementation that satisfies tests.\"\"\"\n        # Implementation would call LLM with the spec and test context\n        # Returns list of created/modified implementation file paths\n        ...\n\n    def _run_tests(self, test_files: list[Path]) -> TestRunResult:\n        \"\"\"Execute tests and capture full output.\"\"\"\n        import subprocess\n        test_paths = [str(f.relative_to(self.project_root)) for f in test_files]\n        result = subprocess.run(\n            [\"pytest\", *test_paths, \"--tb=long\", \"--cov\", \"--cov-report=term-missing\"],\n            capture_output=True, text=True, cwd=self.project_root, timeout=300\n        )\n        return self._parse_result(result)\n\n    def _archive_evidence(self, evidence: TDDEvidence) -> None:\n        \"\"\"Persist evidence as a JSON file for audit trail.\"\"\"\n        evidence_path = self.evidence_dir / f\"{evidence.task_id}.json\"\n        evidence_path.write_text(json.dumps(\n            {k: v for k, v in evidence.__dict__.items()},\n            indent=2, default=str\n        ))\n```\n\nAnyone can independently verify that the agent's work was correct:\n\n``` python\n# evidence_verifier.py\nimport json\nimport hashlib\nfrom pathlib import Path\n\nclass EvidenceVerifier:\n    \"\"\"Independently verify TDD evidence integrity.\"\"\"\n\n    def verify(self, evidence_path: Path) -> VerificationReport:\n        evidence = TDDEvidence.from_json(evidence_path.read_text())\n\n        # 1. Verify hash integrity\n        payload = json.dumps({\n            \"spec_hash\": evidence.spec_hash,\n            \"test_files\": evidence.test_files,\n            \"red_output\": evidence.red_phase_output,\n            \"green_output\": evidence.green_phase_output,\n            \"coverage\": evidence.coverage,\n        }, sort_keys=True)\n        computed_hash = hashlib.sha256(payload.encode()).hexdigest()\n\n        if computed_hash != evidence.verification_hash:\n            return VerificationReport(\n                valid=False,\n                reason=\"Evidence hash mismatch - possible tampering\"\n            )\n\n        # 2. Verify test files exist\n        for test_file in evidence.test_files:\n            if not Path(test_file).exists():\n                return VerificationReport(\n                    valid=False,\n                    reason=f\"Test file missing: {test_file}\"\n                )\n\n        # 3. Verify test count matches\n        test_content = \"\\n\".join(Path(f).read_text() for f in evidence.test_files)\n        actual_test_count = test_content.count(\"def test_\")\n        if actual_test_count != evidence.test_count:\n            return VerificationReport(\n                valid=False,\n                reason=f\"Test count mismatch: expected {evidence.test_count}, found {actual_test_count}\"\n            )\n\n        return VerificationReport(\n            valid=True,\n            reason=\"All evidence checks passed\",\n            metrics={\n                \"coverage\": evidence.coverage,\n                \"tests\": evidence.test_count,\n                \"passed\": evidence.pass_count,\n            }\n        )\n```\n\nCombine multiple evidence signals into a single confidence score:\n\n``` python\n# confidence_scorer.py\nfrom dataclasses import dataclass\n\n@dataclass\nclass ConfidenceScore:\n    overall: float          # 0.0 to 1.0\n    components: dict[str, float]\n    deductions: list[str]\n\nclass ConfidenceScorer:\n    \"\"\"Calculate evidence-backed confidence in agent output.\"\"\"\n\n    def score(self, evidence: TDDEvidence, verification: VerificationResult) -> ConfidenceScore:\n        components = {}\n        deductions = []\n\n        # Test pass rate (weight: 0.35)\n        if evidence.test_count > 0:\n            pass_rate = evidence.pass_count / evidence.test_count\n            components[\"test_pass_rate\"] = pass_rate\n        else:\n            components[\"test_pass_rate\"] = 0.0\n            deductions.append(\"No tests executed\")\n\n        # Coverage (weight: 0.20)\n        coverage_score = min(evidence.coverage / 80.0, 1.0)  # 80% = perfect score\n        components[\"coverage\"] = coverage_score\n        if evidence.coverage < 60:\n            deductions.append(f\"Low coverage: {evidence.coverage:.1f}%\")\n\n        # Verification checks (weight: 0.25)\n        checks_passed = len(verification.checks_run) - len(verification.failures)\n        checks_total = len(verification.checks_run)\n        verification_score = checks_passed / checks_total if checks_total > 0 else 0.0\n        components[\"verification\"] = verification_score\n\n        # Diff size appropriateness (weight: 0.10)\n        lines = verification.metrics.get(\"lines_changed\", 0)\n        if lines <= 50:\n            diff_score = 1.0\n        elif lines <= 200:\n            diff_score = 0.7\n        elif lines <= 500:\n            diff_score = 0.4\n        else:\n            diff_score = 0.1\n            deductions.append(f\"Large diff: {lines} lines\")\n        components[\"diff_size\"] = diff_score\n\n        # TDD compliance (weight: 0.10)\n        # Did the RED phase actually fail?\n        tdd_score = 1.0 if \"FAILED\" in evidence.red_phase_output else 0.0\n        if tdd_score == 0.0:\n            deductions.append(\"RED phase did not show failures - TDD compliance questionable\")\n        components[\"tdd_compliance\"] = tdd_score\n\n        # Weighted sum\n        weights = {\n            \"test_pass_rate\": 0.35,\n            \"coverage\": 0.20,\n            \"verification\": 0.25,\n            \"diff_size\": 0.10,\n            \"tdd_compliance\": 0.10,\n        }\n        overall = sum(components[k] * weights[k] for k in weights)\n\n        return ConfidenceScore(\n            overall=round(overall, 3),\n            components=components,\n            deductions=deductions,\n        )\n```\n\nHere's how all the components integrate into a production-ready autonomous coding pipeline:\n\n``` python\n# pipeline.py\nfrom pathlib import Path\nfrom dataclasses import dataclass\nfrom enum import Enum\nimport asyncio\n\nclass PipelinePhase(Enum):\n    SPEC_VALIDATION = \"spec_validation\"\n    TASK_DECOMPOSITION = \"task_decomposition\"\n    TDD_GENERATION = \"tdd_generation\"\n    IMPLEMENTATION = \"implementation\"\n    VERIFICATION = \"verification\"\n    REVIEW = \"review\"\n    EVIDENCE_ARCHIVE = \"evidence_archive\"\n    ESCALATION_CHECK = \"escalation_check\"\n\n@dataclass\nclass PipelineResult:\n    success: bool\n    phase_reached: PipelinePhase\n    evidence: TDDEvidence | None\n    confidence: ConfidenceScore | None\n    risk_level: RiskLevel\n    requires_human_review: bool\n    summary: str\n\nclass AutonomousCodingPipeline:\n    def __init__(\n        self,\n        project_root: Path,\n        model_provider: ModelProvider,\n        human_approver: HumanApprover | None = None,\n    ):\n        self.project_root = project_root\n        self.model_provider = model_provider\n        self.human_approver = human_approver\n\n        # Initialize components\n        self.orchestrator = Orchestrator(agents={\n            AgentRole.Coder: CoderAgent(model_provider),\n            AgentRole.Tester: TesterAgent(model_provider),\n            AgentRole.Reviewer: ReviewerAgent(model_provider),\n            AgentRole.DocWriter: DocWriterAgent(model_provider),\n        })\n        self.tdd_pipeline = TDDPipeline(project_root, project_root / \".evidence\")\n        self.circuit_breaker = CircuitBreaker()\n        self.verifier = PostExecutionVerifier(project_root)\n        self.escalation = EscalationPolicy()\n        self.confidence_scorer = ConfidenceScorer()\n        self.merge_coordinator = MergeCoordinator()\n\n    async def run(self, spec: str) -> PipelineResult:\n        \"\"\"Execute the full autonomous coding pipeline.\"\"\"\n        phases_completed = []\n\n        try:\n            # Phase 1: Validate the specification\n            task_request = self._validate_spec(spec)\n            phases_completed.append(PipelinePhase.SPEC_VALIDATION)\n\n            # Phase 2: Decompose into subtasks\n            tasks = await self._decompose(spec, task_request)\n            phases_completed.append(PipelinePhase.TASK_DECOMPOSITION)\n\n            # Phase 3-4: Execute TDD loop for each task\n            all_evidence = []\n            for task in tasks:\n                can_proceed, reason = self.circuit_breaker.check(task.id)\n                if not can_proceed:\n                    return PipelineResult(\n                        success=False,\n                        phase_reached=PipelinePhase.IMPLEMENTATION,\n                        evidence=None,\n                        confidence=None,\n                        risk_level=RiskLevel.CRITICAL,\n                        requires_human_review=True,\n                        summary=f\"Circuit breaker triggered: {reason}\",\n                    )\n\n                evidence = await self.tdd_pipeline.execute(task.description, task.files)\n                all_evidence.append(evidence)\n                phases_completed.append(PipelinePhase.TDD_GENERATION)\n\n            # Phase 5: Post-execution verification\n            changed_files = [f for e in all_evidence for f in e.implementation_files]\n            verification = self.verifier.verify(changed_files)\n            phases_completed.append(PipelinePhase.VERIFICATION)\n\n            # Phase 6: Code review\n            review = await self._run_review(all_evidence, verification)\n            phases_completed.append(PipelinePhase.REVIEW)\n\n            # Phase 7: Archive evidence\n            combined_evidence = self._combine_evidence(all_evidence)\n            phases_completed.append(PipelinePhase.EVIDENCE_ARCHIVE)\n\n            # Phase 8: Escalation check\n            risk_level = self.escalation.assess(verification, {\"files\": changed_files})\n            confidence = self.confidence_scorer.score(combined_evidence, verification)\n            phases_completed.append(PipelinePhase.ESCALATION_CHECK)\n\n            requires_human = risk_level in (RiskLevel.HIGH, RiskLevel.CRITICAL)\n            if requires_human and self.human_approver:\n                approved = await self.human_approver.request_approval(\n                    summary=confidence,\n                    evidence=combined_evidence,\n                    risk_level=risk_level,\n                )\n                if not approved:\n```\n\n", "url": "https://wpnews.pro/news/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi", "canonical_source": "https://dev.to/tamizuddin/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi-agent-1c22", "published_at": "2026-10-12 12:02:54+00:00", "updated_at": "2026-10-12 12:15:05.010351+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "mlops"], "entities": ["GitHub Copilot", "Cursor", "tamiz.pro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi", "markdown": "https://wpnews.pro/news/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi.md", "text": "https://wpnews.pro/news/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi.txt", "jsonld": "https://wpnews.pro/news/building-autonomous-ai-coding-agent-pipelines-tdd-evidence-guardrails-and-multi.jsonld"}}