{"slug": "i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found", "title": "I Let GitHub Copilot CLI Read a Failed AI-Agent Trace—Here’s What It Found", "summary": "A developer demonstrated that GitHub Copilot CLI can diagnose AI-agent failures from structured execution traces, using the AgentInspect MCP server to analyze a controlled TypeScript fixture. The experiment revealed a subtle bug where a formatter used the wrong optional field, causing a generic 'not enough project context' response despite successful retrieval.", "body_md": "I use GitHub Copilot CLI for practical development work: understanding unfamiliar code, reviewing changes, generating tests, and debugging from the terminal. After writing [my beginner’s guide to GitHub Copilot CLI](https://dev.to/raajaryan/github-copilot-cli-for-beginners-2026-guide-real-use-cases-setup-prompting-workflow-tips-33gd), I wanted to test a harder question:\n\nCan a coding assistant diagnose an AI-agent failure from structured execution evidence instead of a wall of logs?\n\nThis matters to me as a full-stack developer working with Next.js, TypeScript, MERN applications, and AI features across the broader TheCampusCoders ecosystem. A coding assistant can read source code, but an agent failure often depends on the path taken at runtime: which tool ran, what happened between retrieval and generation, and whether a technically successful run produced the expected behavior.\n\nFor this experiment, I connected [AgentInspect](https://github.com/rajudandigam/agent-inspect) to GitHub Copilot CLI through AgentInspect’s read-only MCP server. I used a controlled, keyless TypeScript fixture so that the same failure could be reproduced without an API key or a nondeterministic model response.\n\nDisclosure:I tested AgentInspect independently for the workflow described here. The maintainer reviewed the commands for technical accuracy; the conclusions are my own.`@agent-inspect/mcp-server`\n\nwas inPreviewwhen I ran this experiment.\n\nThe fixture represents a small project-help agent—the kind of assistant that could answer setup questions about a Next.js or MERN repository.\n\nIts path was simple:\n\nThe retriever returned this shape:\n\n```\ntype RetrievedChunk = {\n  id: string;\n  text?: string;\n  content?: string;\n};\n```\n\nBut the formatter used the wrong optional field:\n\n```\nfunction formatContext(chunks: RetrievedChunk[]) {\n  return chunks\n    .map((chunk) => chunk.content ?? \"\") // Bug: the data is in `text`\n    .join(\"\\n\\n\");\n}\n```\n\nTypeScript did not reject this because `content`\n\nwas permitted by the loose integration type. Retrieval succeeded, the formatter did not throw, and the answer step still ran. The visible symptom was only a generic “not enough project context” response.\n\nThis was exactly the kind of bug I wanted to test. There was no dramatic stack trace pointing to one bad line.\n\nI tested with Node.js 20 or newer and pinned both AgentInspect packages to the published `6.17.4`\n\nbaseline:\n\n```\nnpm install agent-inspect@6.17.4\nnpm install --save-dev @agent-inspect/mcp-server@6.17.4\n```\n\nI wrapped the workflow with `inspectRun()`\n\nand named the important boundaries with `step()`\n\n:\n\n``` js\nimport {\n  inspectRun,\n  observeOutcome,\n  step,\n} from \"agent-inspect\";\n\nawait inspectRun(\n  \"project-help-agent-broken\",\n  async () => {\n    const chunks = await step.tool(\n      \"retrieve_project_docs\",\n      retrieveProjectDocs,\n    );\n\n    const context = await step(\n      \"format_context\",\n      () => formatContext(chunks),\n      {\n        type: \"logic\",\n        metadata: {\n          returnedField: \"text\",\n          mapperField: \"content\",\n          retrievedChunkCount: chunks.length,\n        },\n      },\n    );\n\n    await step(\n      \"validate_context\",\n      async () => {\n        await observeOutcome(\"context_available\", {\n          expectation:\n            \"At least one retrieved chunk contributes usable context\",\n          status: context.trim().length > 0 ? \"passed\" : \"failed\",\n          method: \"custom\",\n          actual: {\n            retrievedChunkCount: chunks.length,\n            usableCharacterCount: context.trim().length,\n          },\n        });\n      },\n      {\n        type: \"logic\",\n        metadata: {\n          retrievedChunkCount: chunks.length,\n          usableCharacterCount: context.trim().length,\n        },\n      },\n    );\n\n    return step.llm(\n      \"deterministic-demo-model\",\n      () => generateAnswer(context),\n    );\n  },\n  {\n    traceDir: \".agent-inspect\",\n    silent: true,\n    metadata: { scenario: \"field-mapping-regression\" },\n  },\n);\n```\n\nTwo details are important here.\n\nFirst, `step.llm()`\n\nlabels a boundary; it does not call a provider by itself. My fixture used a deterministic function so the experiment focused on the debugging loop rather than model variance.\n\nSecond, I recorded counts and field names—not raw project documents, prompts, or answers. AgentInspect uses metadata-only capture by default, but metadata is still data, so I kept it bounded and non-sensitive.\n\nI ran the fixture, listed the latest trace, and inspected its report:\n\n```\nnpx agent-inspect list --dir .agent-inspect\nnpx agent-inspect report <run-id> --dir .agent-inspect\n```\n\nThe interesting result was this:\n\n```\nStatus: success\n\nSteps: 4 (1 LLM, 1 tool, 2 logic)\n\nObserved outcomes\nTotal: 1 (passed 0, failed 1, unknown 0, skipped 0)\n\ncontext_available: failed\nExpectation: At least one retrieved chunk contributes usable context\n```\n\nThe JavaScript function completed. The tool completed. The model-shaped step completed. Yet the behavioral outcome failed.\n\nI turned that observation into a deterministic check:\n\n```\nnpx agent-inspect check <run-id> \\\n  --dir .agent-inspect \\\n  --fail-on-observation failed\nCheck status: fail\nSummary: 1 failed, 0 warning(s), 0 error(s)\noutcome.status: Observed outcome count 1 matched [failed]\n```\n\nThat exit code is useful in CI, but I still needed to diagnose the cause. This is where I brought in Copilot CLI.\n\nGitHub Copilot CLI supports local stdio MCP servers. I added a project-level `.mcp.json`\n\nfile:\n\n```\n{\n  \"mcpServers\": {\n    \"agent-inspect\": {\n      \"type\": \"local\",\n      \"command\": \"npx\",\n      \"args\": [\n        \"-y\",\n        \"@agent-inspect/mcp-server@6.17.4\",\n        \"--dir\",\n        \".agent-inspect\"\n      ],\n      \"tools\": [\n        \"list_recent_runs\",\n        \"get_trace_facts\",\n        \"get_execution_tree\",\n        \"get_first_causal_failure\",\n        \"get_failed_observations\",\n        \"compare_runs\"\n      ]\n    }\n  }\n}\n```\n\nCopilot CLI also supports user-level configuration at `~/.copilot/mcp-config.json`\n\n, but I preferred project-level scope for this test. On first launch, Copilot asked me to trust the folder before loading its MCP configuration.\n\nI verified the server and its tools from Copilot CLI:\n\n```\n/mcp show agent-inspect\n```\n\nThe MCP server exposes 12 flagship read-only tools, plus compatibility tools. I enabled only the six needed for this experiment to keep the available tool set focused.\n\nI did not ask Copilot to “fix my agent.” That is too open-ended and makes it easy to mix evidence with guesses.\n\nInstead, I used this prompt:\n\n```\nUse the AgentInspect MCP tools to inspect the latest\nproject-help-agent-broken run.\n\nSeparate your response into:\n1. persisted facts,\n2. the first causal failure,\n3. your likely diagnosis,\n4. the source code I should inspect.\n\nDo not edit any files until I approve the proposed change.\n```\n\nCopilot’s useful answer was not a magical explanation. It was a short chain grounded in the trace:\n\n| What Copilot reported | Evidence available through AgentInspect |\n|---|---|\n| Retrieval completed |\n`retrieve_project_docs` was a finished tool |\n| Five chunks crossed the retrieval boundary | `retrievedChunkCount: 5` |\n| The formatter expected a different field |\n`returnedField: \"text\"` , `mapperField: \"content\"`\n|\n| The prompt boundary had no usable context | `usableCharacterCount: 0` |\n| The failed behavior was explicitly observed |\n`context_available` had status `failed`\n|\n| The answer step still ran | One finished LLM-labelled step appeared after validation |\n\nAgentInspect’s `get_first_causal_failure`\n\nresult identified the failed observed outcome and its parent step. It did not claim that adjacent events were causally linked just because their timestamps were close.\n\nCopilot then inspected the formatter source and proposed a one-line change:\n\n``` js\n- .map((chunk) => chunk.content ?? \"\")\n+ .map((chunk) => chunk.text ?? \"\")\n```\n\nThat diagnosis made sense, but I still reviewed the retriever’s runtime shape and TypeScript type before accepting it. A coding assistant’s explanation is a hypothesis; the persisted trace facts and the source contract are the evidence.\n\nAfter changing the mapping, I ran the same fixture again and checked the new trace:\n\n```\nnpx agent-inspect check <fixed-run-id> \\\n  --dir .agent-inspect \\\n  --fail-on-observation failed\nCheck status: pass\nSummary: 0 failed, 0 warning(s), 0 error(s)\n```\n\nThe important result was not simply that the answer looked better. The same deterministic expectation that failed before now passed, while the intended execution path remained retrieval → formatting → validation → generation.\n\nThis gave me a more disciplined loop:\n\nFailed behavior → local trace → bounded MCP facts → Copilot hypothesis → human review → code change → rerun → deterministic check\n\nThe boundary is worth understanding.\n\nThe AgentInspect MCP server did:\n\nIt did **not**:\n\nCopilot CLI may have its own file-editing and shell tools. Those capabilities belong to Copilot, not to AgentInspect’s read-only MCP server. I deliberately required approval before any edit.\n\nAgentInspect’s core capture and MCP process are local and do not upload data to AgentInspect by default. That does **not** mean the combined Copilot workflow is entirely offline.\n\nWhen I connect an MCP server, its returned facts become available to the connected client. Copilot may send that context to its model service according to my GitHub plan, settings, organization policies, and applicable terms.\n\nMy practical rules are:\n\nLocal trace storage is valuable, but it does not remove the need for a data-handling decision when another tool reads those traces.\n\nThe strongest part of the experiment was **search-space reduction**. Instead of asking Copilot to read hundreds of interleaved log lines, I gave it a small evidence surface with named steps, bounded metadata, and an explicit failed outcome.\n\nIt was especially useful because the program did not crash. Traditional error-first debugging would have started in the wrong place.\n\nThere were also clear limitations:\n\nI did not find an autonomous debugging system—and I do not think that should be the goal.\n\nWhat I found was a useful division of responsibility:\n\nFor TypeScript developers already using Copilot CLI, this is a more reliable pattern than pasting a giant log and asking, “What went wrong?” The coding assistant still reasons probabilistically, but it starts from better evidence.\n\nIf you want to reproduce the same loop, start with a synthetic failure, keep capture metadata-only, and make Copilot separate facts from inferences. That separation was the most valuable part of the experiment.", "url": "https://wpnews.pro/news/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found", "canonical_source": "https://dev.to/raajaryan/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found-2cl3", "published_at": "2026-09-02 08:24:09+00:00", "updated_at": "2026-09-02 08:54:01.131303+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["GitHub Copilot CLI", "AgentInspect", "TheCampusCoders", "Next.js", "TypeScript", "MERN"], "alternates": {"html": "https://wpnews.pro/news/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found", "markdown": "https://wpnews.pro/news/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found.md", "text": "https://wpnews.pro/news/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found.txt", "jsonld": "https://wpnews.pro/news/i-let-github-copilot-cli-read-a-failed-ai-agent-trace-heres-what-it-found.jsonld"}}