I Let GitHub Copilot CLI Read a Failed AI-Agent Trace—Here’s What It Found

I use GitHub Copilot CLI for practical development work: understanding unfamiliar code, reviewing changes, generating tests, and debugging from the terminal. After writing my beginner’s guide to GitHub Copilot CLI, I wanted to test a harder question:

Can a coding assistant diagnose an AI-agent failure from structured execution evidence instead of a wall of logs?

This matters to me as a full-stack developer working with Next.js, TypeScript, MERN applications, and AI features across the broader TheCampusCoders ecosystem. A coding assistant can read source code, but an agent failure often depends on the path taken at runtime: which tool ran, what happened between retrieval and generation, and whether a technically successful run produced the expected behavior.

For this experiment, I connected AgentInspect to GitHub Copilot CLI through AgentInspect’s read-only MCP server. I used a controlled, keyless TypeScript fixture so that the same failure could be reproduced without an API key or a nondeterministic model response.

Disclosure:I tested AgentInspect independently for the workflow described here. The maintainer reviewed the commands for technical accuracy; the conclusions are my own.@agent-inspect/mcp-server

was inPreviewwhen I ran this experiment.

The fixture represents a small project-help agent—the kind of assistant that could answer setup questions about a Next.js or MERN repository.

Its path was simple:

The retriever returned this shape:

type RetrievedChunk = {
  id: string;
  text?: string;
  content?: string;
};

But the formatter used the wrong optional field:

function formatContext(chunks: RetrievedChunk[]) {
  return chunks
    .map((chunk) => chunk.content ?? "") // Bug: the data is in `text`
    .join("\n\n");
}

TypeScript did not reject this because content

was permitted by the loose integration type. Retrieval succeeded, the formatter did not throw, and the answer step still ran. The visible symptom was only a generic “not enough project context” response.

This was exactly the kind of bug I wanted to test. There was no dramatic stack trace pointing to one bad line.

I tested with Node.js 20 or newer and pinned both AgentInspect packages to the published 6.17.4

baseline:

npm install agent-inspect@6.17.4
npm install --save-dev @agent-inspect/mcp-server@6.17.4

I wrapped the workflow with inspectRun()

and named the important boundaries with step()

:

import {
  inspectRun,
  observeOutcome,
  step,
} from "agent-inspect";

await inspectRun(
  "project-help-agent-broken",
  async () => {
    const chunks = await step.tool(
      "retrieve_project_docs",
      retrieveProjectDocs,
    );

    const context = await step(
      "format_context",
      () => formatContext(chunks),
      {
        type: "logic",
        metadata: {
          returnedField: "text",
          mapperField: "content",
          retrievedChunkCount: chunks.length,
        },
      },
    );

    await step(
      "validate_context",
      async () => {
        await observeOutcome("context_available", {
          expectation:
            "At least one retrieved chunk contributes usable context",
          status: context.trim().length > 0 ? "passed" : "failed",
          method: "custom",
          actual: {
            retrievedChunkCount: chunks.length,
            usableCharacterCount: context.trim().length,
          },
        });
      },
      {
        type: "logic",
        metadata: {
          retrievedChunkCount: chunks.length,
          usableCharacterCount: context.trim().length,
        },
      },
    );

    return step.llm(
      "deterministic-demo-model",
      () => generateAnswer(context),
    );
  },
  {
    traceDir: ".agent-inspect",
    silent: true,
    metadata: { scenario: "field-mapping-regression" },
  },
);

Two details are important here.

First, step.llm()

labels a boundary; it does not call a provider by itself. My fixture used a deterministic function so the experiment focused on the debugging loop rather than model variance.

Second, I recorded counts and field names—not raw project documents, prompts, or answers. AgentInspect uses metadata-only capture by default, but metadata is still data, so I kept it bounded and non-sensitive.

I ran the fixture, listed the latest trace, and inspected its report:

npx agent-inspect list --dir .agent-inspect
npx agent-inspect report <run-id> --dir .agent-inspect

The interesting result was this:

Status: success

Steps: 4 (1 LLM, 1 tool, 2 logic)

Observed outcomes
Total: 1 (passed 0, failed 1, unknown 0, skipped 0)

context_available: failed
Expectation: At least one retrieved chunk contributes usable context

The JavaScript function completed. The tool completed. The model-shaped step completed. Yet the behavioral outcome failed.

I turned that observation into a deterministic check:

npx agent-inspect check <run-id> \
  --dir .agent-inspect \
  --fail-on-observation failed
Check status: fail
Summary: 1 failed, 0 warning(s), 0 error(s)
outcome.status: Observed outcome count 1 matched [failed]

That exit code is useful in CI, but I still needed to diagnose the cause. This is where I brought in Copilot CLI.

GitHub Copilot CLI supports local stdio MCP servers. I added a project-level .mcp.json

file:

{
  "mcpServers": {
    "agent-inspect": {
      "type": "local",
      "command": "npx",
      "args": [
        "-y",
        "@agent-inspect/mcp-server@6.17.4",
        "--dir",
        ".agent-inspect"
      ],
      "tools": [
        "list_recent_runs",
        "get_trace_facts",
        "get_execution_tree",
        "get_first_causal_failure",
        "get_failed_observations",
        "compare_runs"
      ]
    }
  }
}

Copilot CLI also supports user-level configuration at ~/.copilot/mcp-config.json

, but I preferred project-level scope for this test. On first launch, Copilot asked me to trust the folder before its MCP configuration.

I verified the server and its tools from Copilot CLI:

/mcp show agent-inspect

The MCP server exposes 12 flagship read-only tools, plus compatibility tools. I enabled only the six needed for this experiment to keep the available tool set focused.

I did not ask Copilot to “fix my agent.” That is too open-ended and makes it easy to mix evidence with guesses.

Instead, I used this prompt:

Use the AgentInspect MCP tools to inspect the latest
project-help-agent-broken run.

Separate your response into:
1. persisted facts,
2. the first causal failure,
3. your likely diagnosis,
4. the source code I should inspect.

Do not edit any files until I approve the proposed change.

Copilot’s useful answer was not a magical explanation. It was a short chain grounded in the trace:

AgentInspect’s get_first_causal_failure

result identified the failed observed outcome and its parent step. It did not claim that adjacent events were causally linked just because their timestamps were close.

Copilot then inspected the formatter source and proposed a one-line change:

- .map((chunk) => chunk.content ?? "")
+ .map((chunk) => chunk.text ?? "")

That diagnosis made sense, but I still reviewed the retriever’s runtime shape and TypeScript type before accepting it. A coding assistant’s explanation is a hypothesis; the persisted trace facts and the source contract are the evidence.

After changing the mapping, I ran the same fixture again and checked the new trace:

npx agent-inspect check <fixed-run-id> \
  --dir .agent-inspect \
  --fail-on-observation failed
Check status: pass
Summary: 0 failed, 0 warning(s), 0 error(s)

The important result was not simply that the answer looked better. The same deterministic expectation that failed before now passed, while the intended execution path remained retrieval → formatting → validation → generation.

This gave me a more disciplined loop:

Failed behavior → local trace → bounded MCP facts → Copilot hypothesis → human review → code change → rerun → deterministic check

The boundary is worth understanding.

The AgentInspect MCP server did:

It did not:

Copilot CLI may have its own file-editing and shell tools. Those capabilities belong to Copilot, not to AgentInspect’s read-only MCP server. I deliberately required approval before any edit.

AgentInspect’s core capture and MCP process are local and do not upload data to AgentInspect by default. That does not mean the combined Copilot workflow is entirely offline.

When I connect an MCP server, its returned facts become available to the connected client. Copilot may send that context to its model service according to my GitHub plan, settings, organization policies, and applicable terms.

My practical rules are:

Local trace storage is valuable, but it does not remove the need for a data-handling decision when another tool reads those traces.

The strongest part of the experiment was search-space reduction. Instead of asking Copilot to read hundreds of interleaved log lines, I gave it a small evidence surface with named steps, bounded metadata, and an explicit failed outcome.

It was especially useful because the program did not crash. Traditional error-first debugging would have started in the wrong place.

There were also clear limitations:

I did not find an autonomous debugging system—and I do not think that should be the goal.

What I found was a useful division of responsibility:

For TypeScript developers already using Copilot CLI, this is a more reliable pattern than pasting a giant log and asking, “What went wrong?” The coding assistant still reasons probabilistically, but it starts from better evidence.

If you want to reproduce the same loop, start with a synthetic failure, keep capture metadata-only, and make Copilot separate facts from inferences. That separation was the most valuable part of the experiment.