{"slug": "github-copilot-mcp-usage-metrics-without-false-signals", "title": "GitHub Copilot MCP Usage Metrics Without False Signals", "summary": "GitHub's Copilot usage metrics API now reports the customer-configured names of skills, custom agents and MCP servers, and each record can carry up to 25 entries instead of five, letting organizations see which internal tools are actually invoked. A developer writing on mudassirkhan.me cautions that the richer data still measures activity, not outcomes, and recommends normalizing tool names, flagging records that hit the 25-entry cap, and joining usage counts with per-run evaluation results at a tool-name-plus-day grain rather than per user.", "body_md": "GitHub's latest Copilot usage metrics update finally names things. Instead of lumping every extension into one undifferentiated bucket, the usage metrics API now reports the names of your customer configured skills, custom agents and MCP servers, and each record can carry 25 entries instead of five. That is great news for adoption dashboards, and a trap for anyone who reads \"used a lot\" as \"works well\".\n\n**TL;DR**\n\nShort version: named entries, and more of them.\n\nBefore this change, if your org ran three internal MCP servers and a handful of custom agents, the usage data mostly told you that \"something custom\" got used. Now the record carries the names you configured, which means you can answer questions like \"is anyone actually using the read only billing server we shipped last sprint?\" without sending a survey nobody fills in.\n\nThe jump from five to 25 entries per record matters more than it sounds. With five slots, a busy developer who touched a dozen tools in a day got truncated, and the long tail (usually the internal tools you care about most) fell off the end. At 25, most of that tail survives.\n\nWhat it does not tell you: whether the call helped. A connection is not a success, and an invocation is not a merged pull request. That gap is the whole point of this post, and it is the same gap I keep running into when [evaluating LLM agents in production](https://mudassirkhan.me/blog/llm-agent-evaluation-production): activity metrics look fantastic right up to the moment you check outcomes.\n\nPull the data on a schedule and land it in your own table. Don't build dashboards straight off the API response. You'll want history, and you'll want to join it with your own data later.\n\nI'm not going to reproduce GitHub's response schema here; read it from the API docs for your plan and map the field names once. What matters is the normalized shape you store. Here is the normalizer I'd write, with the raw record type kept deliberately loose:\n\n```\n// normalize-usage.ts\n// Map whatever the usage metrics API returns into one row per named tool, per user, per day.\n\ntype RawUsageRecord = {\n  day: string; // map from the API's date field\n  user: string; // map from the API's user identifier\n  entries: Array<{\n    kind: \"skill\" | \"agent\" | \"mcp_server\";\n    name: string; // the customer configured name, now reported\n    interactions: number;\n  }>;\n};\n\nexport type ToolUsageRow = {\n  day: string;\n  user: string;\n  toolKind: \"skill\" | \"agent\" | \"mcp_server\";\n  toolName: string;\n  interactions: number;\n  truncated: boolean; // the record hit the entry limit, so its tail may be missing\n};\n\nconst ENTRY_LIMIT = 25;\n\nexport function normalize(records: RawUsageRecord[]): ToolUsageRow[] {\n  return records.flatMap((record) =>\n    record.entries.map((entry) => ({\n      day: record.day,\n      user: record.user,\n      toolKind: entry.kind,\n      toolName: entry.name.trim().toLowerCase(),\n      interactions: entry.interactions,\n      truncated: record.entries.length >= ENTRY_LIMIT,\n    }))\n  );\n}\n```\n\nTwo details in there are easy to skip and expensive later. Normalize tool names, because people rename servers and casing drifts. And keep the `truncated` flag: if a record is full at 25, the long tail might still be cut, and your dashboard should say so instead of pretending the count is complete.\n\nThis is where usage metrics turn into something you can actually decide with. Counts tell you who touched a tool. Outcomes tell you whether touching it was worth it. You need both on the same row.\n\nThe join key is the hard part. GitHub's numbers are aggregated, your evaluation results are per run, so pick a grain both sides can honestly share. Tool name plus day is the one I'd reach for. Per user is tempting, but it quietly turns an adoption report into a surveillance report, and people will route around it.\n\nOn your side, emit your own spans for agent operations and their outcomes. OpenTelemetry's semantic conventions (version 1.44.0) include GenAI attributes for operations and for evaluations, which gives you a vendor neutral way to separate \"the agent did something\" from \"what it did was any good\":\n\n``` js\n// record-outcome.ts\nimport { trace } from \"@opentelemetry/api\";\n\nconst tracer = trace.getTracer(\"agent-evals\");\n\nexport async function recordToolRun(\n  toolName: string,\n  run: () => Promise<{ passed: boolean; score: number }>\n) {\n  return tracer.startActiveSpan(`execute_tool ${toolName}`, async (span) => {\n    span.setAttribute(\"gen_ai.operation.name\", \"execute_tool\");\n    span.setAttribute(\"gen_ai.tool.name\", toolName);\n    try {\n      const result = await run();\n      // Check the GenAI attribute registry for the exact evaluation names in your semconv version.\n      span.setAttribute(\"gen_ai.evaluation.name\", \"task_passed\");\n      span.setAttribute(\"gen_ai.evaluation.score.value\", result.score);\n      return result;\n    } finally {\n      span.end();\n    }\n  });\n}\n```\n\nThen the join itself is boring SQL, which is exactly what you want. Aggregate each side first, otherwise the join multiplies your counts:\n\n```\nWITH usage AS (\n  SELECT day, tool_name,\n         SUM(interactions) AS interactions,\n         BOOL_OR(truncated) AS any_truncated\n  FROM tool_usage\n  GROUP BY day, tool_name\n),\nevals AS (\n  SELECT day, tool_name,\n         COUNT(*) AS evaluated_runs,\n         AVG(CASE WHEN passed THEN 1.0 ELSE 0.0 END) AS pass_rate\n  FROM tool_evals\n  GROUP BY day, tool_name\n)\nSELECT u.day, u.tool_name, u.interactions, u.any_truncated,\n       e.evaluated_runs, e.pass_rate\nFROM usage u\nLEFT JOIN evals e USING (day, tool_name)\nORDER BY u.day DESC, u.interactions DESC;\n```\n\nA `NULL` in `evaluated_runs` is not a bug in the query. It is a finding: a tool people use that nobody is measuring.\n\nCopilot counts are GitHub telemetry, and telemetry has holes. Three of them will bend your numbers if you ignore them.\n\nSo treat every count as a floor, compare a tool against its own trend rather than against other tools, and never turn the data into a team leaderboard. My opinion, from building dashboards like this: the first chart someone screenshots is \"most used tools\", and it is the chart most likely to mislead.\n\nThere is a security angle too. OWASP LLM06 describes excessive agency as coming from three drivers: excessive functionality, excessive permissions and excessive autonomy. Instrumentation alone does not enforce any of those boundaries. A popular MCP server with write access it doesn't need is still a footgun; the usage chart just tells you how many people are holding it. If you're reviewing those boundaries, my [production defense guide for AI agent security](https://mudassirkhan.me/blog/ai-agent-security-prompt-injection) covers the permission side in detail.\n\nOnce usage and outcomes sit on the same row, the rollout call stops being a vibe. This is the table I'd pin in the platform team's channel:\n\n| Usage | Outcomes | Decision | \n|---|---|---|\n| High | Good | **Expand.** Make it a default and document it. | \n| High | Poor | **Fix or restrict.** People want it and it is hurting them. | \n| Low | Good | **Promote.** This is a discoverability problem. | \n| Low | Poor | **Retire.** Remove the server and its permissions. | \n| Any | No evaluations | **Wait.** Add evaluation before you decide anything. | \n\nThe last row is the one teams skip. A tool with heavy usage and zero evaluations feels like a win, and that feeling is exactly the false signal this whole setup exists to catch.\n\nPull the usage metrics API on a schedule into your own table, normalize tool names, and keep a flag for records that hit the 25 entry limit. Emit your own OpenTelemetry spans for agent operations and evaluation results, join both sides on tool name and day, and make rollout decisions from usage and outcomes together. Keep permission boundaries enforced separately, because instrumentation does not enforce them.\n\nGitHub's usage metrics API now reports the names of customer configured skills, custom agents and MCP servers instead of grouping them together, and each record can hold 25 entries instead of five. That makes internal tool adoption visible, but the counts still measure activity, not success.\n\nRelated reading: [The LLM evaluation metrics that actually predict production quality](https://mudassirworks.hashnode.dev/llm-evaluation-metrics-what-actually-predicts-production-quality) and [MCP server security in production: what actually breaks](https://dev.to/mudassirworks/mcp-server-security-in-production-what-actually-breaks-3n3a).\n\n*If you're still deciding which agent stack to measure in the first place, the [AI agent framework chooser](https://mudassirkhan.me/tools/ai-agent-framework-chooser) narrows it down in a few questions.*\n\n*Drop a comment if your setup looks different. Curious how people are joining usage data to outcomes in production.*", "url": "https://wpnews.pro/news/github-copilot-mcp-usage-metrics-without-false-signals", "canonical_source": "https://dev.to/mudassirworks/github-copilot-mcp-usage-metrics-without-false-signals-303e", "published_at": "2026-10-30 18:43:18+00:00", "updated_at": "2026-10-30 18:50:38.912933+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "developer-tools", "mlops", "ai-tools"], "entities": ["GitHub", "GitHub Copilot", "OpenTelemetry", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/github-copilot-mcp-usage-metrics-without-false-signals", "markdown": "https://wpnews.pro/news/github-copilot-mcp-usage-metrics-without-false-signals.md", "text": "https://wpnews.pro/news/github-copilot-mcp-usage-metrics-without-false-signals.txt", "jsonld": "https://wpnews.pro/news/github-copilot-mcp-usage-metrics-without-false-signals.jsonld"}}