{"slug": "github-copilot-evals-is-now-in-public-preview", "title": "GitHub Copilot Evals is now in public preview", "summary": "GitHub released GitHub Copilot Evals in public preview for Copilot Individual, Copilot Business, and Copilot Enterprise plans, letting users build repeatable tests for the skills, custom agents, and MCP tools in their Copilot setup. The tool, powered by GitHub's extensible local eval engine Vally, is available through a Visual Studio Code extension, a Copilot CLI plugin, and the GitHub Copilot app's Customize marketplace, and supports creating evals via natural language or the /eval command, running them with per-run approval, diagnosing failed checks, and comparing runs before and after a customization change. GitHub said the feature addresses output variability, since a single good answer does not show whether a customization works consistently.", "body_md": "# GitHub Copilot Evals is now in public preview\n\nYou can now use [GitHub Copilot Evals](https://aka.ms/evals) to optimize GitHub Copilot for yourself. As your prompts, models, skills, and tools change, are you still getting the most out of Copilot? Evals helps you answer that question for the skills, agents, and MCP tools in your setup. Create an eval, inspect how they behave, and use the results to decide what to improve.\n\n### [What’s an eval?](#whats-an-eval)\n\nAn eval is a repeatable test for AI behavior. You give an agent a realistic task, describe what a good result looks like, and check both the result and how the agent got there.\n\nAI output can change from one run to the next, so one good answer doesn’t tell you whether a customization works well for you. After changing a skill, agent, or MCP in your Copilot setup, rerun the same eval to check the behavior you care about and catch regressions.\n\n### [What you can do](#what-you-can-do)\n\nAsk Copilot in your own words or use the `/eval` command to:\n\n- **Create an eval** for a skill, custom agent, or MCP tool in your project.\n- **Run it** and see which checks passed and failed. You approve each run before it starts.\n- **Find out why a check failed** and decide what to change in your Copilot setup.\n- **Compare runs** from before and after a customization change to see whether it helped.\n\nGitHub Copilot Evals uses [Vally](https://aka.ms/vally), an extensible local eval tool, as its evaluation engine.\n\n### [Get started](#get-started)\n\nInstall GitHub Copilot Evals where you already use Copilot:\n\n- **Visual Studio Code:** Install the[GitHub Copilot Evals extension](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-evals) from the Visual Studio Marketplace.\n- **Copilot CLI:** Install the GitHub Copilot Evals plugin by following the[installation instructions](https://github.com/github/evals#install) .\n- **GitHub Copilot app:** Open**Customize** , add the`github/evals` marketplace, and install the GitHub Copilot Evals plugin.\n\nGitHub Copilot Evals is in public preview for Copilot Individual, Copilot Business, and Copilot Enterprise plans.\n\nTo learn more, see the [GitHub Copilot Evals documentation](https://aka.ms/evals). Share feedback by [opening an issue](https://github.com/github/evals/issues) or joining the discussion in [GitHub Community](https://github.com/orgs/community/discussions/categories/copilot-conversations).", "url": "https://wpnews.pro/news/github-copilot-evals-is-now-in-public-preview", "canonical_source": "https://github.blog/changelog/2026-11-17-github-copilot-evals-is-now-in-public-preview", "published_at": "2026-11-17 18:00:33+00:00", "updated_at": "2026-11-17 19:55:49.016852+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "ai-products"], "entities": ["GitHub", "GitHub Copilot Evals", "GitHub Copilot", "Vally", "Visual Studio Code", "Copilot CLI", "GitHub Copilot app"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/github-copilot-evals-is-now-in-public-preview", "markdown": "https://wpnews.pro/news/github-copilot-evals-is-now-in-public-preview.md", "text": "https://wpnews.pro/news/github-copilot-evals-is-now-in-public-preview.txt", "jsonld": "https://wpnews.pro/news/github-copilot-evals-is-now-in-public-preview.jsonld"}}