{"slug": "run-vally-evals-on-github-copilot-in-vs-code", "title": "Run Vally evals on GitHub Copilot in VS Code", "summary": "Microsoft released vally-vscode, an executor plugin for its Vally eval platform that runs evaluations against GitHub Copilot Chat inside the VS Code desktop app, closing the gap left by the earlier VS Code Controller extension, which drove the IDE but did not prepare workspaces or score outcomes. The plugin requires Node.js 22.14 or newer and VS Code 1.140 or newer with GitHub Copilot Chat installed and signed in, supports only stable VS Code (not Insiders), and is installed alongside the Vally CLI via npm. Because the executor determines which agent is evaluated, the same scenarios and graders can run against Copilot in VS Code or other Vally-supported agents, letting teams isolate whether a skill's pass or fail comes from the agent rather than the eval.", "body_md": "Recently, I introduced [VS Code Controller](https://blog.mastykarz.nl/evaluating-github-copilot-vs-code/), an extension that lets you control GitHub Copilot Chat in VS Code from an external process. It solved the hardest part of evaluating Copilot in VS Code: driving the IDE without brittle UI automation. But it deliberately stopped there: it doesn't prepare workspaces or score outcomes, so if you wanted to run evals, you had to build everything around it yourself.\n\nI've just released [vally-vscode](https://www.npmjs.com/package/vally-vscode), which fills that gap. It's an executor plugin for [Vally](https://aka.ms/vally) that runs your evals against GitHub Copilot Chat in the VS Code desktop app. You describe the scenario and how to grade it, and Vally takes care of the rest.\n\nVally is an eval platform from Microsoft. Every run in Vally moves through the same pipeline: a stimulus goes in, an executor runs the agent, the trajectory records what happened, and graders score the outcome. Executors are the swappable part of that pipeline, so adding VS Code as an executor lets you reuse the rest of Vally as is. You get prepared trial workspaces and built-in graders, and after a run, you explore the results in Vally's reports.\n\nAnd because the executor determines which agent you evaluate, VS Code is no longer a special case. Your scenarios and graders stay the same whether you run them against Copilot in VS Code or another agent that Vally supports. Since [the same model can behave differently depending on the agent it runs in](https://developer.microsoft.com/blog/the-hidden-variables-in-your-agent-eval/), keeping the scenario constant while changing the agent is how you find out whether your results transfer to the environment your users work in. When a skill passes with one agent and fails with another, you know that the difference comes from the agent rather than from a change to the eval.\n\nYou need Node.js 22.14 or newer and VS Code 1.140 or newer with GitHub Copilot Chat installed and signed in. The executor supports only stable VS Code, not Insiders, so that's where you install VS Code Controller, which vally-vscode uses to drive Copilot Chat. Then, add the Vally CLI and the executor to the project that contains your evals:\n\n```\ncode --install-extension waldek.vscode-controller\nnpm install --save-dev @microsoft/vally-cli vally-vscode\n```\n\nNext, describe what you want to evaluate. A Vally eval is a YAML file that lists stimuli, which are the prompts you send to the agent, together with the graders that score each outcome. The following eval, saved as `eval.yaml`, asks Copilot to create a file and then checks the workspace for the result:\n\n```\nname: copilot-file-creation\ndefaults:\n  executor:\n    name: vscode\n  model: auto\n  timeout: 5m\nstimuli:\n  - name: creates-the-requested-file\n    prompt: \"Create hello.txt containing exactly: Hello from Vally\"\n    graders:\n      - type: file-contains\n        config:\n          path: hello.txt\n          value: Hello from Vally\n```\n\nNotice that the grader looks at the file on disk rather than at Copilot's response. Agents sometimes report that they completed a task when they didn't, which is why [your graders should check what actually happened](https://blog.mastykarz.nl/eval-lying/). Checking the workspace also gives you a result that you can trust when you rerun the eval after changing your skill.\n\nFrom here, you can add your skills, instructions, fixtures, and MCP servers, and replace `auto` with a specific model. Use the model name exactly as it appears in Copilot's model picker. The executor checks which model served the request and fails the trial instead of silently accepting a substitute, so when you compare models, you know which one you measured.\n\nBefore you run the eval, configure Copilot to work without you. By default, Copilot asks for approval before it runs tools, and when an eval runs unattended, nobody is there to answer, so the trial stalls until it times out. To prevent that, add the following to your VS Code user settings:\n\n```\n{\n  \"chat.defaultToCopilotHarness\": true,\n  \"chat.defaultConfiguration\": {\n    \"mode\": \"autopilot\",\n    \"approvals\": \"allowAll\"\n  }\n}\n```\n\nThese settings let Copilot run terminal commands and edit files without confirmation, with your credentials and permissions, so only run evals that you trust, including their fixtures and MCP servers. Because these are user settings, they also apply to your regular chats, so restore your preferences when you're done evaluating.\n\nWith everything in place, close any other stable VS Code windows and start the eval with the following command. The `--executor-plugin` option tells Vally which plugin provides the `vscode` executor that your eval uses. If you don't want to pass it on every run, list the plugin in your project's `.vally.yaml` file instead.\n\n```\nnpx vally eval -e eval.yaml --executor-plugin vally-vscode\n```\n\nFor each trial, the executor opens VS Code in the workspace that Vally prepared, sends the prompt to Copilot Chat, waits for it to finish, and closes VS Code. If your stimulus has multiple turns, the executor sends them to the same chat, just like a developer following up on a response. Vally then runs your graders and reports the results.\n\nA passing grade confirms that the agent completed the task, but to understand [how it got there](https://blog.mastykarz.nl/agent-eval-more-than-result/) and [what it cost](https://blog.mastykarz.nl/eval-score-half-result/), you need to see what happened during the run. Out of the box, the trajectory of each trial contains only the prompts and Copilot's responses. To see more, install [Dev Proxy](https://aka.ms/devproxy) and trust its root certificate. Dev Proxy stores the certificate as `rootCert.pfx`, but the executor needs it in PEM format, so export it and point the executor to the exported file with the `DEV_PROXY_PEM` environment variable. On macOS, run:\n\n```\nexport DEV_PROXY_PEM=\"$HOME/Library/Application Support/dev-proxy/rootCert.pem\"\nopenssl pkcs12 -in \"$HOME/Library/Application Support/dev-proxy/rootCert.pfx\" -nokeys -passin pass: \\\n  | openssl x509 -out \"$DEV_PROXY_PEM\"\n```\n\nOn Linux, the commands are the same, except that Dev Proxy keeps its certificate in `~/.config/dev-proxy`:\n\n```\nexport DEV_PROXY_PEM=\"$HOME/.config/dev-proxy/rootCert.pem\"\nopenssl pkcs12 -in \"$HOME/.config/dev-proxy/rootCert.pfx\" -nokeys -passin pass: \\\n  | openssl x509 -out \"$DEV_PROXY_PEM\"\n```\n\nOn Windows, you don't need OpenSSL. Instead, export the certificate in PowerShell:\n\n``` php\n$env:DEV_PROXY_PEM = \"$env:LOCALAPPDATA\\Programs\\Dev Proxy\\rootCert.pem\"\n$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new(\"$env:LOCALAPPDATA\\Programs\\Dev Proxy\\rootCert.pfx\", \"\")\n\"-----BEGIN CERTIFICATE-----`n\" + [Convert]::ToBase64String($cert.RawData, \"InsertLineBreaks\") + \"`n-----END CERTIFICATE-----\" | Set-Content $env:DEV_PROXY_PEM\n```\n\nThese commands set `DEV_PROXY_PEM` only for the current terminal session, so start the eval from the same terminal, or add the variable to your shell profile to keep it. If Dev Proxy later regenerates its certificate, trust the new one and export it again, because the executor needs the certificate that Dev Proxy currently uses.\n\nWhen this variable is set, the executor starts its own Dev Proxy instance to record Copilot's API traffic. It then converts the recording to an [ATIF](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trajectory that shows the agent's reasoning and tool calls along with the tokens that each request used. You'll find the trajectory next to the recorded traffic in each trial's artifacts. With that information, you can answer the questions that a score leaves open: Did the agent use the tool you built for it? How many requests did it take to get to the result? Did your new instructions make the agent more effective, or just more expensive? Recordings contain your task and everything the agent generated while working on it, so review them before you share them with anyone.\n\nThe executor runs the same VS Code that developers use, which is both its purpose and its main constraint. Instead of running headless, it opens and closes the desktop app with your signed-in profile. And because VS Code runs one process per profile, trials run one at a time, and you can't use stable VS Code yourself while an eval is running.\n\nRunning with your profile also means that your installed extensions, settings, organization policies, and available models are part of the environment you measure. That's realistic, because your users run Copilot with their own setup too, but it also means that you need to keep that environment consistent when you compare runs. If you're evaluating your own VS Code extension, keep it disabled in your profile and use the executor's `extensions` option to enable it only for the trial workspace. That way, you can run the same eval with and without it, and see whether it's [worth the tokens it adds](https://developer.microsoft.com/blog/is-your-agent-extension-actually-working/).\n\nIf your users work with GitHub Copilot in VS Code, that's where you should evaluate what you build for them. [vally-vscode is available on npm](https://www.npmjs.com/package/vally-vscode), with detailed configuration and troubleshooting guidance in its readme. It's open source, and I'd love to hear how you use it, so share your feedback and ideas [on GitHub](https://github.com/waldekmastykarz/vally-vscode).", "url": "https://wpnews.pro/news/run-vally-evals-on-github-copilot-in-vs-code", "canonical_source": "https://blog.mastykarz.nl/vally-evals-github-copilot-vs-code/", "published_at": "2026-11-12 14:40:00+00:00", "updated_at": "2026-11-15 16:19:18.684882+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "large-language-models"], "entities": ["Microsoft", "Vally", "vally-vscode", "GitHub Copilot Chat", "VS Code", "VS Code Controller", "Node.js"], "also_reported_by": [], "takeaway": "Installing vally-vscode with the Vally CLI lets developers run YAML-defined evals against Copilot Chat in stable VS Code 1.140+ on Node.js 22.14+, with graders such as file-contains checking the workspace on disk rather than the agent's response.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/run-vally-evals-on-github-copilot-in-vs-code", "markdown": "https://wpnews.pro/news/run-vally-evals-on-github-copilot-in-vs-code.md", "text": "https://wpnews.pro/news/run-vally-evals-on-github-copilot-in-vs-code.txt", "jsonld": "https://wpnews.pro/news/run-vally-evals-on-github-copilot-in-vs-code.jsonld"}}