Code Review Tools: Copilot Setup in 12 Steps, 80 Min [2026]

GitHub released Copilot code review Max in public preview on October 28, 2026, and on November 6, 2026 added a bypass list for the automatic-review rule plus REST and GraphQL management of review settings from enterprise policy down to a single repository. GitHub describes Max as the highest-F1 option on its own ReviewBench evaluation for high-severity issues, and code review is listed on Copilot Pro, Pro+, Max, Business and Enterprise plans, with each review consuming GitHub AI Credits and GitHub Actions compute. The article's 12-step, roughly 80-minute lab plants four bugs and one harmless control change in a repository and uses a scorecard script to count what Copilot flags, what it invents, and how many minutes the first review takes.

GitHub introduced Copilot code review Max https://tooltier.dev/blog/ai-tool-pricing-changes-nov-2-2026 in public preview on October 28, 2026, then gave the automatic-review rule a bypass list on November 6. That same November 6 release put REST and GraphQL management behind the review settings, from enterprise policy down to a single repository. Taken together, the changes turn Copilot into one of the few code review tools you can configure, audit and script like infrastructure. The lab repo below plants four bugs and one harmless control change. A scorecard script then counts what Copilot flags, what it invents, and how many minutes the first review takes. Twelve steps, about 80 minutes, and you finish with a working project you can fork for a team pilot https://shattered.io/best-ai-coding-assistants-team-pilot-12-steps-2026/ . If you would rather assemble your own reviewer from Semgrep, reviewdog and a model API, the script-based Actions tutorial https://shattered.io/ai-code-review-github-actions-semgrep-reviewdog-12-steps-2026/ covers that route. Measuring matters because the headline numbers come from vendors. GitHub describes Max as the highest-F1 option on its own ReviewBench evaluation for high-severity issues. That tells you something about GitHub’s test set. Your codebase, your conventions and your instructions decide whether the result holds for you, and a five-PR experiment answers the question in an afternoon. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F What Copilot Code Review Does in November 2026 Copilot code review reads a pull request, leaves inline comments, and proposes fixes you can apply with a click. Two meters apply to every review. Each review consumes GitHub AI Credits, and it also draws on GitHub Actions compute at the applicable rates. A review that looks free in the pull request UI still shows up on two bills. Plan access is straightforward. GitHub lists code review on Copilot Pro, Pro+, Max, Business and Enterprise, and the November 6 settings API covers the same five plans. Copilot Free keeps limited completions, chat and agent usage but does not appear in the code review plan list. Effort is the main quality dial. GitHub’s medium review tier routes pull requests to a higher-reasoning model. On October 28, it introduced Max as its deepest review mode and made Copilot approvals generally available. Step 8 tests those tiers against your own bugs instead of trusting a benchmark chart. Model news can distract from this. Copilot Fusion https://shattered.io/github-copilot-fusion-ga-three-effort-tiers-30-percent-discount-2026/ went generally available on October 28 for VS Code, the Copilot app, the CLI and JetBrains IDEs, where it routes work between models. The sources checked for this build do not say that code review runs through Fusion, and the November 6 release describes product and API changes rather than a new underlying model. Treat review effort as its own setting and test it as one. The table lists the changes that shape this setup, newest last. The final column points to the step where each one matters. | Date 2026 | Change | Used in | |---|---|---| | June 2 | Medium review tier routes pull requests to a higher-reasoning model. Admins set repository guidance to low or medium. | Step 8 | | June 12 | Organization-level runner controls standard, self-hosted, larger . The 4,000-character limit on instruction files is gone. | Steps 4, 8 | | July 17 | Review reads instructions from the pull request head branch. A repository can define .github/workflows/copilot-code-review.yml for review-specific setup. | Steps 9, 12 | | October 28 | Copilot code review Max enters public preview. Copilot approvals reach general availability. | Step 8 | | November 6 | REST and GraphQL management of review settings, from enterprise policy to single repositories. | Advanced tips | | November 6 | The automatic-review rule supports a bypass list for exempt actors. | Step 7 | | November 6 | Review comments can expose the skills, MCP servers and custom instructions used as their sources. | Step 12 | | November 6 | Actions, Dependabot and org-owned apps can request reviews on bot-authored pull requests when external-license blocking is on and the organization pays. | Pitfalls | Prerequisites, Versions and Plan Costs Collect these before Step 1. Everything runs from a terminal, and the whole lab fits inside a private repository. - GitHub account on Copilot Pro, Pro+, Max, Business or Enterprise. Free is not in the code review plan list. - Admin rights on a repository you create for the lab. Rulesets need them. On an organization repo, ask an owner to confirm Copilot may review there. - A recent version of Git for the branch and patch commands. - GitHub CLI , a current release, signed in with gh auth login . It must run gh pr and gh api . - A recent Python release for the scorecard. The script uses only the standard library. - A credit budget set in your Copilot settings before the first review, so a runaway loop cannot surprise you. GitHub prices Copilot usage in AI Credits, and one credit equals $0.01. A $10 additional-usage budget therefore covers 1,000 credits, and GitHub sends alerts at 75%, 90% and 100% of any budget you configure. Copilot Pro costs $10 a month and includes 1,500 monthly credits 1,000 base plus a 500-credit flex allotment , which works out to $15 of usage. The Actions side changed on October 28, when GitHub replaced fixed standard-runner minutes with a shared, dollar-based compute pool https://lizard.build/blog/github-actions-pricing that standard runners, cloud sandboxes and, on Team and Enterprise Cloud, larger runners all draw from. | Plan | Price | Monthly AI credits | Code review | |---|---|---|---| | Copilot Free | $0 | Limited usage | Not in the plan list | | Copilot Pro | $10 per month | 1,500 $15 of usage | Yes | | Copilot Pro+ | $39 per month | 7,000 $70 of usage | Yes | | Copilot Max | $100 per month | 20,000 $200 of usage | Yes | | Copilot Business | $19 per user per month | Credits aligned with seat price | Yes | | Copilot Enterprise | $39 per user per month | Credits aligned with seat price | Yes | | Copilot Standalone | $10 per user per month | Pay from available budget | Needs budget | Enterprise buyers got a packaging change on the same date. GitHub said GitHub Enterprise now includes Copilot access https://pondero.ai/news/2026-11-03-copilot-enterprise-bundle/ for eligible licensed users, delivered through GitHub or Microsoft account teams while self-service availability is still staged. Users inside an enterprise account who lack a GitHub Enterprise license can take Copilot Standalone at $10 per user per month. Code review, chat and cloud features still need available budget in every case. For seat-price context against other agents, the three-way pricing comparison https://shattered.io/claude-code-vs-cursor-vs-github-copilot-pricing-limits-2026/ lines up Copilot, Cursor and Claude Code. The sources behind this build publish no per-review credit cost, so measure your own: run the five lab pull requests, read the usage page, and multiply by your weekly pull request volume. Steps 1 to 3: Create the Lab Repository and Plant the Bugs Step 1: scaffold the project. The baseline app is deliberately boring. Four small modules, each written correctly, so every later pull request changes exactly one thing. mkdir review-lab && cd review-lab git init -b main mkdir -p app tests scripts/patches fixtures .github/instructions .github/workflows touch app/ init .py printf ' pycache /\n.pytest cache/\n.seed-cases.tmp\n' .gitignore Create the four modules and one test file with the contents below. Money lives in integer cents, the SQL uses bound parameters, file reads resolve against a base directory, and token checks use a constant-time comparison. python app/db.py import sqlite3 from contextlib import closing def get user conn: sqlite3.Connection, email: str : with closing conn.cursor as cur: cur.execute "SELECT id, email, plan FROM users WHERE email = ?", email, return cur.fetchone app/files.py from pathlib import Path BASE DIR = Path "/srv/reports" .resolve def read report name: str - str: target = BASE DIR / name .resolve if BASE DIR not in target.parents: raise ValueError "report path escapes the reports directory" return target.read text encoding="utf-8" app/auth.py import hmac import os def secret - bytes: return os.environ "API TOKEN" .encode def is valid token candidate: str - bool: return hmac.compare digest candidate.encode , secret app/billing.py from decimal import Decimal, ROUND HALF UP def invoice total unit cents: int, quantity: int, tax rate: Decimal - int: subtotal = Decimal unit cents quantity total = subtotal Decimal 1 + tax rate return int total.quantize Decimal "1" , rounding=ROUND HALF UP python from decimal import Decimal from app.billing import invoice total def test invoice total rounds half up : assert invoice total 1999, 3, Decimal "0.0825" == 6492 def test invoice total zero quantity : assert invoice total 1999, 0, Decimal "0.0825" == 0 Step 2: write the bug patches. Each patch rewrites one correct block into a flawed one. Here is the SQL injection patch. It swaps a bound parameter for an f-string that splices the email straight into the query. diff --git a/app/db.py b/app/db.py index 7b9581b..d2c2f66 100644 --- a/app/db.py +++ b/app/db.py @@ -4,5 +4,6 @@ from contextlib import closing def get user conn: sqlite3.Connection, email: str : with closing conn.cursor as cur: - cur.execute "SELECT id, email, plan FROM users WHERE email = ?", email, + query = f"SELECT id, email, plan FROM users WHERE email = '{email}'" + cur.execute query return cur.fetchone Create the other three patches the same way: traversal.patch removes the resolve-and-check guard in read report , timing.patch replaces hmac.compare digest with a plain == , and float-money.patch converts the invoice math from Decimal to float with int truncation. The fifth patch, docstring-only.patch , adds one docstring to get user and changes no behavior. It is the control: a reviewer that complains about it is generating noise. | Case | File and lines | Bug class | Why a reviewer should care | |---|---|---|---| | sqli | app/db.py, 7-8 | SQL injection | User input reaches the query text. | | traversal | app/files.py, 7-8 | Path traversal | A name like ../etc/passwd escapes the reports directory. | | timing | app/auth.py, 10 | Timing side channel | Plain equality leaks how many leading bytes matched. | | float-money | app/billing.py, 5-7 | Rounding error | Float math and truncation drop cents on invoices. | | docstring-only | app/db.py, 6 | Control | Harmless. Any inline comment counts as noise. | Record where each bug lives so the scorecard can check it later. The line ranges describe the files after the patch applies. { "bugs": {"id": "sqli", "branch": "bug/sqli", "file": "app/db.py", "line start": 7, "line end": 8, "patch": "scripts/patches/sqli.patch"}, {"id": "traversal", "branch": "bug/traversal", "file": "app/files.py", "line start": 7, "line end": 8, "patch": "scripts/patches/traversal.patch"}, {"id": "timing", "branch": "bug/timing", "file": "app/auth.py", "line start": 10, "line end": 10, "patch": "scripts/patches/timing.patch"}, {"id": "float-money", "branch": "bug/float-money", "file": "app/billing.py", "line start": 5, "line end": 7, "patch": "scripts/patches/float-money.patch"} , "controls": {"id": "docstring-only", "branch": "control/docstring-only", "patch": "scripts/patches/docstring-only.patch"} } Step 3: commit the baseline and push main. The ruleset in Step 6 targets the default branch, so it must exist on GitHub first. git add -A git commit -m "baseline: clean app, tests and bug patches" gh repo create review-lab --private --source . --push Expect three confirmation lines from the GitHub CLI: repository created, remote added, commits pushed. Open the repository in a browser and confirm that main shows the app folder and the patches. Steps 4 and 5: Write the Instructions Copilot Reads Step 4: add repository-wide instructions. Copilot code review supports repository and organization guidance through custom instructions, and repository administrators can control whether reviews use them. The repository file lives at .github/copilot-instructions.md . The old 4,000-character ceiling on instruction files has been removed, but keep the file short anyway. Long files bury the three rules that matter. Review instructions for review-lab Python 3.12 service. Money is stored as integer cents. All database access goes through app/db.py. What to flag - String-built SQL. Every query must use bound parameters. - File paths built from user input without resolving them and checking against BASE DIR. - Secret or token comparisons that use == instead of hmac.compare digest. - Float arithmetic on money values. Use Decimal and round half up. - Changed behavior without a changed or new test. What to skip - Formatting and import order ruff handles both in CI . - Docstring wording, naming taste, and type-hint style. - Suggestions to add logging unless an error is swallowed. Comment style - One finding per comment, with the input or the line that breaks. - Start with a severity word: blocker, major, or minor. - Offer a code suggestion only when the fix is under ten lines. Three sections do the work. What to flag names the failure patterns of this codebase in concrete terms. What to skip removes the comments your linter already makes, which is the quickest way to cut noise. Comment style controls shape: one finding per comment, severity first, suggestions only when short. Writing the skip list matters as much as the flag list. Reviewers that comment on everything train developers to ignore all comments. Teams that already keep rules for another assistant can reuse the thinking. The guide on converting Copilot custom instructions into Cursor rules https://shattered.io/cursor-rules-migrate-github-copilot-2026/ shows which parts of an instruction file survive a tool switch and which are Copilot-specific. Step 5: add path-scoped instruction files. Files under .github/instructions/ end in .instructions.md and carry an applyTo glob in their frontmatter. They apply only to matching files, so test rules stay out of application reviews and the reverse. GitHub’s documentation also describes an excludeAgent frontmatter key for limiting a file to the cloud agent or to code review, so check the current docs if you need that split. --- applyTo: "app/ / .py" --- - Treat every argument that reaches sqlite3, pathlib, or os.environ as untrusted. - Raise ValueError for bad input. Do not return None to signal an error. - Flag a missing type hint only on public functions. --- applyTo: "tests/ / .py" --- - A changed rounding or tax rule needs a test at a half-cent boundary. - Do not comment on test names or fixture style. - Flag tests that only assert that no exception was raised. Commit and push both steps together. Do this before opening any lab pull request, because the lab branches fork from main. git add .github git commit -m "review: repo-wide and path-scoped Copilot instructions" git push Steps 6 and 7: Turn On Automatic Review With a Ruleset Step 6: create the ruleset. Open the repository, then Settings , Rules , Rulesets , New branch ruleset . Name it copilot-auto-review , set enforcement to Active , and target the default branch. Under branch rules, enable Automatically request Copilot code review . Two options sit beneath it: review new pushes, and review draft pull requests. Leave drafts off for now and turn on new-push reviews, so every update gets a fresh look. Button labels shift slightly between releases, so match them by meaning. You can create the same ruleset from the terminal, which helps when you roll the setup out to many repositories. The JSON below uses the copilot code review rule type. { "name": "copilot-auto-review", "target": "branch", "enforcement": "active", "conditions": { "ref name": { "include": "~DEFAULT BRANCH" , "exclude": } }, "rules": { "type": "copilot code review", "parameters": { "review on push": true, "review draft pull requests": false } } , "bypass actors": } gh api repos/{owner}/{repo}/rulesets --method POST --input ruleset.json gh api repos/{owner}/{repo}/rulesets --jq '. | {id, name, enforcement}' The second command reads the rulesets back. Expect one entry named copilot-auto-review with enforcement active . If the first command returns a 422 error, build the ruleset once in the web UI, read it back with gh api repos/{owner}/{repo}/rulesets/ID , and copy the schema it shows. The rule type and parameter names are the part most likely to drift between releases. Step 7: set the bypass list. On November 6, 2026, GitHub added bypass support to the automatic-review rule, so exempt actors no longer get automatic Copilot reviews. Use it for the authors whose changes do not justify a review bill: release bots, documentation-only contributors, and version-bump automation. Add them under Bypass list in the ruleset, or fill the bypass actors array with the actor types the REST documentation lists. Two behaviors are easy to misread. A bypass stops the automatic request. Anyone can still ask for a review by hand on that pull request. And bot-authored pull requests follow their own rule: since November 6, GitHub Actions, Dependabot and organization- or enterprise-owned apps can request reviews on bot-authored pull requests when external-license blocking is enabled, provided the organization pays for the review. Decide deliberately which of those you want, because Dependabot can open dozens of pull requests in a week. Teams weighing how far Copilot’s admin controls go can read the Copilot versus Claude Code matchup https://shattered.io/github-copilot-vs-claude-code-2026/ , which covers admin controls and cloud agents side by side. For the lab, leave the bypass list empty. The five test pull requests come from your own account and should all get reviewed. Step 8: Pick the Review Effort and Runner Effort and runner choices decide both quality and cost. GitHub exposes them at different scopes, so know which one you are changing. The table summarizes what the sources confirm. | Setting | Scope | Options confirmed | Effect to measure | |---|---|---|---| | Repository review guidance | Repository | Low or medium | Medium routes pull requests to a higher-reasoning model. | | Max review mode | Eligible plans | Public preview since October 28 | Deepest mode. GitHub calls it the highest-F1 option on its ReviewBench for high-severity issues. | | Runner type | Organization | Standard GitHub-hosted default , self-hosted, larger | Queue time and Actions spend. Org owners can lock a default. | | Review on new pushes | Ruleset | On or off | More reviews per pull request, more credits. | | Draft pull request review | Ruleset | On or off | Early feedback versus cost on unfinished work. | Run the experiment in two passes. First, leave the repository on its default guidance and score the five pull requests Steps 10 and 11 . Then switch the repository to the highest level your plan offers, close and reopen the pull requests to trigger fresh reviews, and score again. Keep both outputs. The point is a number for your repo: how many extra planted bugs the higher tier catches, and what it costs in credits. Reviews run on a standard GitHub-hosted runner unless your organization changed that. Self-hosted runners suit teams that need private network access or fixed hardware, but they add a failure mode: if the runner is offline, the review sits pending instead of failing loudly. Larger hosted runners draw from the shared compute pool on Team and Enterprise Cloud plans. These settings govern pull request review only. Developers who work in Xcode get Copilot through a separate surface, which entered public preview with the Xcode 27.3 beta, and the Xcode and JetBrains Air coverage https://shattered.io/github-copilot-xcode-jetbrains-air-two-days-2026/ explains how that rollout fits the wider Copilot lineup. Max costs more to run than the lower tiers in all likelihood, since a deeper review does more work, but the sources checked here give no credit figure, so treat that as a hypothesis and read the usage page. One more caution on interpretation: GitHub’s F1 claim covers high-severity issues on its benchmark. Your four planted bugs are a different distribution, and a tier that wins there may tie on your code. Step 9: Add the Review Setup Workflow On July 17, 2026, GitHub let a repository define .github/workflows/copilot-code-review.yml for review-specific setup steps. That covers dependency installation, tooling, runner preparation and other environment configuration, so the agent that reviews a pull request starts from an environment that resembles your CI. The lab app has no heavy dependencies, but the file shows the shape you would extend for a real service. name: Copilot code review setup on: workflow dispatch: pull request: paths: - ".github/workflows/copilot-code-review.yml" jobs: copilot-setup-steps: runs-on: ubuntu-latest timeout-minutes: 15 permissions: contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install review tooling run: python -m pip install pytest ruff The job id copilot-setup-steps follows the convention GitHub uses for Copilot’s cloud agent setup file. The changelog names the review file but the sources checked here do not spell out a required job id, so confirm the name against the current documentation before you rely on it. The extra triggers let you test the file by hand and whenever you edit it. git add .github/workflows/copilot-code-review.yml git commit -m "ci: review setup workflow" git push gh workflow run copilot-code-review.yml gh run list --workflow copilot-code-review.yml --limit 1 Expect the first command to confirm a workflow dispatch event for the workflow, and the run list to show one run moving from queued to completed within a couple of minutes. A red run here means the setup environment is broken, so fix it before you trust any review that depends on it. Keep this file lean. Every minute spent installing dependencies draws from the same Actions compute pool as the review itself, so install only what a reviewer would plausibly run: the test runner and the linter, not the whole production image. Step 10: Seed the Pull Requests The seed script reads planted.json , creates one branch per case from main, applies the matching patch, and opens a pull request. It refuses to run on a dirty tree, because git add on a branch would otherwise sweep unrelated files into the bug pull requests. bash /usr/bin/env bash set -euo pipefail Opens one PR per patch in planted.json. Run from the repo root on a clean, pushed main. test -z "$ git status --porcelain " || { echo "Commit or stash first: tree is dirty" &2; exit 1; } git checkout -q main python3 - <<'PY' .seed-cases.tmp import json plan = json.load open "planted.json" for case in plan "bugs" + plan "controls" : print f'{case "branch" }|{case "patch" }|{case "id" }' PY while IFS='|' read -r branch patch id; do git checkout -q main git checkout -q -B "$branch" git apply "$patch" git add app tests git commit -qm "lab: $id" git push -q -u origin "$branch" --force gh pr create --base main --head "$branch" --title "lab: $id" --body "Planted case: $id" /dev/null echo "opened PR for $id" done < .seed-cases.tmp git checkout -q main rm -f .seed-cases.tmp Save it as scripts/seed prs.sh , commit it, and run it from the repository root. git add scripts/seed prs.sh git commit -m "lab: seed script" git push bash scripts/seed prs.sh opened PR for sqli opened PR for traversal opened PR for timing opened PR for float-money opened PR for docstring-only That output comes from running the script against a local test remote with a stub for the GitHub CLI, so the branch logic is verified. Against GitHub, each line arrives a second or two apart as the pull requests open. Now wait. Open one pull request and look at the Reviewers sidebar. Copilot should appear within moments, and the review lands as a normal review with inline comments. If the sidebar stays empty after ten minutes, jump to the troubleshooting list. Do not edit the pull requests while reviews run, because a push on a branch with new-push review enabled starts another review and muddies the first measurement. To request a review by hand, for example after switching effort levels, use the Reviewers menu on the pull request. Some GitHub CLI releases also accept the @copilot handle, as in gh pr edit 3 --add-reviewer @copilot . Try it, and fall back to the menu if your version rejects the handle. Step 11: Score the Reviews The scorecard turns five pull requests into three numbers: catch rate, noise, and minutes to first review. It lists pull requests with gh pr list , pulls reviews and inline comments from the REST API, and keeps only the ones written by a login containing copilot . A bug counts as caught when a Copilot inline comment lands on the right file within two lines of the planted range. Every other Copilot inline comment counts as noise, including all comments on the control. bash /usr/bin/env python3 """Score Copilot code review against planted bugs. Usage: scorecard.py OWNER/REPO --fixture FILE """ import argparse import json import subprocess import sys from datetime import datetime SLACK = 2 lines of tolerance around a planted range def gh json args : out = subprocess.run "gh", args , check=True, capture output=True, text=True .stdout dec, pos, items = json.JSONDecoder , 0, while pos < len out : while pos < len out and out pos .isspace : pos += 1 if pos = len out : break obj, pos = dec.raw decode out, pos items.extend obj if isinstance obj, list else obj return items def ts value : return datetime.fromisoformat value.replace "Z", "+00:00" def is copilot user : return "copilot" in user or {} .get "login", "" .lower def load repo, fixture : if fixture: data = json.load open fixture return data "prs" , data "reviews" , data "comments" prs = gh json "pr", "list", "--repo", repo, "--state", "all", "--limit", "100", "--json", "number,headRefName,createdAt" reviews, comments = {}, {} for pr in prs: n = str pr "number" reviews n = gh json "api", f"repos/{repo}/pulls/{n}/reviews", "--paginate" comments n = gh json "api", f"repos/{repo}/pulls/{n}/comments", "--paginate" return prs, reviews, comments def score plan, prs, reviews, comments : by branch = {p "headRefName" : p for p in prs} rows = for case in plan "bugs" + plan "controls" : pr = by branch.get case "branch" if not pr: rows.append {"id": case "id" , "status": "NO PR"} continue n = str pr "number" revs = r for r in reviews.get n, if is copilot r.get "user" inline = c for c in comments.get n, if is copilot c.get "user" hit = noise = 0 for c in inline: line = c.get "line" or c.get "original line" or 0 planted = "file" in case and c.get "path" == case "file" and \ case "line start" - SLACK <= line <= case "line end" + SLACK hit += planted noise += not planted first = min ts r "submitted at" for r in revs if r.get "submitted at" , default=None mins = round first - ts pr "createdAt" .total seconds / 60, 1 if first else None rows.append {"id": case "id" , "pr": pr "number" , "kind": "bug" if "file" in case else "control", "reviewed": bool revs , "caught": hit 0, "noise": noise, "minutes": mins} return rows def main : ap = argparse.ArgumentParser ap.add argument "repo" ap.add argument "--plan", default="planted.json" ap.add argument "--fixture" ap.add argument "--min-catch", type=float, default=0.0 args = ap.parse args plan = json.load open args.plan rows = score plan, load args.repo, args.fixture print f"{'case':<16}{'PR': 4} {'reviewed':<9}{'caught':<8}{'noise': 5}{'min': 7}" for r in rows: if r.get "status" == "NO PR": print f"{r 'id' :<16} missing PR" continue print f"{r 'id' :<16}{r 'pr' : 4} {str r 'reviewed' :<9}{str r 'caught' :<8}" f"{r 'noise' : 5}{ r 'minutes' if r 'minutes' is not None else '-' : 7}" bugs = r for r in rows if r.get "kind" == "bug" rate = sum r "caught" for r in bugs / max len bugs , 1 noise = sum r "noise" for r in rows if r.get "kind" print f"\ncatch rate: {rate:.0%} total noise comments: {noise}" sys.exit 0 if rate = args.min catch else 1 if name == " main ": main Three design choices deserve a note. The login match uses a substring because the reviewer’s bot login has varied in presentation, so a fixed string would silently report zero catches. The line check falls back from line to original line , since comments on code that later changed lose their current line. And the output exits non-zero below a catch-rate threshold, which Step 12 uses in CI. Reading the output A fixture with synthetic review data lets you check the script before trusting it on live pull requests. Save the file below as fixtures/sample.json . It describes five pull requests, with Copilot comments that hit three bugs and miss one. { "prs": {"number": 1, "headRefName": "bug/sqli", "createdAt": "2026-11-08T10:00:00Z"}, {"number": 2, "headRefName": "bug/traversal", "createdAt": "2026-11-08T10:01:00Z"}, {"number": 3, "headRefName": "bug/timing", "createdAt": "2026-11-08T10:02:00Z"}, {"number": 4, "headRefName": "bug/float-money", "createdAt": "2026-11-08T10:03:00Z"}, {"number": 5, "headRefName": "control/docstring-only", "createdAt": "2026-11-08T10:04:00Z"} , "reviews": { "1": {"user": {"login": "copilot-pull-request-reviewer bot "}, "submitted at": "2026-11-08T10:03:30Z"} , "2": {"user": {"login": "copilot-pull-request-reviewer bot "}, "submitted at": "2026-11-08T10:05:00Z"} , "3": {"user": {"login": "copilot-pull-request-reviewer bot "}, "submitted at": "2026-11-08T10:06:12Z"} , "4": {"user": {"login": "copilot-pull-request-reviewer bot "}, "submitted at": "2026-11-08T10:07:00Z"} , "5": {"user": {"login": "copilot-pull-request-reviewer bot "}, "submitted at": "2026-11-08T10:07:45Z"} }, "comments": { "1": {"user": {"login": "copilot-pull-request-reviewer bot "}, "path": "app/db.py", "line": 7} , "2": {"user": {"login": "copilot-pull-request-reviewer bot "}, "path": "app/files.py", "line": 7} , "3": {"user": {"login": "copilot-pull-request-reviewer bot "}, "path": "app/auth.py", "line": 1} , "4": {"user": {"login": "copilot-pull-request-reviewer bot "}, "path": "app/billing.py", "line": 6}, {"user": {"login": "someone"}, "path": "app/billing.py", "line": 6} , "5": } } Run the script against the fixture with a 0.75 threshold. python3 scripts/scorecard.py acme/review-lab --fixture fixtures/sample.json --min-catch 0.75 case PR reviewed caught noise min sqli 1 True True 0 3.5 traversal 2 True True 0 4.0 timing 3 True False 1 4.2 float-money 4 True True 0 4.0 docstring-only 5 True False 0 3.8 catch rate: 75% total noise comments: 1 The fixture is synthetic, so those timings are not measurements of Copilot. They show the format. Against live pull requests, drop the fixture flag and pass your own owner and repository name. In the fixture, the timing case was reviewed but the comment landed on line 1 instead of line 10, which the script correctly scores as a miss plus one noise comment. What a miss tells you A miss has three possible causes, and each needs a different fix. The reviewer may have seen the bug and commented on the wrong line, which a manual look at the comment reveals. It may have ignored the bug because your instructions told it to skip that category. Or it may have missed the bug outright, which is the finding you ran the experiment to learn. Read every miss by hand before you change anything. For a broader harness that compares assistants across editors rather than reviewers on pull requests, the Cursor versus Copilot benchmark https://shattered.io/benchmark-cursor-vs-github-copilot-2026/ reuses the same idea of fixed tasks and a scored result. Step 12: Gate the Score and Tune the Instructions A one-off score tells you where you stand today. A repeatable score tells you whether the next instruction change helped. Add a manually triggered workflow that runs the scorecard on GitHub’s infrastructure and fails when the catch rate drops under the threshold. name: Review score on: workflow dispatch: jobs: score: runs-on: ubuntu-latest permissions: contents: read pull-requests: read steps: - uses: actions/checkout@v4 - name: Score Copilot reviews env: GH TOKEN: ${{ github.token }} run: python3 scripts/scorecard.py "$GITHUB REPOSITORY" --min-catch 0.75 git add .github/workflows/review-score.yml scripts/scorecard.py fixtures git commit -m "ci: scorecard, fixture and score workflow" git push gh workflow run review-score.yml The workflow needs only read access to pull requests, and the built-in token supplies it. It uses workflow dispatch on purpose. Reviews finish at unpredictable times, so a scheduled or push-triggered score would often read half-finished results. Now tune. Change one instruction per round, push it, reopen the five pull requests, and rerun the scorecard. Three rules keep the loop honest: 1. One change per round. Two edits at once make a score change impossible to attribute. 2. Three runs before you believe a result. Model-written reviews vary between runs, so a single pass can flatter or punish an instruction. 3. Use the source trail. Since November 6, review comments can expose the skills, MCP servers and custom instructions that informed them. When a comment cites a file you did not expect, you have found an instruction collision. Then protect the instructions themselves. Since July 17, review reads instructions from the pull request’s head branch. That helps you test a rewrite before merging, but it also means a pull request author can edit the rules that judge their own change. Require code owner approval on the instruction paths. .github/CODEOWNERS /.github/copilot-instructions.md @your-org/platform-leads /.github/instructions/ @your-org/platform-leads /.github/workflows/ @your-org/platform-leads Save that as .github/CODEOWNERS , commit it, and pair it with a branch rule that requires code owner review. The same people who own the instructions should own the review workflow, since both shape what the reviewer sees. If tuning stalls and the misses persist across three runs, put a rival through the same five pull requests. The AI code review tools comparison https://shattered.io/ai-code-review-tools-bugbot-vs-codex-2026/ names the main contenders. Common Pitfalls That Skew a Review Pilot - Pasting a linter config into the instructions. Formatting and import-order rules produce comment volume without catching bugs. Keep those in ruff or your linter and list them under skip. - Letting authors edit their own judge. Head-branch instruction loading is useful and exploitable. A pull request can quietly add a skip rule for the exact pattern it introduces. Lock the paths with code owners. - Reviewing every push by default. New-push review on a busy repository multiplies credit spend, and it runs even on draft pull requests when you enable that option. Start with it on for small repos and measure before widening. - Scoring a single run. One pass at medium effort tells you little. Run each configuration three times and compare the spread, not the best result. - Planting bugs that are too obvious. Textbook SQL injection is the easy case. Add one bug that needs project context, such as a business rule from your own domain, before you draw conclusions. - Ignoring bot-authored pull requests. Since November 6, Actions, Dependabot and org apps can request reviews on bot pull requests when external-license blocking is on and your organization pays. Without a plan, a dependency bot becomes your largest review customer. - Applying suggestions blindly. One-click fixes are convenient, and they still need your test suite. Run CI on the suggestion commit before you merge. Troubleshooting: 12 Failures and Fixes 1. No Copilot review appears. Check that your plan is Pro, Pro+, Max, Business or Enterprise, that the ruleset is Active, that the pull request targets main, and that it is not a draft with draft review disabled. 2. Ruleset creation returns 422. The rule type or parameters differ from your release. Create the ruleset in the web UI, read it back with gh api repos/{owner}/{repo}/rulesets/ID , and copy the exact schema. 3. Instructions seem ignored. Verify the applyTo glob matches the changed files, the frontmatter fences are three dashes on their own lines, and the files exist on the pull request head branch. Then check the comment’s source trail. 4. A bypassed author still gets reviewed. The bypass stops automatic requests only. Someone may have requested the review by hand, or another ruleset at organization level may apply. 5. The review sits pending forever. A self-hosted or larger runner configured for code review may be offline or lack a matching label. Check the organization’s runner default. 6. The setup workflow never runs. The file name must be exactly .github/workflows/copilot-code-review.yml , Actions must be enabled for the repository, and the YAML must parse. Validate it by running gh workflow run on the file. 7. The scorecard prints missing PR. Branch names in planted.json must match the pull request head branches exactly. The script lists up to 100 pull requests in any state, so raise the limit if your repo is busy. 8. Reviewed is True but caught is False. Open the comment and read its line anchor. If it sits on a nearby line, widen the tolerance constant. If the comment is on the wrong file, the miss is real. 9. gh api returns 403 or 404. Re-run gh auth login with the repo scope. In Actions, grant pull-requests: read . Fork pull requests get a read-only token and may hide review data. 10. Reviews stop mid-month. You hit your credit allowance. Set a dollar budget so Copilot continues with paid usage, and watch the 75% alert. 11. Bot pull requests get no review. Reviews on bot-authored pull requests depend on the external-license blocking setting and on the organization paying. Check both before filing a bug. 12. The seed script says the tree is dirty. Commit or stash everything first. The guard exists so patches land on clean branches, and a crashed earlier run may have left .seed-cases.tmp , which the lab .gitignore already covers. Advanced Tips: Scaling Code Review Tools Past the Lab Script the rollout with the settings API The November 6 release added REST and GraphQL management for review settings at enterprise, organization and repository level. That makes a policy-as-code setup realistic: a nightly job reads each repository’s review configuration, compares it with a baseline file, and opens an issue when a repo drifts. Read the current REST reference for exact paths and fields before you write the job, because the endpoints are new. Combine that with the ruleset API from Step 6. A single script can create the ruleset, apply the same bypass list and confirm enforcement across every repository in an organization, which beats clicking through settings pages fifty times. Budget alerts belong in the same automation. Because every review draws credits, a team of twenty with new-push review enabled can burn through included usage faster than expected. Put the 75% alert in a channel someone reads. A staged policy works well in practice. Use medium effort everywhere, Max only on release branches or security-sensitive paths, and no automatic review for bot authors until you have seen a month of usage. Those are judgment calls rather than GitHub guidance, so confirm them with your own scorecard. Reviews also reach beyond the web UI. GitHub’s @GitHub integration in Slack and Microsoft Teams became generally available on November 6, according to the Microsoft Partner Center announcements https://learn.microsoft.com/en-us/partner-center/announcements/2026-november , so teams can turn a chat thread into code-related work without opening the repo first. Agent workflows are the next layer. GitHub says its Copilot automations can run on events or schedules, including code review and issue triage, and a workflow powered by the default Copilot CLI engine requires a Copilot license. Reports also say Agentic Workflows reached general availability on October 28, but the source for that is single, so verify it in GitHub’s docs before planning around it. One security note for public repositories. GitHub reportedly plans to block pull request target by default in public repositories on December 2, 2026. That single-source report is worth checking, because any custom review or scoring workflow that relies on that trigger would break. Finally, know where a native reviewer stops. It sees pull requests on GitHub and follows your instructions, but it does not replace a security scanner, a type checker or a human who knows the product. Dedicated vendors typically pitch context beyond a single pull request and support for platforms other than GitHub. The software and developer tools hub https://shattered.io/category/software/ collects coverage of those rivals. Whatever you pick, reuse the lab: swap the reviewer, keep the five pull requests and the scorecard, and compare numbers on your own code. The Finished Project: Layout and Final Checklist When every step is done, the repository contains the files below. Compare your tree against it. review-lab/ .github/ CODEOWNERS copilot-instructions.md instructions/ python.instructions.md tests.instructions.md workflows/ copilot-code-review.yml review-score.yml app/ init .py auth.py billing.py db.py files.py fixtures/sample.json planted.json ruleset.json scripts/ patches/ sqli, traversal, timing, float-money, docstring-only scorecard.py seed prs.sh tests/test billing.py .gitignore - Main branch holds clean code, instructions, workflows, patches and scripts, with nothing uncommitted. - The ruleset copilot-auto-review is Active on the default branch, with review on new pushes enabled. - Five lab pull requests exist, each from its own branch, and Copilot appears under Reviewers on each. - The scorecard prints a catch rate, a noise count and minutes to first review for all five cases. - A credit budget with alerts is configured, and CODEOWNERS covers the instruction paths. - You have saved at least two scorecard outputs, one per effort level, with three runs behind each. The takeaway: Copilot code review is easiest to judge when you treat it like any other code review tool, with fixed inputs, a scored output and a cost line. After the November 2026 changes, the configuration surface is wide enough that guessing is a choice. Run the lab, keep the numbers, and let your own repository decide how much effort the reviewer deserves. Frequently Asked Questions Which plans include Copilot code review? GitHub lists it on Copilot Pro, Pro+, Max, Business and Enterprise. Copilot Free keeps limited completions, chat and agent usage and is not on the code review plan list. Does Copilot code review cost extra? Reviews consume GitHub AI Credits, where one credit equals $0.01, and they also use GitHub Actions compute at the applicable rates. Included credits depend on your plan, and a dollar budget lets usage continue past the allowance. What is Copilot code review Max? GitHub introduced Max on October 28, 2026, in public preview, as its deepest review mode. GitHub describes it as the highest-F1 option on its ReviewBench evaluation for high-severity issues, which is a vendor benchmark and not an independent test. Where do custom instructions live? Repository-wide guidance goes in .github/copilot-instructions.md . Path-specific files go in .github/instructions/ with the .instructions.md suffix and an applyTo pattern. Review reads them from the pull request head branch. Can I stop automatic reviews for some authors? Yes. Since November 6, 2026, the automatic-review rule supports a bypass list for exempt actors. Manual review requests still work on those authors’ pull requests. Can I manage review settings through an API? GitHub added REST and GraphQL management for Copilot code review settings on November 6, 2026, from enterprise policy down to individual repositories. Check the current REST reference for exact endpoints. How do I know whether the reviewer is good enough? Plant bugs you understand, add a harmless control change, run each configuration three times, and score catches against noise. A catch rate on your own code beats any published benchmark. Should I use a different reviewer instead? That depends on needs the lab can measure. If you need cross-repository context or non-GitHub platforms, compare dedicated code review tools with the same five pull requests. If you want zero extra vendors, Copilot’s native review is the lower-friction place to start.