Reading JetBrains' Refactoring-Skill Benchmark Honestly

JetBrains' refactoring-skill benchmark, published August 19, 2026, reports that its AI skill for Rider and ReSharper is 83% faster, 64% cheaper, and uses 63% fewer tool calls, based on median results across fifteen C# refactoring tasks run roughly ten times each. The benchmark used gpt-5.5 through the Codex CLI for both arms; Claude Code was not benchmarked, and the speed-up requires the Rider MCP server running. JetBrains disclosed losses and applied a paired permutation test, but no p-values or third-party reproduction are provided.

JetBrains’ refactoring-skill benchmark is the rare vendor-run benchmark that discloses most of what you would want to know: the task count, the model, the harness, the runs per task, the baseline definition, the cost unit, and one result that went against the skill. It still describes exactly one harness and one model, and the skill it measures only reaches its headline speed with Rider’s own MCP server running. Both facts are in JetBrains’ own text. Neither is in the three-number stat callout that carries them, which repeats verbatim on both JetBrains posts and is the part most likely to travel on its own. The three numbers travelling around are JetBrains’ own labels: “83% faster” median task time 157.9 s to 26.6 s , “64% cheaper” USD 0.52 to 0.19 per solved task and “63% fewer” tool calls 17.0 to 6.2 per task , medians across fifteen C refactoring tasks run roughly ten times each. The sentence that should sit next to them, verbatim from the benchmark post, is that both arms ran gpt-5.5 through the Codex CLI. Claude Code was not benchmarked. The skill installs into Claude Code and Codex alike, so “works with both” is true at the install level while the measured claim covers one harness and one model. This post is a read, not a rebuttal. We do not think the benchmark is wrong or inflated; JetBrains published losses alongside wins and applied a real statistical test. What we do is walk the numbers back to their disclosure, recompute the derived figures from the published inputs, record what the posts are silent on, and extend the disclosure rubric from our vendor benchmark reproducibility audit /blog/vendor-benchmark-reproducibility-audit-2026 to this specific case. The closing section is about something larger: an IDE vendor shipping skills for third-party agents, and what that makes the skill layer. - 01The headline numbers are JetBrains’ own, verbatim.Median task time 157.9 s to 26.6 s, which JetBrains itself labels “83% faster”; cost per solved task USD 0.52 to 0.19 “64% cheaper” ; tool calls 17.0 to 6.2 per task “63% fewer” . Medians across fifteen C refactoring tasks, roughly ten runs each. The percentages are citable as JetBrains’ phrasing, not our derivation. - 02One harness, one model: Codex CLI running gpt-5.5.JetBrains states that both arms ran gpt-5.5 through the Codex CLI. Claude Code was not benchmarked and no other model was. The skill installs into both Claude Code and Codex via each agent’s plugin marketplace, so the install claim and the measured claim have different scopes. - 03The skill is a pointer, not a self-contained file.JetBrains’ own README says the skills drive Rider’s refactoring engine through the Rider MCP server and degrade gracefully to grep and file edits when that server is not connected. The measured speed-up is a property of the skill plus a running Rider MCP server, not of the SKILL.md on its own. - 04Significance is a label, not a number.JetBrains applied a paired permutation test and reports both improvements as significant. No p-value, confidence interval or effect size is published. No third party has reproduced the benchmark; every figure traces to JetBrains’ two August 19 posts. - 05JetBrains disclosed the losses, and the selection rule that favours the skill.Eight of fifteen tasks cleared the full-win bar; one task was solved without the skill but not with it. The eight operations were chosen for the cleanest pass/fail contracts, which are also the operations Rider’s engine exposes as tools. JetBrains says this itself; it is a scope note, not a concealment. 01 — What ShippedWhat JetBrains shipped, and what it said about it. Two JetBrains posts carry August 19, 2026 bylines. The release post, “Rider 2026.2.1 and ReSharper 2026.2.1 Are Here ” https://blog.jetbrains.com/dotnet/2026/08/19/rider-resharper-2026-2-1/ , lists what the minor release contains and says the build is ready to download. The companion benchmark post, “Rider Hands AI Agents The Keys To Its Refactoring Engine For Safer, Faster, And Cheaper Results” https://blog.jetbrains.com/dotnet/2026/08/19/rider-refactoring-code-skill/ by Sasha Ivanova, is where the numbers and the methodology live. Neither page uses a status word such as beta or GA for the refactoring-code skill; it simply ships bundled with the IDE, with nothing to switch on. We follow JetBrains’ own dating throughout and say “announced August 19” rather than asserting a separate release day. The refactoring-code skill covers eight C operations by name: rename, extract method, extract interface, extract base class, change API signature, move type to namespace, reorganize namespaces and safe delete. The benchmark wrapped those eight operations in fifteen tasks, most in two variants, a straightforward case and a harder one with more call sites or more entangled dependencies. The three headline figures below are JetBrains’ stat callout, reproduced with its own labels. 157.9 s → 26.6 s JetBrains’ own label for the median across all fifteen tasks, roughly ten runs each. The 95th percentile also fell, from 346.4 s to 56.9 s, because the slowest runs were the ones trapped in the edit-build-read-error cycle. USD 0.52 → USD 0.19 US dollars implied by gpt-5.5 token pricing on the recorded input, cache-read and output tokens. JetBrains also reports a separate median cost per task across all tasks, USD 0.33 to 0.12. Different denominators; the post keeps them apart and so do we. 17.0 → 6.2 per task Across the whole evaluation, 2,513 tool calls fell to 926. The dotnet build count fell from 163 calls to 3. The eight refactoring operations account for 167 of the 926 remaining calls; most of the rest are ordinary text edits. The release post carries four other items that belong in the record at one line each. Everything below is verified against JetBrains’ release notes for Rider and ReSharper; we do not expand on any of them beyond what the post says. debugging-code skill Lets agents set breakpoints, step through code, inspect values and thread context, follow the call order, and check which branches are taken or whether execution reaches a specific line. Quality-check hooks support Codex This is the only sentence on either JetBrains page that names Claude Code explicitly, and it is about hooks, not the refactoring skill. Do not source the skill’s dual-agent support to it; source that to the repository. ReSharper Out-of-Process by default Possible because dotCover now supports OOP mode. Type and project dependency diagrams now also work out of process, where previously they required switching back to in-process. Junie adds Claude Opus 5 JetBrains updated Junie to a version that adds support for Claude Opus 5, so you can pick it as the model powering your sessions. That is the full extent of the release note and the full extent of what we will say about it. 02 — The ReadEach claim, what it establishes , and what it does not. The table is the post. Each row takes one claim in circulation, records what JetBrains’ own text discloses for it, records what the text is silent on, and says how we would read it. The last row is the exception: it is not a claim anyone has made, it is the question to settle before repeating any of the others. The “how to read” column is our interpretation; everything in the middle two columns is traceable to the two JetBrains posts or the public skill repository. Where we say “not disclosed” we mean the posts do not say, not that JetBrains is hiding something. | Claim or question | What JetBrains discloses | What it does not | How to read it | |---|---|---|---| | The three headline numbers | ||| | “83% faster” | Median task time 157.9 s to 26.6 s across all fifteen tasks, roughly ten runs each; 95th percentile 346.4 s to 56.9 s. The aggregate explicitly covers all fifteen tasks, not only the eight full wins. | The full distribution: no histogram, interquartile range, minimum, or the individual run times. Hardware and execution environment: not disclosed. | A percentage of time removed, not a multiplier. From the published medians the multiplier is roughly 5.9×, our arithmetic, not JetBrains’ framing. The one extra percentile moved the same direction as the median, which argues against a hidden fat tail. | | “64% cheaper” | Cost per solved task USD 0.52 to 0.19. A second figure, median cost per task across all tasks, USD 0.33 to 0.12. Underlying token counts per task: input 436,745 to 208,524; cache reads 2,973,158 to 1,257,600; output 32,532 to 15,538. | Cost on any other model or harness. Hardware. The two cost figures have different denominators and the post keeps them separate; merging them is a reader error, not a vendor one. | Dollars implied by one model’s token prices on one harness. The figure moves whenever OpenAI’s gpt-5.5 pricing moves and says nothing about cost on a different agent. | | “63% fewer tool calls” | 17.0 to 6.2 per task; 2,513 to 926 across the evaluation; dotnet build 163 calls to 3. The eight refactoring operations account for 167 of the 926 skill-arm calls. | Any quality signal beyond the pass/fail test bar both arms were held to. Fewer calls is not evidence of better code on its own. | A cost and latency proxy, not a quality measure. The mechanism JetBrains describes is the removal of the build-as-oracle loop; the quality bar is the tests, and it is the same bar for both arms. | | Method and scope | ||| | “Statistically significant” | A paired permutation test on timing, cost and tool counts from the recorded traces; JetBrains states both improvements are significant under it. | No p-value, no confidence interval, no effect size. The test’s output is not published, only its verdict. | A pass/fail label from a real test. More than most vendor benchmarks disclose; still not a number you can carry into your own analysis. Do not invent one. | | “Works with Claude Code and Codex” | The benchmark post: both arms ran gpt-5.5 through the Codex CLI, roughly ten times per task. The repository: install paths for Claude Code and Codex, plus a manual copy for any agent that reads SKILL.md. | Any measurement on Claude Code, on any other harness, or on any model other than gpt-5.5. | True at the install level, unmeasured at the benchmark level. Keep the two claims apart: “installs into both” is verified; “measured on both” is false. | | “A portable skill” | The README: the skills drive Rider’s refactoring engine through the Rider MCP server, and degrade gracefully to standard file and grep tools when that server is not connected. | Any numbers for the degraded path. The benchmark arms are skill-available versus skill-unavailable with Rider running, not skill-with-server versus skill-without-server. | The SKILL.md travels; the capability does not. The measured speed-up depends on a running Rider MCP server, which only JetBrains ships. | | Reproduced by anyone else? | A named statistical test, an all-task aggregate, one task that regressed, and seven tasks outside the full-win bar, all stated by JetBrains itself. | The task corpus, the harness or scoring code, and any third-party rerun. None is published. | Nobody outside JetBrains has reproduced this. Every number in this post traces to JetBrains’ two August 19 posts. Label it vendor-run and treat it as an invitation to measure, not a result to quote as settled. | solved without the skill but not with it , two were solved by neither arm, and four were already quick enough that invoking the IDE did not pay for itself. The headline medians cover all fifteen, not just the eight full wins. That is better disclosure than our CursorBench read /blog/cursorbench-v3-1-vendor-benchmark-analysis found on the wins-and-losses axis, where the vendor only partially cleared it. 03 — Harness and ModelOne harness, one model: Codex CLI on gpt-5.5 . The fact the stat callout leaves out is stated in plain language further down the benchmark post. The methodology passage names the harness and the model once, and it names only one of each. “Both arms ran gpt-5.5 through the Codex CLI, roughly ten times per task, and the only difference between them was whether refactoring-code was available.”— Sasha Ivanova, JetBrains .NET blog, refactoring-code skill benchmark post, August 19, 2026 So the baseline is not a different agent, a different IDE or a competitor’s skill. It is the same Codex CLI, the same gpt-5.5, the same prompts, with the refactoring-code skill present or absent. That is a clean design for the question JetBrains asked, which was whether giving this agent access to Rider’s engine helps. It is not a design that can answer whether the skill helps Claude Code, or whether it helps a different model on Codex, because neither was run. The install story is separate and independently verifiable. The JetBrains/rider-skills repository https://github.com/JetBrains/rider-skills Apache 2.0 carries a .claude-plugin/ directory with Claude Code plugin and marketplace metadata and a .codex-plugin/ directory with Codex plugin metadata, both pointing at the same skills/<skill /SKILL.md content underneath. Dual-agent packaging is real at the metadata level. Dual-agent measurement does not exist. Codex CLI running gpt-5.5 The only harness and the only model in the benchmark. Every one of the three headline figures, the 95th percentile, the token counts and the per-task table describe this configuration and no other. Claude Code Then /plugin install rider-skills@rider-skills and /reload-plugins. Rider registers itself with the agent from Settings → Tools → MCP Server → Auto-Configure. Not benchmarked; the only Claude Code mention on either JetBrains page concerns the separate quality-check hooks. Codex Then install Rider Skills from the /plugins browser inside Codex. This is the harness JetBrains measured, so the benchmark numbers are most directly relevant here, on gpt-5.5 specifically. Any agent that reads SKILL.md The README’s manual path drops a skill folder into ~/.claude/skills/ or <project /.claude/skills/. The requirement line still reads: JetBrains Rider with its MCP server enabled and the target solution open. 04 — What the Skill IsA thin instruction layer over a running Rider MCP server . The word “skill” suggests a self-contained file you can drop into any agent and get the measured behaviour. That is not what this is, and JetBrains’ README is explicit about it. The skills are described as giving coding agents IDE-grade code intelligence inside Rider by driving Rider’s refactoring engine, debugger, code coverage data and static analysis through the Rider MCP server. The skill is the pointer; the engine is the capability; the MCP server is the wire between them. The README’s “How it works” section describes three steps at invocation time. First, the skill checks for the Rider MCP tools in the session’s deferred-tool list and loads their live schemas, which it treats as authoritative for parameter names. Second, it uses Rider’s semantic engine, the ReSharper or IDE index, for anything grep cannot resolve. Third, it degrades gracefully: when the Rider MCP server is not connected, it states the blocker, falls back to standard file and grep tools, and documents which IDE-backed quality steps were skipped. plus a running Rider MCP server , with the target solution open. Without that server the skill falls back to the same grep-and-edit loop the baseline arm used. JetBrains publishes no numbers for the degraded path, and the benchmark never tested it; the two arms were skill-available and skill-unavailable, both with Rider running. A team that installs the SKILL.md and skips the Rider-side setup has installed a pointer to nothing. The worked example in the benchmark post makes the mechanism concrete. For extract base class, the task JetBrains itself flags as the slowest without the skill and the largest single improvement with it, the baseline arm spent 336.7 s and 24 tool calls at USD 1.15, including nine further edit-and-build cycles hunting a compile error. The skill arm took 19.8 s, 3 calls and USD 0.09: one rg search, one execute tool::extract base class call with target, name and members as parameters, then git diff --stat . That middle call is the Rider MCP server doing the work. JetBrains presents this as the best case, not the typical one, and so do we. This matters for the portability argument we made in our Codex cross-harness skill post /blog/codex-cli-cross-harness-skill-portability-lock-in , which found that Claude Code, Codex and Cursor all read the same open SKILL.md standard. The format is portable. The JetBrains case shows that a vendor-shipped skill can still be non-portable in practice if what it calls is proprietary and requires that vendor’s own software running. We return to that in Section 08. 05 — DistributionWhat a median hides , and what JetBrains showed anyway. A median is one point on a distribution. “83% faster” at the median is compatible with a skill that helps every task a little and one that helps some tasks enormously and others not at all. JetBrains published two things that narrow the range: a 95th-percentile figure and a per-task breakdown of wins and losses. Both are to its credit, and both are still short of the full distribution. The bars below put every published before-and-after pair on the same scale, the skill arm as a percentage of its own baseline. Percentages are our arithmetic from JetBrains’ printed figures; the baseline row is 100 by construction. Skill arm as a share of the baseline arm · JetBrains-stated inputs Source: JetBrains benchmark post, August 19, 2026 · percentages computed by Digital Applied from the published figuresTwo things stand out. The 95th percentile fell by a slightly larger share than the median, 83.6% against 83.2% by our arithmetic, which JetBrains explains as the slowest runs being the ones trapped in the edit-build-read-error cycle; with the skill, those runs stop existing. A median that hides a fat tail would show the opposite pattern. The one extra percentile JetBrains chose to publish therefore runs in the skill’s favour. That is the honest reading, and it cuts against the reflexive “medians hide things” objection in this case. What is still missing is everything between those two points: no interquartile range, no minimum, no histogram, and none of the individual run times behind either arm. The per-task breakdown partly compensates, and it is where the regression lives. Faster, cheaper, no more tool calls, both arms passing JetBrains’ own bar for a full win requires all four conditions. Eight tasks clear it. The headline aggregates are not drawn from these eight alone; the post states they cover all fifteen. Solved without the skill, not with it The result that does not fit the headline, published by JetBrains itself. It is the strongest evidence that the write-up reports losses alongside wins, and it rules out any “wins uniformly” paraphrase. Invoking the IDE did not pay for itself On four tasks the baseline was fast enough that calling Rider’s engine added more than it saved. A useful scope note: the skill earns its keep on the entangled cases, not the trivial ones. Solved by neither arm Two tasks defeated both configurations. Together with the one regression and four already-quick tasks, that accounts for the seven tasks outside the full-win bar: 8 + 1 + 4 + 2 = 15. 06 — Cost ProxyTool-call count is a cost proxy , not a quality score. “63% fewer tool calls” is the headline most likely to be misread as a quality claim. It is not one. Tool calls cost tokens and wall time, so fewer of them is a cost and latency result; whether the resulting code is better is answered by the pass/fail tests, and both arms were held to the same tests. JetBrains’ own breakdown of where the baseline arm spent its calls makes this clear, and it is worth seeing in full. Where the baseline arm spent its tool calls · JetBrains-stated Source: JetBrains benchmark post, August 19, 2026 · bar length relative to the largest categoryJetBrains’ explanation for the build-heavy baseline is vendor narrative, and we attribute it as such: in its account, the agent was not compiling to check finished work, it was compiling to find out what its last edit had done, because the information a rename or extraction needs is not recoverable from a regular expression, so the agent guesses in text and lets the build score the guess. That is a plausible mechanism and it matches the tool-call composition. It is also exactly the story a refactoring-engine vendor would tell, so take it as JetBrains’ reading rather than a neutral finding. Even with the skill, most calls are still ordinary edits. The eight refactoring operations account for 167 of the 926 skill-arm calls, about 18% by our arithmetic. The skill is invoked for a minority of calls; its effect is to remove the build-as-oracle loop that dominated the baseline. The table below recomputes every derived figure from JetBrains’ printed inputs, so you can see where the labels come from. | Metric | Without skill | With skill | Change, recomputed | JetBrains’ label | |---|---|---|---|---| | Per task · medians across 15 tasks, ~10 runs each | |||| | Median task time | 157.9 s | 26.6 s | −83.2% ≈5.9× | “83% faster” | | 95th-percentile task time | 346.4 s | 56.9 s | −83.6% ≈6.1× | “fell further”, no percentage printed | | Cost per solved task | USD 0.52 | USD 0.19 | −63.5% | “64% cheaper” | | Median cost per task, all tasks | USD 0.33 | USD 0.12 | −63.6% | none printed | | Tool calls per task | 17.0 | 6.2 | −63.5% | “63% fewer” | | Input tokens per task | 436,745 | 208,524 | −52.3% | none printed | | Cache-read tokens per task | 2,973,158 | 1,257,600 | −57.7% | none printed | | Output tokens per task | 32,532 | 15,538 | −52.2% | none printed | | Whole evaluation · totals across all runs | |||| | Total tool calls | 2,513 | 926 | −63.2% | none printed | | dotnet build calls | 163 | 3 | −98.2% | none printed | | Worked example · extract base class, flagged by JetBrains as its largest single improvement | |||| | Task time | 336.7 s | 19.8 s | −94.1% | none printed | | Tool calls | 24 | 3 | −87.5% | none printed | | Cost | USD 1.15 | USD 0.09 | −92.2% | none printed | One small reading note falls out of the recompute. From the printed inputs, cost per solved task and tool calls per task both land at 63.5%, yet JetBrains labels one “64% cheaper” and the other “63% fewer”. The dollar figures are printed to the cent and the tool counts to one decimal, so the unrounded inputs can legitimately round either way. Neither label is wrong. It is simply a reminder that the headline percentages are rounded summaries of rounded inputs, which is the normal state of a vendor stat callout and a reason to carry the underlying figures rather than the labels into your own comparison. 07 — Task SelectionThe skill looks fastest on the tasks it targets , and JetBrains says so. Rider has dozens of C refactorings. JetBrains evaluated eight and gives its reason: they were chosen because they have the cleanest contracts, a defined target, a defined result, and a refusal when the change is unsafe. That selection rule is also, necessarily, a description of the operations Rider’s engine exposes as callable tools. A skill that front-loads the IDE’s own refactoring engine will look strongest on tasks built around the operations that engine implements. JetBrains does not conceal this; it states the rule. The right reading is that the benchmark measures the skill on its home ground, and says nothing about refactorings outside those eight or about code changes that are not refactorings at all. Put against the rubric from our 42-row vendor benchmark audit /blog/vendor-benchmark-reproducibility-audit-2026 , where no row was both fully disclosed and independently confirmed disclosure checked on all 42, independent locatability attempted on 15 of them , JetBrains names its harness, its model, its runs per task and its statistical test, and still fails the reproducibility bars that matter most: no published corpus, no released harness, no third-party rerun. We have not scored JetBrains against those 42 rows, so read this as the same rubric applied to a new case rather than a ranking inside that dataset. The table records each axis, with the status taken from the two posts and the repository, and “not disclosed” wherever the text is silent. | Axis | Status | What the posts say | |---|---|---| | Disclosed | || | Task count | Disclosed | Fifteen tasks covering eight operations, most in two variants: a straightforward case and a harder one with more call sites or more entangled dependencies. | | Model | Disclosed | gpt-5.5, in both arms. No second model. | | Harness | Disclosed | The Codex CLI, in both arms. Claude Code not run. | | Runs per task | Disclosed | Both arms, “roughly ten times per task”. | | Baseline definition | Disclosed | Same agent, same model, same prompts; the only difference is whether the agent could call Rider’s refactoring engine. | | Cost unit | Disclosed | USD per solved task for the headline; a separate median USD per task across all tasks; the underlying input, cache-read and output token counts. | | Task selection rule | Disclosed | Eight operations chosen for the cleanest contracts: a defined target, a defined result, a refusal when unsafe. | | Losses and exclusions | Disclosed | One task regressed with the skill; two solved by neither; four already quick. Aggregates cover all fifteen. | | Partially disclosed | || | Statistical treatment | Partial | Paired permutation test named; “significant” stated. No p-value, confidence interval or effect size published. | | Distribution | Partial | Median and 95th percentile for time. No interquartile range, minimum, histogram or individual run times. | | Not disclosed | || | Hardware and execution environment | NOT DISCLOSED | No machine specification, no cloud-versus-local statement, nothing beyond “the same model and the same prompts”. | | Test set publicly downloadable | NOT DISCLOSED | Tasks are referenced by name in the per-task table; the underlying task and repository corpus is not published or linked. | | Harness or scoring code | NOT DISCLOSED | “Recorded traces” and a paired permutation test are described, not published as code. | | Independent reproduction | None published | Single vendor-run study. No third party has rerun it. | | Cross-model or cross-harness generalisation | NOT DISCLOSED | One model, one harness, no comparison point. | The contrast is the finding. Against the axes our CursorBench analysis /blog/cursorbench-v3-1-vendor-benchmark-analysis set out, JetBrains names the harness, defines the baseline, names a statistical test and volunteers its one regressed task without being asked. That is more than most vendor benchmarks we have read bother with. It still does not publish the corpus, the code or the test output, and nobody has reproduced it. “More disclosed than most, still not reproducible” is a more useful verdict than either “trust it” or “ignore it”, and it is the verdict the evidence supports. 08 — The Agentic AngleAn IDE vendor shipping skills for other people’s agents . Step back from the numbers and the release is more interesting than the benchmark. JetBrains did not ship an agent. It shipped a directory of SKILL.md files, wrapped in Claude Code and Codex plugin metadata, that teach third-party agents to call JetBrains’ engine over MCP. The agent belongs to Anthropic or OpenAI. The skill format is the open Agent Skills specification https://agentskills.io/specification that both Anthropic https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview and OpenAI https://learn.chatgpt.com/docs/build-skills document: a directory with a SKILL.md file, YAML frontmatter carrying a name and description, discovered from the filesystem rather than uploaded to either vendor’s API. We covered it in our Agent Skills framework guide /blog/claude-agent-skills-framework-guide , and the Agent Plugins 1.0 standard /blog/agent-plugins-1-0-open-standard-portable-ai-skills packages it alongside MCP servers. The capability the skill calls belongs to JetBrains, and it only answers when Rider is running. That makes the skill layer a vendor surface. An IDE vendor that cannot win the agent can still win the tools the agent reaches for, and a benchmark showing the agent works better with those tools is the marketing for that position. Read this way, the selection rule in Section 07 is not a flaw; it is the point. JetBrains measured the operations its engine does well because those are the operations it is offering to every agent. The interesting question for buyers is not whether the numbers are real but what the dependency costs: a team that builds its refactoring workflow on this skill has built it on Rider, on the Rider MCP server, and on JetBrains’ continued willingness to maintain both install paths. Our cross-harness portability post /blog/codex-cli-cross-harness-skill-portability-lock-in made the case that the portable frontmatter surface of a skill is small and countable. JetBrains’ skill confirms it from the other direction. The frontmatter travels between Claude Code and Codex without edits. The MCP server behind execute tool::extract base class does not travel at all. For what it is worth, this is a low-risk dependency by the standards of our skill-pack supply-chain post /blog/agent-skill-packs-package-ecosystem-supply-chain-risk : Apache 2.0, a named publisher, and a README that says what it calls. The risk here is lock-in, not compromise. Trial it on your own tasks, keep the traces This is the configuration JetBrains measured, so the numbers are most directly relevant to you, on gpt-5.5. Run ten of your own refactorings with and without the skill, record time, cost and tool calls the way JetBrains did, and compare against your own baseline rather than the 83% label. Treat the benchmark as a hypothesis The skill installs cleanly via the plugin marketplace, but nothing in JetBrains’ posts measures it on Claude Code or on any model other than gpt-5.5. Expect the mechanism to carry; do not expect the percentages to. Your trial is the first data point. Count the Rider MCP server as part of the skill The measured speed-up requires Rider running with the solution open and the MCP server enabled. A skill that degrades to grep without it is not the skill that was benchmarked. Decide on the IDE dependency first; the SKILL.md is the easy part. Carry the inputs, not the labels 157.9 s to 26.6 s on Codex CLI plus gpt-5.5, fifteen tasks, roughly ten runs, one regression, no reproduction. That sentence survives contact with your own evaluation. “83% faster” on its own does not. Looking forward, expect more of this. If the skill layer is where tool vendors compete for agent attention, the next year will bring more vendor-shipped skills, more vendor-run benchmarks of those skills, and more headline percentages measured on one harness and quietly generalised to all of them. The read table in Section 02 is designed to be reused on each of them. If you want help running the measurement on your own codebase before you commit a workflow to a vendor’s MCP server, that is the kind of comparative evaluation our AI transformation engagements /services/ai-transformation start with, and the kind of agent-assisted engineering our development team /services/web-development runs day to day. 09 — ConclusionA good benchmark with a narrow scope. Real numbers, one harness, one model, one running IDE. JetBrains’ refactoring-code benchmark is better disclosed than most vendor benchmarks we have read: fifteen tasks, roughly ten runs each, a defined baseline, a named statistical test, USD costs with token counts behind them, and a published regression. The three headline labels are JetBrains’ own and they recompute cleanly from the printed inputs. Nothing here suggests the numbers are wrong. What the numbers are is narrow . They describe gpt-5.5 on the Codex CLI, with Rider running and its MCP server connected, on eight operations chosen because Rider’s engine does them well. Claude Code was not measured. No other model was measured. The degraded path without the server was not measured. Significance is a label without a number, and nobody outside JetBrains has reproduced any of it. The larger signal is the shape of the release: an IDE vendor shipping skills for agents it does not own, with a benchmark as the pitch. The skill layer is now a vendor surface, and the portable file is the smallest part of it. Carry the inputs, measure on your own harness, and price the IDE dependency before you price the speed-up.