Announcements How Cline migrated its VS Code extension for 11 million developers to the new Cline SDK, built a safe A/B rollout, and reduced agent failures by 10x.
The Cline extension was first built in 2024, right after Claude 3.5 Sonnet was released. It created one of the very first agentic coding experiences used by millions of developers, but both models and the way they work inside agent harnesses have changed a lot since then. The extension itself had also grown around the IDE, making the underlying harness increasingly difficult to evolve.
Earlier this year, we rebuilt Cline’s foundation around the Cline SDK, pulling the agent harness out of the extension into a shared, modular runtime and improving the harness along the way. We’ve seen firsthand how much the harness around a model can affect the way it works, especially for open weight models.
We couldn’t wait to bring this new harness to our flagship VS Code extension, installed by over 11 million developers. But a migration like this is not an easy job.
Our first attempt took months of work, and when we finally shipped it, things broke badly enough that we had to roll it back immediately. That forced us to rethink not just the migration itself, but how you safely ship a change this large to millions of developers without disrupting the workflows they rely on every day. And, the VS Code Marketplace gave us no way to gradually roll out a release like this.
What followed was the biggest refactor we've made to the extension since its inception, along with a rollout process we had to build from scratch to get it into production safely.
This is how we did it.
Why we migrated to the Cline SDK at all #
Making this migration was a big investment for us as a business, and a risk of causing regressions and pain for our users if not done correctly. In this fast-moving AI coding space, taking the time to slow down and do this right was not a decision we took lightly.
Cline started as a VS Code extension. It then became a JetBrains plugin, and then a CLI, then the SDK; each of these surfaces requires the same things: an agent loop, tool execution, provider integrations, context management, and all the bells and whistles of being an agent harness.
And while we developed all the newer product surfaces on the Cline SDK, the VS Code extension, the oldest and most-used product, ran on its own private implementation: a ~76,000-line monolithic core where every agent improvement, provider addition, and new model required hands-on work and a new release.
The migration would replace this monolith with the SDK, which automates most of the manual labor our old core required as new models and capabilities come out on a monthly basis. We needed a codebase that could move as fast as the space we compete in.
The new Cline SDK also brings significantly improved performance, particularly with open-weights models. From our post introducing the Cline SDK:
Best-in-class agent harness. With Cline 2.0, we invested heavily into improving our harness. We rewrote the prompts, simplified the loop, tightened context management, improved feedback loops and error handling, and rethought how tools are defined and surfaced to the model. Those gains carry across every Cline surface because they live in the runtime, not the app.
Open-weights models: Note: Cline CLI scores are pass@1; N/A means no published run for that agent + model combination on tbench.ai (terminal-bench 2.0).
Fortunately, the older extension was built on gRPC, which used a messaging system with a shared protocol to communicate between the core and the GUI (a React webview), components we could easily reuse in this new version of the extension built on the SDK. The UI our users have muscle memory for carried over seamlessly with a translation layer that would convert Cline SDK session events back into the message types the webview has always consumed.
Limitations of the VS Code Marketplace #
The Marketplace gives you exactly one release lever: publish, and 100% everyone gets it.
You also can't revert to a lower version. Versions are strictly monotonic, whatever happens, you can only move the version number forward. If a release goes badly, the only remediation is to prepare another higher version release, and wait for the fleet to update: a loop measured in hours to days, not minutes.
And while a slow rollout is standard software practice everywhere else, it's not something VS Code extensions support natively. You publish a new version and everyone gets it: no gradual rollout, no test group, no instant pullback.
Aha moment: what if we bundle two extensions in one? #
To work around this limitation, we bundled two versions of the extension in one release: the legacy original version and the new version. We made the extension itself the rollout mechanism. Installing Cline actually installs three things:
- a ~46 KB script
legacy/
, the original pre-migration extensionnext/
, the new SDK based extension
On every window launch, the decides from cached state which bundle to activate, driven by a percentage rollout on a PostHog feature flag. It works as follows:
Users with the feature flag off run the legacy extension; flag on runs the migrated extension. The legacy cohort gets the same pre-migration code, so nothing changes or breaks in any existing user's experience in prod. That's our control group, and we slowly dial up the percentage with the flag turned on.If the new extension crashes, we fall back automatically. Ifnext
fails during activation, the starts legacy in the same window and pins that machine tolegacy
. The user gets a functioning product, while we capture the crash in telemetry.The flag is the kill switch. If something starts going wrong, we can turn the rollout down to 0%.State is shared. Both bundles read and write the same settings, credentials, and task storage, so moving between cohorts round-trips your data.The flag refresh happens in the background after activation and takes effect on the next window. Nobody's agent switches engines underneath them mid-task.
And most importantly: legacy
and next
had to expose identical views and extension lifecycle entry points to work with the VS Code APIs, otherwise the package wouldn't build. VS Code extensions declare most of their IDE integration statically, in a package.json
manifest: the sidebars and views they contribute, their commands, keybindings, settings schemas, and activation events. One extension gets one manifest, so our build generates a union manifest from both bundles. Contributions both bundles declare pass through; commands that exist in only one bundle get gated behind a context key the sets; and the build hard-fails if the two branches' views or settings schemas ever diverge. That build-time contract is what kept the two codebases swappable on reload while both lived behind the same package.json
.
A/B test, and data, data, data #
Shipping safely was half the problem; proving the new engine better was the other half. Every telemetry event from a rollout build carries extension_variant: next | legacy
.
We spent a tremendous amount of time ensuring the telemetry events captured from the old vs. new extensions are comparable, including adding a bunch of events and metric observability to our old harness. One thing we would recommend to anyone running a migration like this: instrument the old system before you start comparing it with the new one.
With that work, we put up a rigorous side-by-side dashboard to watch the most important events across the old and new harnesses.
For example, the metric that matters most is task.mistake_limit_reached, an artifact from our old harness: the agent stops if it hits three consecutive mistakes in a row and simply asks the human for guidance. This has been the most complained-about agent behavior in our old harness, and a lot of those mistakes were tool call failures caused by inefficiencies in the harness itself. We'd been tracking it for the new SDK harness, but not in the old extension. By adding this metric to the old harness with identical semantics, we got a clean comparison of this behavior across the two. To get a confident enough signal, we spent nearly a month slowly dialing up the percentage of users on the new version, while observing both quantitative metrics from our dashboard and qualitative feedback on our GitHub. We held near a 50/50 split for over a week, our cleanest A/B window, and then took the dial to 100%.
The observed share lags the flag on purpose: the first window after any update always runs legacy, and promotion lands on the second reload. The mechanism being slow is the mechanism being safe. The same lag explains the tail after the 100% dial: the remaining legacy share is machines that haven't updated or reloaded yet, draining a little more every day.
The results #
The clearest indicator of improvement was the task.mistake_limit_reached, the metric that measures how often a model makes three mistakes in a row (malformed tool calls, failed edits, responses that go nowhere), where Cline stops to ask you for guidance. Here's how often that happened with the cohorts near 50/50 and each version seeing a full day of production traffic:
6.34% of tasks on the old harness hit the mistake limit → 0.62% on the new harness. 10x fewer tasks erroring out.
But the split by model is where it gets interesting, as the improvement is more skewed towards open weights models:
What improved from the old harness #
The legacy harness isn't bad code. It's a period piece. It was built for a different year, and this is the part of the story we think generalizes beyond Cline in this fast-moving AI landscape. Cline's original harness was designed in mid-2024 around Claude 3.5 Sonnet, the best coding model of its day, and models of that day needed real help. Tools were described in the system prompt and invoked through XML tags parsed out of the model's text stream, because that was the reliable way to get tool use from a 2024 model. The harness guarded, compensated, and spelled everything out.
Models in 2026 are RL-trained to call tools natively. Wrap one in a 2024-shaped harness and you pay a tax on every turn: format issues, parse failures, retries, and eventually the mistake limit. Moving all Cline surfaces to the upgraded harness that being the best performing open weights model is a must - and this is why we went through this journey with such caution and care.
Our journey doesn't stop here. We are always committed to be the best harness for the open weights model, and we are always reflecting and improving. We have also been deeply researched into recursive self improvements that ensure our harness is always the frontier for open weights model. Read more here.
Thank you #
To all our users who helped report issues, sat with us on Google Meets to walk us through their workflows and what broke, helped debug regressions, and gave us feedback to ensure a smooth rollout for everyone: thank you. Cline was built by and for our wonderful community, and we would be nothing without you.
As of August 23, the rollout is at 100%: every Cline window that updates and reloads now runs the SDK extension, and the remaining legacy sessions are draining as machines update. The extension now runs on the same engine as the CLI and the SDK. Every agent improvement, every new tool, every new provider and model capability gets shipped quicker and fixed faster than ever before, across all our products. While the migration required deprecating some less-used features ( see here for the full list), this update turned out to be a hard-fought success bringing the original Cline onto a harness built for today's models, and we're excited for you to feel the difference in token efficiency, reduced costs, and better results, and stay on the journey of continuous improvement with us.