Cognition's SWE-2 coding model scored 83.75% on independent testing, beating DeepSeek V4.1 Flash and Kimi K3, with free access through Devin Pro.
What is Cognition SWE-2? #
SWE-2 is a coding model released by Cognition, the company behind the Devin AI software engineer. It’s post-trained from Kimi K3 using reinforcement learning, with Cognition tuning it specifically for coding performance while also factoring in the cost of completing a task. Independent testing on an eight-task benchmark called KingBench 3 put SWE-2 at 83.75%, ahead of DeepSeek V4.1 Flash at 81.25% and roughly 6 percentage points ahead of its own base model, Kimi K3. It’s currently bundled into Devin Pro, which costs $20 a month, with SWE-2 usage included free through October 10, 2026.
TL;DR #
- SWE-2 outperformed its base model Kimi K3 by a meaningful margin, moving from 77.5% to 83.75% on the same eight-task benchmark, which suggests Cognition’s reinforcement learning post-training added real capability rather than just marginal polish.
- Independent hands-on testing edged out DeepSeek V4.1 Flash , winning three of eight tasks, losing two, and tying three, for a final score of 67/80 versus 65/80.
- Cognition’s own published benchmarks show mixed results : SWE-2 jumped from 44.2% to 50% on Frontier Code 1.1 and from 88.3% to 92.8% on Terminal Bench 2.1, but scored only 27.3% on Terminal Bench 4 compared to 57.9% for GPT-6 Astra.
- Pricing makes it an easy model to try : Devin Pro is $20 a month, and SWE-2 is currently included at no extra cost in Devin desktop and CLI through an offer running to October 10, 2026.
- Longer, multi-step tasks are where it shines , including a full pipeline task involving generating a dataset, fine-tuning a Gemma 2B model, and serving results through a local web interface, where it scored a perfect 10/10.
- The model asks too many clarifying questions , which adds friction for users who want an agent to make reasonable assumptions and move straight to execution.
How does SWE-2 compare to Kimi K3? #
SWE-2 is a post-trained version of Kimi K3, meaning Cognition built on top of an existing strong open model rather than training from scratch. According to Cognition, the training process used reinforcement learning aimed at improving coding output while also weighing the computational cost of finishing a task. The company trained multiple reasoning effort levels together rather than treating them as separate models.
The gap between the two models shows up clearly in testing. On the KingBench 3 leaderboard, SWE-2 landed at 83.75%, while Kimi K3 sat at 77.5%, a difference of about 6.25 percentage points. Cognition’s own reported numbers point in the same direction: Frontier Code 1.1 went from 44.2% (Kimi K3) to 50% (SWE-2), and Terminal Bench 2.1 rose from 88.3% to 92.8%. Since Kimi K3 was already considered a capable model before this post-training, squeezing more performance out of it is a meaningful result rather than a marginal tweak.
How does SWE-2 perform against DeepSeek V4.1 Flash? #
Head-to-head testing across eight tasks on KingBench 3 gives a detailed picture. Each task is scored out of 10, for a maximum of 80 points.
- Elevator simulation (three elevators, boarding logic, wait states): both models tied at 8/10.
- 3D contact lens case in Three.js (interactive caps, correct left/right markings): SWE-2 scored 8, DeepSeek scored 7.
- Folding table with slider animation : SWE-2 scored 8, DeepSeek scored 6, the biggest gap in SWE-2’s favor.
- SVG of a panda eating a burger : SWE-2 scored 8, DeepSeek scored 6.
- Bow and arrow game with targets and a leaderboard : DeepSeek scored 8, SWE-2 scored 7, its lowest result in the run.
- Math permutation problem (expected answer: 20,460): both models scored a perfect 10.
- Dataset generation, Gemma 2B fine-tuning, and local web UI : both models scored a perfect 10.
- 3D wristwatch with live time, date, day, and two time zones : DeepSeek scored 10, SWE-2 scored 8.
Final tally: SWE-2 finished at 67/80 (83.75%), DeepSeek V4.1 Flash at 65/80 (81.25%). SWE-2 won three tasks, lost two, and tied three, giving it a narrow but real edge. On the broader KingBench 3 leaderboard, SWE-2 placed fifth overall, sitting above Fable 5 (82.5%) and both Kimi K3 and Opus 5 (77.5% each), and just below Qwen 3.8 Max 0902, GPT-6 Astra, GLM 5.3, and Fable 5.1.
Where does SWE-2 struggle? #
Cognition’s own benchmark data shows the improvement over Kimi K3 isn’t universal. On Terminal Bench 4, SWE-2 scored 27.3%, well behind GPT-6 Astra’s 57.9%. That’s a large gap and a reminder that gains on one benchmark suite don’t necessarily transfer to others, especially ones testing different terminal or agentic workflows.
In hands-on testing, SWE-2’s weakest showing came on the bow-and-arrow game task, where it lost to DeepSeek V4.1 Flash by a point, and on the 3D wristwatch task, where DeepSeek’s rendering of live time tracking across two time zones outscored SWE-2 by two points. Neither gap is enormous, but they show DeepSeek still holds an edge on certain visually precise, real-time interactive tasks.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Is SWE-2 worth using right now? #
For anyone already paying for Devin Pro, yes, largely because of the pricing. Devin Pro costs $20 a month, and SWE-2 is currently included in Devin desktop and CLI at no additional charge, with that offer running through October 10, 2026. That combination of a strong benchmark model and no extra usage cost makes it a low-risk way to test an agentic coder against real work. The model held up well on longer, multi-stage tasks in testing, including one asking it to generate a dataset, fine-tune a Gemma 2B model, and wire up a local web interface that produces a new output on every refresh. That kind of task requires several stages to work correctly in sequence, since a mistake early on breaks everything downstream. SWE-2 scored a full 10/10 there, matching DeepSeek. A separate test asking it to build a movie-tracking terminal app using the TMDB API also produced one of the stronger results seen across models tested, including working search, ratings, and detail views with no terminal flickering.
What’s the biggest complaint about SWE-2? #
The model asks too many clarifying questions before starting work. One clarifying question at the start of a task is reasonable. Two is tolerable. Beyond that, it starts adding friction that undercuts the appeal of handing a task to an autonomous coding agent in the first place.
For routine implementation choices, most users want the model to pick a sensible default, proceed, and explain its assumptions afterward rather than pausing to ask. System instructions that cap clarifying questions to one (unless truly blocked) and encourage bundling any necessary questions into a single message may help reduce this, though that approach hasn’t been formally verified as a fix. It’s a workaround rather than a guaranteed solution, and it’s worth testing directly if this becomes a recurring annoyance in your own workflow.
Frequently Asked Questions #
What is Cognition SWE-2 based on?
SWE-2 is post-trained from Kimi K3 using reinforcement learning, with Cognition optimizing for both coding accuracy and the computational cost of completing tasks.
How much does SWE-2 cost to use?
SWE-2 is included in Devin Pro, which costs $20 a month. Usage is currently free within Devin desktop and CLI as part of an offer running through October 10, 2026.
Does SWE-2 beat DeepSeek V4.1 Flash?
In one round of independent testing across eight tasks, SWE-2 scored 83.75% versus 81.25% for DeepSeek V4.1 Flash, winning three tasks, losing two, and tying three.
Is SWE-2 better than Kimi K3?
Yes, in this testing SWE-2 scored roughly 6 percentage points higher than Kimi K3, its base model, and Cognition’s own benchmarks show similar gains on Frontier Code 1.1 and Terminal Bench 2.1.
What’s the main downside of SWE-2?
It tends to ask multiple clarifying questions before starting a task, which can slow down workflows for users who prefer the model to make reasonable assumptions and proceed.