Mistral Large 4 Benchmark for Coding vs DeepSeek V4.1 Flash An independent benchmark run by Leanzero found that Mistral Large 4 scored 0.5462 on its Gauntlet agentic coding benchmark, ranking 28th of 44, and 0.1086 and 0.3304 on its two Forge 2.0 builds, trailing DeepSeek V4.1 Flash 1017, which scored 0.6957 on Gauntlet and 0.2422 and 0.6618 on Forge at roughly a tenth of the cost and a quarter of the time. The tester attributed much of the Gauntlet shortfall to a scorer bug that misread a payments table, noting all 12 rendered amounts were correct, while the Forge code losses were the model's own. Mistral released Large 4 on 6 October as a public preview with 1.05 trillion parameters, 52 billion active and a 1M-token context, claiming it beats DeepSeek V4 Pro on its own Coding Agent Index at 49.8%. Mistral Large 4 scored 0.5462 on our Gauntlet benchmark, 28th of 44. About a third of what it lost came from a check of ours that read its table the wrong way round. All 12 amounts it put on screen were correct. I went in expecting to write about a slow, expensive model, and on our Forge benchmark that part holds up. If you pay for a coding agent, this model is pitched at your budget. Mistral released Large 4 on 6 October as a Public Preview https://docs.mistral.ai/models/mistral-large-4-0.md , 1.05 trillion parameters with 52 billion active and a 1M-token context, at half its list price for the first two weeks. Their launch post puts it ahead of DeepSeek V4 Pro on their own Coding Agent Index 49.8% , which is the kind of claim that moves a budget, so we ran it through both of our agentic coding benchmarks. The Gauntlet https://leanzero.net/agentic-benchmarks?family=gauntlet&utm source=devto&utm medium=referral&utm campaign=crosspost has an agent build a payments console with two services and a 3D view inside 150 model calls, then scores it on 99 checks. Forge 2.0 https://leanzero.net/agentic-benchmarks?family=forge&utm source=devto&utm medium=referral&utm campaign=crosspost asks for a v2 upgrade of a working Jira Forge app inside 300 calls and scores it on 105 checks against three seeded Jira sites. Getting a Gauntlet score took four attempts. The first, on 9 October, ended after 49 calls on OpenRouter's "Upstream idle timeout exceeded". On 18 October the second was ended by Mistral's servers. Early on 19 October the third was ended by a bug in goose, the open-source coding agent we run our own fork , and the fourth finished that afternoon. Its Forge 2.0 pair scored 0.1086 and 0.3304 for at least $10.95. DeepSeek V4.1 Flash 1017 did better on the same Forge task. It's the 17 October snapshot I wrote about in my DeepSeek piece https://leanzero.net/blog/deepseek-v4-1-flash-1017-benchmark?utm source=devto&utm medium=referral&utm campaign=crosspost , and it scored 0.2422 and 0.6618 for at least $0.62 in about a quarter of the time. On the Gauntlet the low score is mostly our scorer's fault. On Forge the code losses were Mistral's own. Our harness also gave it a smaller context window than DeepSeek. The build trees show where each loss came from. This is one scored Gauntlet run and one Forge pair, in our setup, on a model whose vendor says it can change silently while it's in preview. | | Mistral Large 4 | DeepSeek V4.1 Flash 1017 | |---|---|---| | Gauntlet 7.2 | 0.5462 https://leanzero.net/agentic-benchmarks/run/brun-c5d5019f-93d7-4838-9d40-20b392c8d736?utm source=devto&utm medium=referral&utm campaign=crosspost , 144 min, $4.93 | 0.6957, 23 min, $0.21 | | Forge 2.0 build 1 | 0.1086 https://leanzero.net/agentic-benchmarks/run/brun-929df321-c68b-4543-a796-6d575b530837?utm source=devto&utm medium=referral&utm campaign=crosspost , 118 min, at least $5.40 | 0.2422, 28 min, $0.30 | | Forge 2.0 build 2 | 0.3304 https://leanzero.net/agentic-benchmarks/run/brun-f0a33b87-62a6-4eaa-a998-c0928ebe0816?utm source=devto&utm medium=referral&utm campaign=crosspost , 106 min, $5.55 | 0.6618, 30 min, at least $0.32 | | Unscored Gauntlet attempts | 9 Oct: idle timeout after 49 calls, at least $1.43. 18 Oct: Mistral 503 after 30 calls, at least $1.18. 19 Oct: our 400 after 97 calls, $3.04 | none | I checked these on the live boards on 21 October; if a run page says something else now, trust the run page. All six Mistral runs together cost at least $21.53, with three of the bills only lower bounds. Most Gauntlet checks just add to a weighted average, but a handful are critical, and when one of those fails it multiplies the whole score down, by as much as x0.6. Wrong money in a payments console is one of those. The money check is called b money rendered . A browser probe reads the amount cell of the first 12 rows of the payments table. Then the scorer checks that the digits match a stored amount and that the decimals match the currency, because amounts are stored as integers in the currency's smallest unit cents, or fils for the Kuwaiti dinar . A correct cell shows two decimals for EUR and USD, none for yen and three for dinars. Mistral's run scored 0.00 on it, with the detail line "0.00 of 12 cells exponent-correct; JPY/KWD traps 0.00", and the multiplier took the whole run from 0.9104 down to 0.5462. These are the 12 cells the probe read, verbatim: KWD 111.000 , ¥594 , ¥199 , $4.47 , ¥809 , €224.00 , KWD 82.400 , €33.10 , KWD 2.420 , ¥1,060 , $1,840.00 , KWD 72.400 . Our probe also records each row's id, and joined to the fixture, pay 12287 is stored as KWD 111000, so 111.000 dinars is right, and pay 12286 is JPY 594, which needs no decimals. All 12 are right. So are the probe's 12 extra rows. Its formatter in web/app.js lines 7 to 25, with the source's \u escapes shown as characters never touches a float: js const CUR = { EUR: { sym: '€', exp: 2 }, USD: { sym: '$', exp: 2 }, JPY: { sym: '¥', exp: 0 }, KWD: { sym: 'KWD ', exp: 3 }, }; ... const s = String a .padStart exp + 1, '0' ; const int = s.slice 0, s.length - exp || '0'; const frac = exp 0 ? s.slice s.length - exp : ''; It planned this before writing it. At 10:58 our time EEST , 27 minutes in, it checked the contract's example and two real payments in its thinking: "129900 EUR → €1,299.00 ✓; JPY 8564 → ¥8,564 ✓; KWD 8263000 → KWD 8,263.000 ✓". Our scorer's own screenshot of the inspector shows that dinar payment as KWD 8,263.000 , and the inspector check q payment context passed. Here is the scorer, score sb7.py around line 973: first page = c.pack.payments :50 by cur: Dict str, set = {} for row in first page: by cur.setdefault row "currency" , set .add row "amount minor" payments is stored in date order, so payments :50 is pay 00000 to pay 00049 , the oldest fifty. A rendered cell only passes if its digits equal one of those fifty amounts, so the check assumes, without saying it, that the table opens oldest first. Mistral's opens newest first sort: '-created at' , web/app.js:49 . None of its 12 cells could ever match. Our probe even collects the row ids "so the scorer can join against fixtures", in its own comment. No scorer file reads them. I re-graded the run with the scorer's own money ok function, copied unchanged. It gives 0.0 as written, 1.0 against the 50 rows actually on screen, and 24 of 24 when each cell is joined to its payment by row id. Was opening newest first a mistake by Mistral? Its contract fixes the API's default order " created at default, ascending by INSTANT " and says the Date header toggles, but it never says which way the table starts. Mistral saw the gap and decided, at 11:48 our time: "The grader might check the first page shows specific rows? Unlikely to assume an order. I'll default to '-created at' newest first — most console-like." It bet that the grader would not assume an order, and the grader did. You can argue the UI should inherit the API's default. I'd listen. Even then it's a sort-order quibble. The check charged it as wrong money. All five models I compared passed with 1.0: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol the model behind Jev Router's top Forge score https://leanzero.net/blog/jev-router-benchmark-gpt-6-1-sol?utm source=devto&utm medium=referral&utm campaign=crosspost , GPT-6.1 Sol Pro and DeepSeek V4.1 Flash 1017. All of them open their table with sort: 'created at' , and so does our own reference app, which is why our release validation never met a descending default. Grade those five tables against the newest 50 payments and four score 0.0, Sol 0.05. DeepSeek's formatter is close to a copy of Mistral's: js let digits = String Math.abs minor ; if exponent 0 digits = digits.padStart exponent + 1, '0' ; const whole = exponent 0 ? digits.slice 0, digits.length - exponent : digits; Same idea, same output. It got 1.0 and Mistral got 0.0. Sol and Sol Pro call Intl.NumberFormat and divide the integer by 10 exp first, and Sonnet does the same with Math.pow , which is fine for amounts this size. So on money code it's a tie between Mistral and DeepSeek. The sort order made the difference. Mistral did leave one real money bug, in its own test. Its browser self-test checks the first amount against /^ €$¥ \d \d, \.\d{2}$|^KWD \d, +\.\d{3}$/ , which would reject a correct ¥594 because it demands two decimals after a yen sign. It printed "PASS money formatted" four times, and my guess is that its own first row was never a yen payment. I can't see which row it was. The score path for this run goes like this. Its checks earned 0.9104. Our false critical multiplied that by 0.6, to 0.5462. Separately, a run that fails some of the 3D checks gets a ceiling on its score, and Mistral's partial failures on picking, the coasting camera and the overview's legibility set it at 0.709. The ceiling formula keeps a score just under the cap, so without the false critical the run would come out at about 0.7048. That's my arithmetic, not a re-score. The board still says 0.5462. It would sit 11th of 44, two places above DeepSeek 1017's 0.6957. The 3D failures, which are Mistral's own, cost it about 0.2 on top. My audit of that check turned up a second flaw. The yen and dinar "traps" score 0 when no yen or no dinar row happens to be among the cells the probe reads up to 12 . A run whose visible amounts are all right can still pay x0.92. Four ranked runs paid it that way: Elorian Glimpse, Step 5 Preview, Qwen3.8 Omni Flash and GLM-5.3. Across every Gauntlet 7.2 build I could reach, Mistral is the only one the sort assumption hit. Forge 2.0 tests the same habits harder: the agent reads a long spec, upgrades a working Forge app under a Jira rate model, and has to prove it all in a local sandbox our Forge emulator kit before the calls run out. My piece on the September DeepSeek V4.1 Flash https://leanzero.net/blog/deepseek-v4-1-flash-coding-benchmark?utm source=devto&utm medium=referral&utm campaign=crosspost covered this same task, with medians of 0.33 for it and 0.67 for Opus 5.5 in an earlier comparison group. Both Mistral builds and DeepSeek 1017's pair are in the same comparison group on our board. That means the same task, seeds, scorer and engine build. All four asked for the same effort setting. A Forge 2.0 score is the earned score times a critical multiplier times a reliability factor, where reliability starts at 1 and loses up to 10% for each group of checks whose worst check failed, with a floor of 0.25. Build 1 sat on that floor its unfloored product was 0.2194 and build 2 came out at 0.4718. Neither triggered a critical. My reading is that most of the 0.22 gap comes from build 1's throttle and the order it ran its migration in. The task gives each Jira site a rate budget of 2,400 points an hour, and background work may use 70% of it, so 1,680. Build 1 wrote a gate for that in src/dose.js lines 76 to 88 : js if dose.bg + estimatedCost limit { const untilMs = hourStartOf now + 3600000; await kvs.set PAUSE KEY, { untilMs } ; throw new QuotaPaused untilMs ; The gate is fine. The estimates feeding it were not. With no points header from the site, it charged its own counter from guesses. It booked 51 points for a board or sprint list page src/config.js:43 , cost: 1 + 50 , 105 for a search and 260 for a changelog bulk fetch. On the scoring site its first scheduled run had counted 1,567 points after 21 seconds. The next estimated call would have taken it past 1,680. So it paused for the rest of the hour. Yet the site's own ledger for that run says it had spent 715 points over 27 requests, with list pages costing 1.5 to 2 points each. That would only have been a nuisance if the migration ran first. It ran last. reconcile in src/index.js calls reconcileAll , then saveClosed , then migrateV1Rows , and the pause in the first step skipped the rest. Its rescan had no cursor. Each hour it started over and paused in the same place. Our scorer's verdict reads "0/130 historical changes v1 never recorded" and "h2: 0 of 59 v1 rows in scope-ledger", and it traces 15 checks to that one root cause. On the development site, which our spec gives the same shape on another seed, the gate tripped too, twice. At 21:19 our time it wrote "The reconcile hit my own dose gate". After a reset one run got through and it reported "migration complete 88/88 ". Build 2 got the same thing right. I think it was more luck than design. Its cost counter under-counted instead its own words near the end: "my costOf under-counts returned records" . It ran over the background share in 4 of 31 hours, and on our scoring sites it never paused early. In its last minutes its own dev run did pause early. At call 300 of 300 it spotted the ordering problem: "I'm at the call budget 300/300 — this is my final turn". Then it moved the migration to the top with the comment "it is the graded deliverable and needs the fewest points". It never ran that version. It still scored 59 of 59 migrated and 130 of 130 backfilled. DeepSeek 1017's build 2 had that ordering from the start src/sync.js:409 , "Migrate v1's rows first" . Both Mistral builds score 0.00 on the admin panel tier. That one failure takes down the panel labels, the CI web trigger no secret can exist before the panel's Rotate button creates one and the admin boot check with it. Build 1's src/frontend/admin.jsx ends with export default AdminPage; and never calls ForgeReconciler.render , so UI Kit had nothing to draw. It found that out at 21:39, on the 298th of its 300 calls, when its only admin-page test printed " nothing rendered ". Build 2 does call ForgeReconciler.render