As you probably know, I run open-weight models on our company's rented GPUs for our own internal developers team of developers who work with long-running coding agents. We are currently using Qwen3.8-Flash-Next for most of our tokens but earlier this week I was benchmarking few alternatives, including NVIDIA Nemotron 3.5 Super VL: 120B total params, 12B active, a hybrid Mamba-Transformer MoE (mostly Mamba layers, whose memory use barely grows with context length). I used NVFP4, a 4-bit weight format, on vLLM 0.33.0, an open-source serving engine .. and a B200 GPU at UpCloud, Helsinki.
.. however, having tried some of the earlier Nemotron releases and been disappointed by their coding capability after few hours in real world coding tasks this seemed to bite again. It seems to be no match for Flash Next. But how much poorer? How to find out?
Well, the obvious answer was benchmarks. Plenty of those to pick from. I went with SWE-bench Verified with mini-swe-agent. SWE-bench Verified is 500 real GitHub bug fixes that actual humans have checked are solvable. The agent gets the repository as well as the issue text and writes a patch, one shotting it. Then the official benchmark harness runs project's own tests to see if the patch passes. mini-swe-agent is the bare-bones agent for it, one bash tool and nothing else, so you can compare models and not some fancy scaffolds. We (me and ze agents) did not run all 500. The set was weighted toward long tasks because that is what most of our token consuption is .. agents writing or testing software in long running sessions. We took all 45 tasks that are rated at more than an hour of human work, plus 55 drawn at random from the 15 minutes to 1 hour tier. Four matplotlib tasks would not run on our rootless Podman setup for any model, which left 96 as the final tally.
In the benchmark runs Qwen3.8-Flash-Next solved 93 of the 96 tasks, including 43 of the 45 long ones. Nemotron solved 45. Massive gap. Agents and yours truly were looking at those figures and it was pretty clear that Flash Next's figures could not be real. I asked agents to have a look and they were able to quickly tell me that Flash Next agents had, for many tasks, landed on exactly on official solutions.
Median similarity between the agents' solutions and the official solutions was 0.81 for Qwen and 0.13 for Nemotron. 28 of Qwen's patches were byte-identical to the merged fix! Nemotron had some byte-identical patches too, 5 of 96, and 8 at 0.9 similarity or above, against 43 for Qwen.
The model has been trained on the tasks! Either on purpose or accidentally, we'll probably never know. The dataset has been public since 2024 after all. In the end the run was absolutely no use as a comparison.
So, in semi-empirical tests my agent teams' leader agents told me that "please can we switch back to Qwen3.8-flash-next for subagents, this Nemotron sucks" and I yielded under pressure. But I was still able to get some value out of the Nemotron trial.
It feld like a pretty good generic model for general office work: "fan out agents, use research skills and research topic X for me, ingest into wiki". Lighting fast.
Here is what it did on the B200 with simulated coding agents, each holding about 100k tokens of conversation:
| Agents | Nemotron: time to first token (median / slowest 5%) | Nemotron: speed per agent | Nemotron: all agents together | Qwen3.8-Flash-Next: speed per agent | Qwen3.8-Flash-Next: all agents together |
|---|---|---|---|---|---|
| 1 | 0.61 s / 2.22 s | 499 tok/s | 281 tok/s | 369 tok/s | 369 tok/s |
| 8 | 0.78 s / 2.44 s | 211 tok/s | 1,048 tok/s | 96 to 109 tok/s | about 800 tok/s |
| 32 | 0.94 s / 2.80 s | 71 tok/s | 1,566 tok/s | about 76 tok/s | about 2,400 tok/s |
| 64 | 1.03 s / 4.11 s | 41 tok/s | 1,614 tok/s | not tested | not tested |
No request failed at any level, 64 agents included. "Speed per agent" is how fast each answer is written, "all agents together" is the total output over the whole 12-minute test window. Past 32 agents the total hardly grows, so more agents just means slower agents: 71 tok/s each at 32, 41 at 64. The 64 is the limit set in the server. The GPU's memory could hold roughly 100 such conversations at once. Most of each prompt was already cached (87.6% by the engine's own counters), so the first-token times are for warm sessions, not for a fresh 100k prompt.
The Qwen3.8-Flash-Next columns come from our own tests on the production B200 before it went live, so it is a different test with different counters, and only indicative. Read that way, Nemotron is faster per agent for one to eight agents and level at 32 (71 against about 76 tok/s). Its total output is lower at one agent (281 against 369) and at 32 agents (1,566 against about 2,400), and higher at eight (1,048 against about 800).
But can it REALLY be used for coding? Nope.
Few months ago I burned some zillions of tokens to create our own benchmark tool, but then abandoned it as its figures would not have meant anything to anyone out there, and I would have had to burn even more zillions of tokens to build some kind of a leaderboard.
But was this the right decision? At least my tasks were tasks that have not been published online ...
If anyone has any good ideas on how to get some reliable metrics to compare models, please let me know!