Summary #
Newer doesn’t always mean better. “Flash” doesn’t always mean faster.
Sol 6 is cheaper and faster than Sol 5.6, with similar overall quality. In my opinion, MiniMax M3 and M3.1 Flash are not worth using: Luna 6 has better overall quality at a lower cost. Mistral Large 4 does well on extraction, while Gemini Flash-Lite leads on speed.
Sol 6: a cheaper, faster upgrade #
Sol 6 is a solid upgrade over the already strong Sol 5.6 in cost and speed. Overall quality is close, and the gains are uneven across tasks.
All five workloads, current run
| model | mean quality | cost / task | latency |
|---|---|---|---|
| GPT-6 Sol | 0.944 | $0.036 | 12.2 s |
| GPT-5.6 Sol | 0.939 | $0.081 | 19.8 s |
| GPT-6 Luna | 0.919 | $0.0019 | 17.0 s |
Sol 6 scored higher on architecture and matching. Sol 5.6 scored slightly higher on repository implementation, extraction, and research. On repository implementation, Sol 6 was faster: 19.8 seconds versus 26.4 seconds.
My takeaway: A cheaper, faster upgrade, with similar overall quality.
MiniMax M3 and M3.1 Flash: not worth using #
In my opinion, MiniMax M3 and M3.1 Flash are not worth using. Luna 6 has better overall quality and costs less. MiniMax Flash scored higher on coding, and M3 scored slightly higher on matching, but Flash was slower than M3 on four of the five workloads.
All five workloads, current run
| model | mean quality | cost / task | latency |
|---|---|---|---|
| MiniMax M3 | 0.793 | $0.013 | 26.9 s |
| MiniMax M3.1 Flash | 0.865 | $0.0073 | 52.9 s |
| GPT-6 Luna | 0.919 | $0.0019 | 17.0 s |
On repository implementation, M3.1 Flash averaged 145.2 seconds, compared with M3’s 79.7 seconds and Luna 6’s 30.9 seconds. That makes MiniMax Flash almost 5× slower and nearly 5× more expensive than Luna 6 on this workload. MiniMax Flash completed 13 of the 16 coding responses. M3.1 Flash scored better than M3 on coding and architecture, while M3 scored better on extraction and matching.
I would not recommend them unless you have your own idle GPUs or need to keep your data in your own environment. I have not tested their self-hosted performance or cost.
My takeaway: Luna 6 offers better overall quality at a lower cost.
Mistral Large 4: strong on extraction #
Mistral Large 4 is a competitive model for tasks like extraction, matching, and research, but it is slow. If you’re in the EU and looking for a model developed within the EU, it is an option to consider.
All five workloads, current run
| model | mean quality | cost / task | latency |
|---|---|---|---|
| Mistral Large 4 | 0.894 | $0.043 | 53.6 s |
| GPT-5.6 Sol | 0.939 | $0.081 | 19.8 s |
Mistral Large 4 scored 0.96 on extraction, ahead of Sol 5.6 and Luna 6. On repository implementation it scored 0.72, compared with Luna 6’s 0.80, while taking 102.2 seconds versus Luna 6’s 30.9 seconds. It was slower than Sol 5.6 on all five workloads.
My takeaway: An EU-developed option worth considering for extraction, if speed is less important.
Gemini 3.1 Flash-Lite: speed for simpler tasks #
While Gemini 3.1 Flash-Lite is not a recent release, I finally got a chance to test it too and compare it with MiniMax M3.1 Flash and others. My first impression: Gemini 3.1 Flash-Lite is really fast.
If you need high speed for relatively easy tasks, it is a good option to consider. It could even be useful for simple, well-validated tasks in CI/CD pipelines.
All five workloads, current run
| model | mean quality | cost / task | latency |
|---|---|---|---|
| Gemini 3.1 Flash-Lite | 0.761 | $0.0044 | 3.6 s |
| GPT-6 Luna | 0.919 | $0.0019 | 17.0 s |
Flash-Lite was the fastest model on all five workloads, averaging about a second on matching and 7.2 seconds on repository implementation. Its extraction score was 0.82, compared with Luna 6’s 0.90. The CI/CD suggestion is a use case to explore, not something this benchmark directly tested.
My takeaway: Speed for simpler tasks, with validation.
Testing results #
Full results #
| model | mean quality | cost / task | latency | Architecture | Repository implementation | Noisy extraction | Semantic matching | Grounded research |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Sol | 0.944 | $0.036 | 12.2 s | 0.970 | 0.900 | 0.910 | 0.990 | 0.951 |
| GPT-5.6 Sol | 0.939 | $0.081 | 19.8 s | 0.937 | 0.905 | 0.924 | 0.967 | 0.961 |
| GPT-6 Luna | 0.919 | $0.0019 | 17.0 s | 0.953 | 0.804 | 0.900 | 0.968 | 0.969 |
| Mistral Large 4 | 0.894 | $0.043 | 53.6 s | 0.865 | 0.721 | 0.959 | 0.968 | 0.958 |
| MiniMax M3 | 0.793 | $0.013 | 26.9 s | 0.620 | 0.674 | 0.890 | 0.973 | 0.811 |
| MiniMax M3.1 Flash | 0.865 | $0.0073 | 52.9 s | 0.868 | 0.827 | 0.863 | 0.946 | 0.820 |
| Gemini 3.1 Flash-Lite | 0.761 | $0.0044 | 3.6 s | 0.472 | 0.689 | 0.823 | 0.939 | 0.882 |
Some responses were unavailable. Costs and latency describe successful model responses.
Methodology #
We tested five workloads with eight cases per workload, repeated twice. Sonnet 5.5 graded responses against task-specific criteria; quality scores measure rubric performance, not task pass rates.
The charts give equal weight to all five workloads; costs exclude judging and setup.