Edge0-35B Benchmarks: What 4-bit Quantization Really Costs You

Edge0-35B-A3B's int4 pipeline scores 3.9 points below its fp16 base across five benchmarks. Here's what that gap means in practice.

What is the benchmark gap between Edge0-35B’s int4 model and its fp16 base? #

Edge0-35B-A3B, a 35B-parameter sparse mixture-of-experts model built on Qwen3.5-MoE, loses an average of 3.9 points across five benchmarks when quantized to 4-bit and run through its edge0 inference pipeline, compared to the original fp16 model. The gap ranges from 1.5 points on GPQA-Diamond to 6.1 points on AIME 2026, based on evaluations the Edge0 team ran with OpenCompass under matched settings for both versions.

That’s the headline number. But the more interesting story is how that gap was produced, and whether a 3.9-point average loss is a good trade for running a 35B model in under 3 GiB of active memory on consumer hardware.

TL;DR #

• Edge0-35B-A3B is a 35B-parameter Qwen3.5-MoE model (256 experts, 4 active per token) quantized to 4-bit and paired with LoRA and prerouter adapters to run in phone-class memory.
• The int4 pipeline scores 79.2 average across five benchmarks versus83.2 for the fp16 base, a3.9-point gap that varies by task.
• The biggest quality loss shows up on AIME 2026 (86.6 vs 92.7, a 6.1-point drop), suggesting quantization hits harder reasoning chains more than factual recall or code generation.
• Recover-LoRA , a distillation technique that trains LoRA adapters against a frozen int4 base using the fp16 model as teacher, is what keeps the gap this small instead of the larger degradation typical of naive 4-bit quantization.
• The model hits 15 tokens/second decode and2.9 GiB peak active memory on a Mac mini M4 Pro, achieved by streaming expert weights from SSD instead of the full 35B parameters into RAM.
• This is a preview release : agentic capability (tool use, multi-step planning) is explicitly weak right now, and the benchmark numbers reflect a model still being extended.

How was the quantization quality loss measured? #

The Edge0 team ran both models, the int4 edge0 pipeline and the fp16 Qwen3.5-MoE 35B-A3B base, through OpenCompass, an open-source evaluation framework, using identical settings and parameters for both. Five benchmarks were used:

• AIME 2026 : competition-level math reasoning problems.

• HumanEval : code generation correctness.

• GPQA-Diamond : graduate-level science questions designed to resist simple lookup.

• MMLU-Pro : a harder, more discriminative version of the standard MMLU knowledge benchmark.

• IFBench : instruction-following fidelity. Here’s the full comparison:

| Benchmark | Edge0-35B (int4) | Qwen3.5-MoE 35B-A3B (fp16) | Gap | 
|---|---|---|---|

| AIME 2026 | 86.6 | 92.7 | 6.1 | | HumanEval | 90.9 | 95.1 | 4.2 | | GPQA-Diamond | 79.8 | 81.8 | 2.0 | | MMLU-Pro | 81.0 | 84.6 | 3.6 | | IFBench | 57.9 | 61.7 | 3.8 | | Average | 79.2 | 83.2 | 3.9 |

Using the same harness and settings for both models matters here. Quantization benchmarks are easy to game or misreport by evaluating the fp16 and quantized versions under different prompt formats, sampling temperatures, or scoring rubrics. Matched conditions make the 3.9-point figure a reasonably clean estimate of what quantization plus recovery training actually costs, rather than an artifact of evaluation setup.

Why does quantization hurt math reasoning more than other tasks? #

The pattern in the table is not uniform. AIME 2026 shows the largest drop (6.1 points), while GPQA-Diamond shows the smallest (2.0 points). IFBench and MMLU-Pro sit in the middle, both losing around 3.6 to 3.8 points.

This lines up with a broader pattern seen across quantization research generally: tasks requiring long, precise chains of intermediate computation (multi-step math proofs, competition problems) tend to be more sensitive to precision loss than tasks that lean on retrieved knowledge or pattern matching (science trivia, general knowledge questions). A single degraded token early in a long reasoning chain can cascade into a wrong final answer, whereas a knowledge question typically depends less on extended, error-compounding computation.

HumanEval’s 4.2-point drop is notable too. Code generation demands exact syntax and logical consistency, and quantization noise can introduce subtle errors that a human grader (or a test suite) treats as an outright failure rather than a partial miss.

What is Recover-LoRA and how does it limit the damage? #

Naive 4-bit quantization of a 35B model would normally produce a larger quality drop than 3.9 points on average. Edge0’s approach uses a technique the team calls Recover-LoRA: the int4 base weights are frozen, and LoRA (low-rank adapter) layers are trained on top via distillation, using the original fp16 model as a teacher.

The idea is straightforward. Instead of quantizing and hoping the model still behaves the same way, you quantize first, then train a small set of additional parameters specifically to compensate for the errors that quantization introduced, using the full-precision model’s outputs as the training signal. Because the adapters are low-rank, they add minimal storage and memory overhead. And because they’re kept unmerged rather than baked into the base weights, a single quantized base model can serve multiple adapter sets without needing to be re-quantized for each one.

Remy is new. The platform isn't. #

Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.

This is different from quantization-aware training from scratch, and different from post-training quantization with no recovery step at all. It’s a middle path: quantize aggressively, then patch the damage with a small, targeted training pass.

Is a 3.9-point average loss good, bad, or unremarkable? #

Context matters more than the raw number. A few ways to read it:

Compared to doing nothing. Naive int4 quantization without any recovery mechanism on a model of this scale would typically be expected to show larger degradation, particularly on reasoning-heavy benchmarks like AIME. The fact that GPQA-Diamond only drops 2.0 points and the average sits under 4 points suggests the Recover-LoRA step is doing real work, not just cosmetic tuning.

Compared to what you get in return. The tradeoff isn’t abstract: the int4 pipeline runs in under 3 GiB of active memory versus what would otherwise require a much larger footprint for a 35B-parameter model at fp16. That’s the difference between “runs on a high-end workstation GPU” and “runs on a Mac mini or similar consumer hardware” via streaming expert weights from SSD. For many use cases, especially on-device or edge deployment, that memory reduction is worth more than 4 benchmark points.

Compared to task sensitivity. If your use case leans on competition math or highly precise multi-step reasoning, a 6.1-point AIME gap is worth weighing seriously. If your use case is more general knowledge or science Q&A, the 2-point GPQA gap is close to noise.

The honest takeaway: 3.9 points is a real, measurable cost, not nothing. But it’s a small cost relative to typical quantization degradation, and it buys a substantial memory reduction.

What are the current limitations? #

Edge0-35B-A3B is explicitly labeled a preview release, and a few limitations are worth flagging beyond the benchmark numbers:

• Agentic tasks are weak. Tool use, multi-step planning, and long-horizon autonomous behavior are called out as underdeveloped in this preview. If you’re building an agent pipeline, this model is not yet tuned for that.
• Language coverage is narrow. The model is primarily tuned for the languages of its base model, Qwen3.5-MoE, so multilingual performance outside that scope is unproven.
• Apple Silicon only, for now. The MLX backend targets Apple Silicon hardware specifically; other backends are described as being on the roadmap but not yet available.
• Long contexts eat into the memory budget. The 3 GiB peak active memory figure applies to short contexts. Longer contexts grow the KV cache and push memory usage up, so the headline efficiency number isn’t guaranteed at every context length.

Frequently Asked Questions #

What causes the quality loss in 4-bit quantized models generally?

Reducing weight precision from 16-bit to 4-bit introduces rounding error into every computation. In small amounts this error is often absorbed without much effect, but it compounds across many layers and many reasoning steps, which is why long chain-of-thought tasks like competition math tend to degrade more than single-step factual recall.

Does Recover-LoRA fully eliminate the quantization gap?

No. It reduces it substantially compared to naive quantization, but a 3.9-point average gap remains across the five benchmarks tested. It’s a mitigation technique, not a full recovery to fp16-equivalent performance.

How much memory does Edge0-35B-A3B actually need to run?

• ✕a coding agent
• ✕no-code
• ✕vibe coding
• ✕a faster Cursor

The one that tells the coding agents what to build.

The measured peak active memory is about 2.9 GiB on a Mac mini M4 Pro for short contexts, achieved by streaming expert weights from SSD on demand rather than the full 35B-parameter model into RAM at once. Longer contexts require additional memory for the KV cache.

Is Edge0-35B-A3B ready for agent or tool-use workloads?

Not yet, according to its own documentation. This preview release is explicitly described as weak on tool use, multi-step planning, and long-horizon autonomy, with agent capability improvements planned for a future full release.

What benchmarks were used to measure the quantization gap?

Five: AIME 2026 (math reasoning), HumanEval (code generation), GPQA-Diamond (graduate-level science), MMLU-Pro (broad knowledge), and IFBench (instruction following), all run through OpenCompass under matched conditions for both the int4 and fp16 versions.