Fermion Research shipped Neutrino-1 8B yesterday — a 3.88 GB model that hits 763 tokens per second on an H100 using a companion 0.6B draft model, with no waitlist, Apache 2.0 licensing, and a two-command install. The model uses a proprietary ternary-family weight format trained natively in a three-value space ({-1, 0, +1}), not post-hoc quantized from fp16. It fits beside its KV cache on an 8 GB GPU and runs at 24.9 tok/s on an M5 MacBook using only the CPU — all from a single 3.88 GB artifact.
Not Quantization — Trained Ternary #
The difference matters. Standard quantization (GGUF, AWQ, GPTQ) takes a trained fp16 model and rounds weights down to fewer bits after the fact. Fermion trained Neutrino-1 in a three-value space from the start: every weight is either -1, 0, or +1. Of the 6.95B coded weights across 36 transformer layers, 62.63% are zero, with the remainder splitting almost exactly between positive and negative. That sign balance emerged without explicit training constraints — the model found it on its own.
The practical effect: weights stay bit-packed at rest and decompress inside matrix kernels during inference. Nothing in the decode path materializes as fp16 or fp32. The closest parallel is Microsoft’s BitNet (1-bit weights), but Neutrino-1 uses three values instead of two, which preserves more expressiveness per parameter. According to the Qwen3 technical report, the base model in fp16 scores 74.7 on MMLU — Neutrino-1 lands at 72.1, a 2.6 point gap from ternary compression.
The Performance Numbers #
Single-stream decode rates across hardware:
| Hardware | Speed | Method |
|---|---|---|
| H100 80GB | 763 tok/s | Speculative decoding (+draft) |
| H100 80GB | 396 tok/s | Plain greedy |
| Apple M5 (MLX) | 33.7 tok/s | Metal kernels |
| Apple M5 (CPU-only) | 24.9 tok/s | Native binary, 9 threads |
| NVIDIA L4 (GGUF) | 30.7 tok/s | CUDA offload, llama.cpp fork |
The speculative decoding is where the design pays off cleanest. A 0.6B draft model ships alongside the 8B verifier. The draft generates a batch of tokens, the 8B scores the full sequence in one forward pass, and only tokens matching the 8B’s own argmax are accepted. Across 27,648 consecutive tokens tested, zero divergences from plain greedy. On counting and list tasks, the draft gets 6 of 6 tokens accepted per pass, yielding a 1.93x speedup. Factual prompts hold at 1.55x, prose at 1.34x. Conversational gets only 1.13x — the draft mispredicts casual language more than structured output.
Worth noting: the NVIDIA L4 GGUF fallback gets 30.7 tok/s, while the M5 MacBook’s native path gets 33.7 tok/s via MLX. The native ternary engine is genuinely faster than the GGUF compatibility route on the same class of hardware.
Where the Quality Lands #
MMLU (5-shot): 72.1. GSM8K (flexible): 53.4. IFEval (prompt-strict): 77.2. The MMLU gap versus fp16 Qwen3-8B (74.7) is comparable to what you lose going to Q4_K_M GGUF quantization, so ternary-native training hits similar quality at a similar compression ratio. However, GSM8K at 53.4 is below average for an 8B model in mid-2026 — math reasoning appears disproportionately hurt by the ternary constraint. The model page lists these scores without spin, which is worth noting. IFEval at 77.2 is respectable; instruction following holds up well.
If you need serious math reasoning from an 8B model, this is not the right choice at this quality level. For chat, RAG, and instruction-following tasks, the numbers are competitive.
Installing Neutrino-1 8B #
pip install fermion-research
fermion chat
fermion chat --draft
fermion serve
The GGUF fallback is available through Fermion’s llama.cpp fork for full CUDA offload environments. That path works but sacrifices the main speed advantage — 30.7 tok/s on an L4 versus noticeably faster on native hardware. The combined model-plus-draft footprint for speculative decoding is 4.2 GB (3.88 GB verifier + 328 MB draft), which fits on an 8 GB GPU with room for the KV cache.
The Tradeoff Worth Acknowledging #
The proprietary weight format is the legitimate risk here. GGUF has become the portable standard for local model deployment — it works in Ollama, LM Studio, and anything built on llama.cpp. Neutrino-1’s native format requires Fermion’s runtime pip package. Apache 2.0 licensing means the community can fork and build, but there is no Fermion integration in Ollama yet. For production workloads, a dependency on a small new lab’s runtime is worth thinking through carefully.
The counter: two commands to install, an OpenAI-compatible endpoint out of the box, and a GGUF fallback for shops that need standard toolchain compatibility. This is not the first model to ship with a custom runtime — for context, NVIDIA Nemotron-Labs TwoTower also requires specific infrastructure for its diffusion decoder. The community will test it and the numbers will speak. Discussions on Hacker News put it on the front page today with engagement from developers running local inference.
Who Should Try It Now #
Run it if you have M-series Apple silicon or an 8 GB GPU and want a sub-4 GB model with a built-in speculative decoding companion that does not require manual GGUF setup. The install is genuinely two commands. Skip it if your workload is math-heavy (GSM8K: 53.4) or if you need the model to drop into an existing Ollama workflow without changes. If Fermion ships Ollama or Hugging Face Transformers integration, that removes the only real friction point. The format is the last thing standing between this and the standard local LLM toolchain.