The real barrier to self-hosting large models has never been "not smart enough" — it's "doesn't fit."
Want to run a 100B-class model? The standard answer is A100s, H100s, or an inference cluster. For small teams, the hardware budget is the wall.
Strata (17002 stars, MIT, C++) pushes that wall back: run a 125-billion-parameter model on a single 12 GB consumer GPU.
Strata applies low-bit quantization (Q2_0 / IQ2_XS) to Qwen3.8-Flash-Next plus a purpose-built inference engine, compressing a model that normally needs a server down to what a gaming PC can hold.
Measured by the authors on two ordinary gaming PCs:
| Hardware | Quant | Generation | Prompt read (32K ctx) |
|---|---|---|---|
| RTX 5070 (12 GB) + Ryzen 5 7600 | Q2_0 | 94 tok/s | 2,650 tok/s |
| RTX 5070 (12 GB) + Ryzen 5 7600 | IQ2_XS | 79 tok/s | 2,090 tok/s |
| RX 9070 XT (16 GB) + Ryzen 9 3900X | — | — | — |
For reference: human reading speed is roughly 5–10 tokens/s. 60 tok/s already outruns reading — so a ~$1,000 gaming rig emits a 125B model's output faster than you can read it. Running 125B on consumer hardware costs quantization precision. Q2_0 / IQ2_XS are 2-bit-class schemes — high compression, but with inevitable capability loss. Not every task substitutes for a full-precision model.
Practical constraints: a 12 GB VRAM floor (NVIDIA or AMD); deeper quantization means measurably weaker complex reasoning and long-horizon tasks; it suits local individual/small-team use and privacy-sensitive, budget-limited scenarios — not high-precision production inference.
I've localized the README and core docs to Chinese: [https://github.com/yangshun2005/Strata-cn](https://github.com/yangshun2005/Strata-cn)
If you find this project useful, a star on the original repo supports the author's ongoing maintenance.