{"slug": "how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide", "title": "How to Run Apex-35B Locally with llama.cpp: A Setup Guide", "summary": "Apex Intelligence released Apex-35B, a sparse mixture-of-experts model built on Qwen3.6-35B-A3B with 35 billion total parameters and roughly 3 billion active per token, under an Apache 2.0 license on Hugging Face as Apexintelligence-AI/Apex-35B-V1.0, with a context window up to 262,000 tokens. In testing, the Q8 quantization served through llama.cpp consumed just over 38GB of VRAM on an 80GB H100, and the model fixed a hidden billing bug across a multi-screen web app in about five and a half minutes. Apex Intelligence said an internal automated research pipeline, Apex Research, generated training data, ran experiments, tuned training recipes and optimized GPU kernels, so part of the model's development was AI-assisted.", "body_md": "# How to Run Apex-35B Locally with llama.cpp: A Setup Guide\n\nRun Apex-35B locally with llama.cpp. Covers quantization options, VRAM needs, and setup for this self-improving Qwen 3.6 MoE model.\n\n## What is Apex-35B?\n\nApex-35B is the first model released by Apex Intelligence, built on top of Qwen3.6-35B-A3B and published on Hugging Face as Apexintelligence-AI/Apex-35B-V1.0. It’s a sparse mixture of experts (MoE) model with 35 billion total parameters, but only around 3 billion are active per token, which keeps inference costs much closer to a small dense model than a 35B one. It’s licensed under Apache 2.0, supports a context window up to 262,000 tokens, and decides on its own when to reason longer versus answer quickly. The model is tagged for both text generation and image-text-to-text, and ships in safetensors format split across 26 shards.\n\n## TL;DR\n\n- **Apex-35B** is a sparse MoE model (35B total, ~3B active params) fine-tuned from Qwen3.6-35B-A3B and released under Apache 2.0.\n- The model supports **long context** up to 262,000 tokens and can adaptively choose how long to “think” before answering.\n- Running the **Q8 quantization** via llama.cpp consumed just over 38GB of VRAM in testing on an H100, meaning lower-memory quants can fit on more modest consumer and prosumer GPUs.\n- Apex Intelligence built it using an internal **automated research pipeline** (Apex Research) that helped generate training data, run experiments, tune training recipes, and optimize GPU kernels, meaning part of the model’s own development was AI-assisted.\n- In hands-on testing, Apex-35B handled **real-world coding and debugging tasks** well, including finding and fixing a hidden billing bug across a multi-screen web app in about five and a half minutes.\n- On **benchmarks** , Apex-35B outperforms its Qwen3.6 base model and a similarly sized competitor, and even edges out much larger models like GLM 5.3 and DeepSeek on select research and biology questions.\n- Weaker spots showed up in **creative/visual coding tasks** (a canvas-based rain effect) where the output matched the functional spec but not the visual polish described in the prompt.\n\n**›** Build a shift-handoff app for 900 nurses.\n\n*already included:*✓ HIPAA✓ access control✓ audit log✓ self-hosted\n\n## How do you install and run Apex-35B with llama.cpp?\n\nRunning Apex-35B locally follows the same general pattern as other GGUF-compatible MoE models served through llama.cpp:\n\n1. **Get the model.** Pull Apex-35B-V1.0 from Hugging Face under Apexintelligence-AI. You can use the full BF16 precision weights or a quantized GGUF conversion, depending on your hardware.\n2. **Choose a quantization level.** Q8 is a solid middle ground for quality retention versus memory footprint. Lower quants (Q4 or Q5 variants) will shrink VRAM requirements further at some cost to output quality, which matters more on tasks requiring precise multi-step reasoning.\n3. **Launch with llama.cpp’s server command.** Point llama.cpp’s server binary at your local GGUF files, set your context length (up to 262K tokens if your hardware and use case support it), and expose the local API endpoint.\n4. **Connect a front end or agent framework.** A local web UI (like llama.cpp’s built-in server GUI) or an agent framework can be wired to the local endpoint for testing prompts, coding tasks, or agentic workflows.\n\nBecause of the MoE architecture, only a fraction of the total parameters activate for any given token, so inference speed and compute demand track closer to a 3B active-parameter model even though the full 35B needs to be loaded into memory.\n\n## How much VRAM does Apex-35B need?\n\nIn testing, the Q8 quantized version of Apex-35B consumed just over 38GB of VRAM when served through llama.cpp on an H100 GPU with 80GB of VRAM. That’s a useful reference point: Q8 keeps nearly all of the original precision while cutting memory use roughly in half compared to BF16 full precision.\n\nThat 38GB figure doesn’t require an H100 specifically. Any GPU or combination of GPUs with similar or greater VRAM headroom (including rented cloud GPU instances) can run the same configuration. If 38GB is still out of reach, lower-bit quantizations (Q4_K_M, Q5_K_M, and similar) will reduce memory needs further, though it’s worth testing output quality against your specific use case since aggressive quantization can degrade performance on complex reasoning or math-heavy tasks.\n\nFor teams without local hardware, renting a GPU instance with enough VRAM is a common workaround, and sizing the rental to the chosen quantization level (BF16 vs Q8 vs lower-bit quants) is the key decision point before spinning up a server.\n\n## What makes Apex-35B’s training process unusual?\n\nThe notable part of Apex-35B isn’t just its architecture, it’s how Apex Intelligence built it. The team developed an internal system called Apex Research that automated parts of the model development pipeline: generating training data, running experiments, tuning the training recipe, and optimizing GPU kernels. In other words, AI systems were used to help build and refine Apex-35B itself, rather than every step being manually engineered by researchers.\n\nThis self-improving research loop is part of why the model is positioned as more than a simple fine-tune of Qwen3.6. The base model provides the architecture and pretraining, but the additional training and tuning work, partly automated, is what Apex Intelligence credits for the model’s gains over its base.\n\n## How does Apex-35B perform on real coding and reasoning tasks?\n\n*Enterprises build with AI on Remy.*\n\n- Self-hosted\n- SOC 2\n- HIPAA\n- Audit log\n- Role-based access\n\nHands-on testing against a real full-stack application (an operations dashboard for a solar panel installation company) showed Apex-35B capable of tracing a billing bug across two different screens of a codebase, a payback-period calculator showing unrealistic numbers and a tariff calculator producing an incorrect staircase-shaped cost chart. Without being told what the bug was, the model rewrote the relevant function, rebuilt the affected API service, added a regression test, and supplied verification evidence, all in roughly five and a half minutes.\n\nOn a messy real-world spreadsheet task (a year of online store orders with duplicate rows, inconsistent date formats, and malformed values), the model cleaned around 1,400 orders, logged every data issue it fixed, generated a chart, and wrote up three actionable findings without fabricating numbers. The output correctly flagged that electronics drove about half of revenue but also carried a high return rate, a plausible and well-supported business insight.\n\nA harder visual/creative coding test (a single HTML file simulating rain on a foggy window with merging droplets, wipeable fog, and lightning) was less convincing. The functional pieces were present, but the visual execution fell short of the prompt description, with thicker-than-intended rain trails and some rendering artifacts.\n\nOn a genetics probability reasoning problem requiring no code or tools, Apex-35B worked through multi-step Bayesian-style reasoning and landed on the correct probability figures after an extended thinking period, missing only a minor correction term on one sub-answer.\n\n## Is Apex-35B worth running locally?\n\nFor teams already comfortable self-hosting LLMs, Apex-35B is a reasonable candidate if your priority is agentic coding and structured reasoning tasks on consumer or prosumer-class hardware rather than high-end visual generation. Its MoE design keeps active compute low relative to its total parameter count, and the Apache 2.0 license removes usage restrictions that block some competing open models.\n\nPublished benchmark comparisons show Apex-35B outperforming its own Qwen3.6 base model and a similarly sized competitor on research-oriented tests, and even beating much larger models like GLM 5.3 and DeepSeek on select individual benchmarks. It does not lead across the board against bigger models, which is expected given the parameter gap, but its results relative to its size are competitive.\n\nIf your workloads lean toward debugging real codebases, cleaning messy data, or stepping through technical reasoning problems, it’s a solid pick. If you need strong visual or creative code generation, test it against your specific use case first since results there were less consistent.\n\n## Frequently Asked Questions\n\n### What base model is Apex-35B built on?\n\nApex-35B is fine-tuned from Qwen3.6-35B-A3B, a sparse mixture of experts model, and Apex Intelligence applied additional training and tuning on top of that base.\n\n### How many parameters does Apex-35B actually use per token?\n\nIt has 35 billion total parameters but activates only about 3 billion per token, thanks to its mixture of experts architecture, which keeps inference costs closer to a small dense model.\n\n### What quantization format should I use for Apex-35B?\n\nQ8 is a good starting point, balancing near-full precision with roughly half the VRAM footprint of BF16. Lower-bit quants like Q4 or Q5 variants reduce memory needs further but may affect quality on complex reasoning tasks.\n\n### How much context can Apex-35B handle?\n\nUp to 262,000 tokens, which supports large codebases, long documents, or extended multi-turn agent sessions without truncation.\n\n### Is Apex-35B free to use commercially?\n\nYes, it’s released under the Apache 2.0 license, which permits commercial use, modification, and redistribution without the restrictions found in some other open-weight model licenses.", "url": "https://wpnews.pro/news/how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide", "canonical_source": "https://www.mindstudio.ai/blog/apex-35b-local-install/", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 10:44:02.200576+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "ai-infrastructure", "machine-learning"], "entities": ["Apex Intelligence", "Apex-35B", "Qwen3.6-35B-A3B", "llama.cpp", "Hugging Face", "Apex Research", "H100", "GLM 5.3"], "also_reported_by": [], "takeaway": "Serving Apex-35B at Q8 in llama.cpp used just over 38GB of VRAM on an 80GB H100, roughly half of BF16, while the sparse MoE's ~3B active parameters keep compute demand near a small dense model.", "recommendation": {"advice": "Load Apex-35B as a Q8 GGUF in llama.cpp to fit just over 38GB of VRAM, or drop to Q4/Q5 quants for smaller GPUs, accepting some quality loss on multi-step reasoning.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide", "markdown": "https://wpnews.pro/news/how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide.md", "text": "https://wpnews.pro/news/how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide.txt", "jsonld": "https://wpnews.pro/news/how-to-run-apex-35b-locally-with-llama-cpp-a-setup-guide.jsonld"}}