{"slug": "lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s", "title": "Lithos-metal: Qwen3.8-27B on 40-core M5 Max at 168 tokens/s", "summary": "Lithos AI released lithos-metal, an open-source inference engine that generates Metal megakernels for Apple silicon and runs Qwen3.8-27B at 168 tokens/s on a 40-core M5 Max with 48 GB of unified memory. The engine combines layer-wise mixer megakernels, GPU-resident generation control and DSpark speculative decoding, with DSpark proposing seven tokens that the target verifies alongside one anchor token. It installs via the Lithos Homebrew tap, serves Chat Completions, Responses and Anthropic Messages adapters plus Ollama's /api/chat on a default 32K context at http://127.0.0.1:8000, and ships launchers for opencode, claude, codex and hermes; the Qwen3.6-35B-A3B hybrid MoE integration is labeled experimental.", "body_md": "**lithos-metal is a fully open-source inference engine that generates Metal megakernels for Apple silicon.**\nIt brings low-latency LLM inference to your Mac, with local serving for coding agents and other applications.\nThe engine combines **layer-wise mixer megakernels**, **GPU-resident generation control**, and **DSpark\nspeculative decoding** to reduce execution overhead and make better use of each target-model pass.\n\n[Quick start](#quick-start) · [How it works](#how-it-works) · [Performance](#performance) · [Documentation](#documentation)\n\nUse an Apple silicon Mac with **macOS 26+**, a registered chip backend, and enough unified memory for the\ntarget model, draft model, and context. The Qwen3.8-27B and Qwen3.6-35B-A3B configurations have been exercised\non a **40-core M5 Max with 48 GB of unified memory**. See [hardware support](#models-and-hardware) for other backends.\n\nInstall the precompiled package from the [Lithos Homebrew tap](https://github.com/lithos-ai/homebrew-tap):\n\n```\nbrew install lithos-ai/tap/lithos-metal\nlithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4\n```\n\nThe server downloads the target and its matching **LithosAI NVFP4 DSpark head**, prepares cached weight packs,\nand selects the device's kernel recipes. Startup includes compilation and warmup. By default, DSpark proposes\nseven tokens, which the target verifies together with one anchor token. Use `--no-draft` for target-only\ngeneration or `--draft PATH_OR_HUB_ID` to select another DSpark checkpoint.\n\nThe default endpoint is `http://127.0.0.1:8000`, with a 32K context capacity. Use `--max-context` to adjust\ncapacity to the model and available memory. Run `lithos-metal serve --help` for all options.\n\nLeave the server running and launch an installed client from another terminal:\n\n```\nlithos-metal opencode\nlithos-metal claude\nlithos-metal codex\nlithos-metal hermes\n\n# Other OpenAI-compatible clients:\nlithos-metal env\nlithos-metal run -- your-client\n```\n\nThe launchers discover the served model and configure the child process without rewriting global configuration or changing client approval settings. Install the clients separately.\n\n```\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"nvidia/Qwen3.8-27B-NVFP4\",\"messages\":[{\"role\":\"user\",\"content\":\"Explain how a GPU megakernel works.\"}],\"max_completion_tokens\":256,\"stream\":true}'\n```\n\nThe server provides **Chat Completions, Responses, and Anthropic Messages** adapters with text generation,\ntool calls, and SSE streaming, plus Ollama's `/api/chat` for Ollama clients. One generation runs at a time.\nResponses is stateless, and tools execute in the client. See the [serving design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/serving.md) for\nprotocol support, caching, sampling, and API limits.\n\nThese target/draft pairs are selected automatically by the serving CLI. Run `lithos-metal models` to list them.\n\n| Target model | Automatic DSpark head | \n|---|---|\n| [Qwen3.8-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4) | [LithosAI/Qwen3.8-27B-DSpark-NVFP4](https://huggingface.co/LithosAI/Qwen3.8-27B-DSpark-NVFP4) | \n| [Qwen3.8-27B-4bit](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit) (affine 4-bit) | [LithosAI/Qwen3.8-27B-DSpark-NVFP4](https://huggingface.co/LithosAI/Qwen3.8-27B-DSpark-NVFP4) | \n| [Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4) | [LithosAI/Qwen3.6-35B-A3B-DSpark-NVFP4](https://huggingface.co/LithosAI/Qwen3.6-35B-A3B-DSpark-NVFP4) | \n\nThe 35B hybrid MoE integration is experimental; its [adapter design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/models.md#qwen-hybrid-moe-model) describes\nthe current numerical qualification. The repository also includes Qwen and Llama-compatible model packages;\nsee the [model adapter design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/models.md) for supported structures and checkpoint conventions. Matching a model family does not imply support for every checkpoint.\n\n| Chip backend | Current status | \n|---|---|\n| M5 Max, 40 GPU cores | Measured target/draft recipes for the large Qwen models, including mixer megakernels | \n| M5 Pro, 20 GPU cores | Measured kernels and configurations for smaller models | \n| M3 Pro, 18 GPU cores | Probe-derived configuration | \n| M3 Max, 30 GPU cores | Unmeasured native fallback; DSpark serving fits through 20480 context on 36 GB | \n| M3 Ultra, 60 GPU cores | Measured profile with the Apple9 matrix tile | \n| M4 Pro, 20 GPU cores | Measured profile with the Apple9 matrix tile | \n| M4 Pro, 16 GPU cores | Unmeasured native fallback | \n| M4 Max, 32 or 40 GPU cores | Unmeasured native fallback | \n| M5 Max, 32 GPU cores | Independent, unmeasured native fallback | \n\nBackend selection uses the chip name, GPU family, and core count. Unmeasured backends have no automatic\nmegakernel fusion and keep the tensor accelerator disabled. See [chip backends](https://github.com/lithos-ai/lithos-metal/blob/main/monolith/backends/metal/README.md)\nfor configuration ownership and validation status.\n\nFollowing the task-based execution model explored by [Mirage Persistent Kernel (MPK)](https://arxiv.org/abs/2512.22219),\nlithos-metal decomposes operators into matrix tiles, attention partitions, and recurrent-state slices.\nThe compiler records their dependencies and generates specialized Metal code. Within a megakernel, persistent\nworkers—Metal threadgroups—execute a sequence of tasks, reusing local intermediates where possible and\nsynchronizing values shared between workers.\n\nlithos-metal uses **layer-wise fusion rather than one whole-model megakernel**. In selected configurations,\nthe Gated DeltaNet (GDN) or full-attention mixer within a layer becomes a single megakernel; the layer's feed-forward network\n(MLP) remains separate. Qwen3.8-27B is the case study for these two schedules:\n\n| Region | Megakernel technique | \n|---|---|\n| **GDN mixer** | Partition the recurrent state by value columns, retain each slice across input tokens, and coordinate projections, recurrence, gating, and normalization through their dependencies. | \n| **Full-attention mixer** | Partition the K/V context, accumulate online-softmax statistics locally, and merge partial results before output projection. | \n| **DSpark mixer** | Apply the same task compiler to block QKV projection, attention, reduction, output projection, and residual/normalization work. Fusion is selected by the draft recipe. | \n\nThe runtime connects megakernels and conventional kernels through a **pre-encoded Metal indirect command buffer**.\nGPU-resident state tracks token positions, accepted tokens, recurrent-state commit/rollback, and stopping,\nwith minimal CPU involvement. Bounded dispatches provide synchronization boundaries and opportunities for GPU sharing.\n\n- **Data reuse:** retain normalization statistics, attention accumulators, and state slices within a worker\nto reduce intermediate writes, rereads, and layout conversions.\n- **Matrix acceleration:** on M5, Metal Performance Primitives expose GPU neural accelerators through inline\ntensor operations, allowing matrix arithmetic and surrounding custom operations to share a shader.\n- **Dynamic memory:** Apple GPUs from M3 onward allocate on-chip storage dynamically. The compiler tunes worker\ncount, tile geometry, and intermediate lifetimes together to balance local reuse and concurrency.\n\nFusion decisions use complete-region latency: the data movement saved must justify task scheduling and\nsynchronization costs. The [Apple GPU execution model](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/apple-gpu.md) and\n[architecture](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/design.md) explain the execution constraints and tradeoffs behind these choices.\n\n[DSpark](https://arxiv.org/abs/2607.05147) combines a small target-conditioned block drafter with a sequential\nMarkov head that models dependencies between proposals. The target verifies the block and commits the accepted\nprefix. When several proposals are accepted, one target pass produces multiple output tokens, amortizing its\nweight reads. lithos-metal also optimizes the draft itself with matrix kernels and selected mixer megakernels\nto reduce the overhead of proposing each block. See the [DSpark design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/speculative-decoding.md)\nand [draft mixer design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/mixers.md#dspark-draft-mixers).\n\nThe Qwen3.8-27B-NVFP4 case study uses a **40-core M5 Max with 48 GB of unified memory**.\nA target-compute projection reaches approximately **168 tokens/s at 128-token context** and\n**127 tokens/s at 32K context**. The [backend comparison](https://github.com/lithos-ai/lithos-metal/blob/46b2bc4cda57826c072d94afe90a58633aaeb6d9/docs/research/m5max-27b-backend-verification.md)\nincludes MLX, Ollama, and vLLM-Metal, with source data and reproduction details.\n\nThese figures assume six accepted tokens per eight-row target pass and exclude drafting, acceptance processing,\nand rewind. Backend timing scopes differ, so they are not measured end-to-end generation rates or a direct\nend-to-end speedup comparison. The [GDN](https://github.com/lithos-ai/lithos-metal/blob/46b2bc4cda57826c072d94afe90a58633aaeb6d9/docs/research/m5max-gdn-mixer-optimization.md),\n[full-attention](https://github.com/lithos-ai/lithos-metal/blob/46b2bc4cda57826c072d94afe90a58633aaeb6d9/docs/research/m5max-27b-attention-optimization.md), and\n[DSpark](https://github.com/lithos-ai/lithos-metal/blob/46b2bc4cda57826c072d94afe90a58633aaeb6d9/docs/research/m5max-27b-dspark-refinement.md) studies report the corresponding kernel and round measurements.\n\nFor a source build, install Xcode Command Line Tools and Python 3.12, then:\n\n```\ngit clone https://github.com/lithos-ai/lithos-metal.git\ncd lithos-metal\npython3.12 -m venv .venv\nsource .venv/bin/activate\npip install '.[serve]'\nlithos-metal --version\n```\n\nThe package build compiles the native runtime and bundles the Metal sources and chip recipes.\nUse `pip install -e '.[serve,dev]'` for an editable development install. The Python import namespace remains\n`monolith` for compatibility. See [CONTRIBUTING.md](https://github.com/lithos-ai/lithos-metal/blob/main/CONTRIBUTING.md) for development and release instructions,\nand [extension contracts](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/extensions.md) for adding models,\nquantization formats, operations, drafters, or chip backends.\n\nInput-normalization fusion is enabled by default and changes BF16 rounding. Use `--no-commute-norm` with the\ngeneration CLI or `Session(..., commute_norm=False)` to disable it; the\n[normalization design](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/mixers.md#input-normalization-fusion) describes the numerical contract.\n\nThe [design documentation](https://github.com/lithos-ai/lithos-metal/blob/main/docs/README.md) describes the current implementation and its contracts.\n\n| Design | Contents | \n|---|---|\n| [Architecture](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/design.md) | Compiler, GPU-resident runtime, state, and module boundaries | \n| [Apple GPU execution](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/apple-gpu.md) | Workers, memory, matrix operations, and synchronization | \n| [Mixer megakernels](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/mixers.md) | GDN, full attention, draft mixers, and normalization fusion | \n| [Speculative decoding](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/speculative-decoding.md) | DSpark, verification policy, and state commit/rollback | \n| [Serving](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/serving.md) | Protocol adapters, session setup, prefix caching, and streaming | \n| [Model adapters](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/models.md) | Checkpoint structures and numerical conventions | \n| [Extension contracts](https://github.com/lithos-ai/lithos-metal/blob/main/docs/design/extensions.md) | Models, formats, operations, drafters, and chip backends | \n\nlithos-metal is licensed under [Apache 2.0](https://github.com/lithos-ai/lithos-metal/blob/main/LICENSE). It is a standalone engine; it does not depend on the\nMPK/Mirage runtime. Code adapted from MPK/Mirage, MLX, llama.cpp, and other projects retains its license headers\nand attribution in [third_party/NOTICE](https://github.com/lithos-ai/lithos-metal/blob/main/third_party/NOTICE).", "url": "https://wpnews.pro/news/lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s", "canonical_source": "https://github.com/lithos-ai/lithos-metal", "published_at": "2026-10-17 04:06:23+00:00", "updated_at": "2026-10-17 04:29:11.475128+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "ai-agents"], "entities": ["Lithos AI", "lithos-metal", "Qwen3.8-27B", "Qwen3.6-35B-A3B", "Apple silicon", "M5 Max", "DSpark", "Homebrew"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s", "markdown": "https://wpnews.pro/news/lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s.md", "text": "https://wpnews.pro/news/lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s.txt", "jsonld": "https://wpnews.pro/news/lithos-metal-qwen3-8-27b-on-40-core-m5-max-at-168-tokens-s.jsonld"}}