{"slug": "show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone", "title": "Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone", "summary": "Swiftlet, a Swift + Metal runtime for Qwen3-Next and Qwen3.5/3.6 MoE hybrid models, runs an 80B Qwen in 4.3 GB of RAM on a Mac and a 35B on an iPhone 17 in about 2.5 GB of RAM at about 1 token per second, marking the first time a model of this class has run natively on a phone. The runtime keeps only the dense core resident and streams routed Mixture-of-Experts weights from storage, with both models generating validated output. The project is open source on GitHub, and the 35B model is available in the Priv AI app on the App Store.", "body_md": "**Run 35B and 80B Qwen models on ordinary Apple devices, including iPhones.**\n\nSwiftlet is a Swift + Metal runtime for the Qwen3-Next and Qwen3.5/3.6 MoE hybrid model family. It keeps only the small dense core of a model resident in memory and streams the routed Mixture-of-Experts weights from storage on demand. The result:\n\n| Model | Disk | Peak RAM | Decode speed (M5 Mac) |\n|---|---|---|---|\n|\n\n[Qwen3-Next-80B-A3B, 4-bit](https://huggingface.co/Leonickson/Qwen3-Next-80B-A3B-qpack)The 35B also runs on an iPhone 17 in about 2.5 GB of RAM, at about 1 tok/s today. As far as we know, that is the first time a model of this class has run natively on a phone.\n\nStatus: working end to end. Both models generate correct, validated output. The current focus is kernel speed (the decode loop is dispatch bound, not IO bound, so there is clear headroom). One expectation to set honestly: only about 3B parameters are active per token, so these models chat and write like large models but recall facts like small ones.\n\n```\ngit clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet\nswift build -c release\n\n# Download the 35B container from Hugging Face (resumable):\n.build/release/swiftlet-repack \\\n  --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \\\n  --output ~/models/qwen3.6-35b.qpack\n\n# Or the 80B (42 GB on disk, still only ~4.3 GB of RAM):\n.build/release/swiftlet-repack \\\n  --from-hf Leonickson/Qwen3-Next-80B-A3B-qpack \\\n  --output ~/models/qwen3-next-80b.qpack\n\n# Chat (applies the model chat template, disables the reasoning block,\n# keeps conversation state so follow-ups prefill only the new turn):\n.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \\\n  \"Who wrote One Hundred Years of Solitude?\" \"What language did he write it in?\"\n\n# One-shot generation with stats:\n.build/release/swiftlet generate ~/models/qwen3.6-35b.qpack \\\n  --gpu --chat --prompt \"Explain expert streaming in one paragraph.\"\n\n# OpenAI-compatible server (loopback only):\n.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080\n```\n\nThe same command also repacks raw MLX checkpoints\n(`--from-hf mlx-community/...`\n\nor `--source /path/to/checkpoint`\n\n).\n\nRequirements: Apple Silicon, macOS 14+ or iOS 17+, free SSD space for the container (18 GB for the 35B, 42 GB for the 80B).\n\nThe 35B runs on iPhone inside\n** Priv AI on the App Store**:\nopen Settings, then Experimental Models, and download the model. It streams\nfrom storage and chats on-device with no server involved.\n\nThe Experimental Models feature ships in the newest app version, which is\nstill in App Store review, so it may not appear for a couple of days. If you\nwant the phone experience today, build the app from source: the app is open\nsource at [leonickson1/localLLM](https://github.com/leonickson1/localLLM).\nClone this repo next to it as `swiftlet`\n\n, open the Xcode project, and run it\non your iPhone.\n\nThese models activate only about 3B of their parameters per token. Each layer routes every token to 10 of 512 experts (80B) or 8 of 256 (35B). Swiftlet:\n\n- keeps the dense weights resident: attention, DeltaNet projections, routers, shared experts, embeddings. About 1.3 GB (35B) or 2.5 GB (80B) at 4-bit;\n- repacks the tens of thousands of routed experts into fixed-stride blobs in\na\n`.qpack`\n\ncontainer, so fetching one expert is exactly one`pread`\n\nfrom SSD, no mmap and no page-cache thrash; - caches hot experts in a bounded pool with LFU plus recency eviction. Cache size barely affects speed (measured 43 to 70 percent hit rates at the same throughput), because Apple SSDs absorb the misses;\n- runs the whole forward pass on Metal with runtime-compiled shaders, so no Metal toolchain is needed at build time and the same code ships on iOS.\n\n75 percent of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so there is no growing KV cache for those layers at any context length.\n\nSwiftlet is a library first:\n\n**The Swift package.** Add SwiftletCore to any macOS or iOS app and use`SwiftletSession`\n\nfor chat with streaming deltas, conversation caching, sampling with repetition control, and memory-pressure handling built in.**The CLI.**`swiftlet chat`\n\nand`swiftlet generate`\n\nfor local use and benchmarking,`swiftlet-repack`\n\nto build containers from MLX checkpoints (including streaming straight from Hugging Face with resume).**The server.**`swiftlet-server`\n\nspeaks the OpenAI chat-completions API on loopback, so any chat UI that talks to OpenAI-compatible endpoints can use a streamed local model.**An app.**[Priv AI](https://apps.apple.com/us/app/priv-ai/id6765706001)on iOS embeds SwiftletCore as its streamed-model engine. End users tap Download and chat. Nothing here is terminal-only. The app itself is open source at[leonickson1/localLLM](https://github.com/leonickson1/localLLM)if you want to build it yourself (clone this repo next to it as`swiftlet`\n\n).\n\nEvery layer of the forward pass (Gated DeltaNet recurrence, gated GQA attention, sparse MoE routing) is validated against mlx-lm reference implementations with per-layer fixtures, in f32 and int4 quantized form. Incremental decoding is verified against whole-sequence processing. Metal kernels are tested against the exact CPU reference, and the fast and scalar GPU kernels are verified to produce identical outputs. Containers are byte-verifiable against their source checkpoints. Streaming placement never changes model semantics: an expert answers identically from cache or disk.\n\n```\nswift test\n```\n\n[TurboFieldfare](https://github.com/drumih/turbo-fieldfare) proved the\nexpert-streaming thesis for Gemma on Macs, and Swiftlet adopts several of its\npublished design lessons with gratitude: stream experts with `pread`\n\ninto a\nbounded slot pool instead of mmap, evict with LFU plus recency, pack experts\nat fixed stride so one fetch is one read, install by routing downloaded bytes\nstraight into their final container positions, and compile shaders at\nruntime.\n\nEverything else is built here, from scratch, in about 10k lines of Swift and Metal written against mlx-lm references rather than TurboFieldfare code:\n\n- support for a different model family with a fundamentally different architecture: the Qwen hybrid stack with Gated DeltaNet linear attention, gated GQA, and high-sparsity MoE with a shared expert (TurboFieldfare runs Gemma, a classical dense transformer);\n- MLX affine int4/int8 group quantization compute in Metal, byte-addressed kernels with 64-bit offsets for multi-gigabyte shards, a cooperative simdgroup GEMV fast path, and explicit hazard management;\n- a validated CPU reference implementation and the fixture infrastructure that gates every kernel change;\n- the\n`.qpack`\n\ncontainer and repacker, the resumable Hugging Face streaming installer with stall recovery, and download cancellation; - the chat session layer: template handling for thinking and non-thinking Qwen variants, sampling with presence and frequency penalties and minimum-length and sentence-completion stopping, conversation caching with delta prefill, and iOS memory-pressure coordination;\n- iPhone support end to end, including the app engine integration.\n\n[colibrì](https://github.com/JustVugg/colibri) informed the caching and\nplacement policy thinking. mlx-lm is the correctness reference throughout.\n\nSwiftlet was built in collaboration with\n[Claude Code](https://claude.com/claude-code).\n\nApache 2.0. Model weights are downloaded separately and remain governed by their own terms (Qwen models: Apache 2.0). See THIRD_PARTY_NOTICES.md.", "url": "https://wpnews.pro/news/show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone", "canonical_source": "https://github.com/leonickson1/Swiftlet", "published_at": "2026-08-03 16:54:15+00:00", "updated_at": "2026-08-04 03:14:52.559121+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products", "developer-tools"], "entities": ["Swiftlet", "Qwen3-Next-80B-A3B", "Qwen3.6-35B-A3B", "Apple", "Priv AI", "GitHub", "Hugging Face", "Metal"], "alternates": {"html": "https://wpnews.pro/news/show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone", "markdown": "https://wpnews.pro/news/show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone.md", "text": "https://wpnews.pro/news/show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone.txt", "jsonld": "https://wpnews.pro/news/show-hn-run-an-80b-qwen-in-4-3-gb-of-ram-on-a-mac-and-a-35b-on-an-iphone.jsonld"}}