{"slug": "perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac", "title": "Perplexity Lily: 1.35x Faster Local AI Than MLX on Mac", "summary": "Perplexity has open-sourced Lily, a Rust and Metal local inference engine that outperforms Apple's MLX framework by 1.35x on decode throughput for Qwen3.6-35B-A3B on Mac, with prefill speeds of 4,156 tokens/s versus MLX-LM's 3,388 tokens/s and decode speeds of 170.0 tokens/s versus 126.4 tokens/s. The code is available in the pplx-garden repository, and Perplexity also released Privacy Gate, an on-device PII classifier, to support its Hybrid Compute architecture.", "body_md": "Perplexity has open-sourced Lily, the local inference engine powering its Mac Hybrid Compute feature — a Rust and Metal runtime that beats Apple’s MLX framework by 1.35x on decode throughput for Qwen3.6-35B-A3B, with no PyTorch or MLX anywhere in its execution path. The code is live in the [pplx-garden repository](https://github.com/perplexityai/pplx-garden) today.\n\n## What Lily Actually Is\n\nLily is not a framework. It is not trying to be the next llama.cpp. It is a single-process runtime built for one model on one hardware family, and that constraint is the whole point.\n\nThe stack is three layers deep. A Rust runtime loads the model checkpoint and drives the generation loop — chosen because Rust gives memory safety without a garbage collector, which matters when you are managing large model weights across long generation sessions. Below that, hand-written Metal kernels execute the model directly on the GPU, bypassing any intermediate abstraction layer. On top, an OpenAI-compatible chat-completions API streams tokens to whatever client is calling it.\n\nNo MLX. No PyTorch. No framework overhead. That is the bet Perplexity made, and it paid off in tokens per second.\n\n## The Numbers\n\nBenchmarked on a 40-core, 128 GB M5 Max MacBook Pro at batch 1, across ten prompt lengths from 256 to 128K tokens:\n\n| Metric | Lily | MLX-LM | Speedup | \n|---|---|---|---|\n| Prefill (tokens/s) | 4,156 | 3,388 | 1.23x | \n| Decode (tokens/s) | 170.0 | 126.4 | 1.35x | \n\nOutput quality was essentially unchanged: perplexity delta of 0.04%, and Lily selected the same top-ranked token as MLX-LM at 96.35% of positions. The speed gain is real; the quality loss is noise.\n\n## Why MLX Falls Behind\n\nMLX is a general-purpose array framework. That generality is also its ceiling. Lazy evaluation and framework-level scheduling introduce overhead that does not exist in a dedicated inference runtime. When Lily handles prefill and decode as separate workloads — mapping Qwen’s specific operations directly to Apple Silicon’s compute and memory architecture — it is exploiting knowledge that MLX, by design, cannot assume.\n\nThis is not a knock on MLX. For running a variety of models experimentally, it is the right tool. But for production serving of a single model, you are paying a framework tax in every token.\n\n## Hybrid Compute and Privacy Gate\n\nLily exists because of [Perplexity’s Hybrid Compute architecture](https://www.marktechpost.com/2026/09/01/perplexity-releases-hybrid-compute-on-mac-cloud-agents-orchestrate-down-to-a-local-model-gated-on-device/), which went live for the Mac app on September 1. The idea: frontier models in the cloud handle planning, reasoning, and web search. The local model handles anything that should never leave your machine.\n\nAlongside Lily, Perplexity also open-sourced Privacy Gate — a small on-device PII classifier that inspects outbound data before it reaches any cloud endpoint. It applies one of four outcomes: keep data local, mask sensitive spans, refuse the action, or ask for explicit user consent. The result is that a prompt like “summarize my bloodwork” can stay entirely on-device without the user needing to manually audit what gets sent where.\n\nOpen-sourcing the inference engine and the privacy classifier together is a deliberate signal: trust in local AI requires transparency about both what runs fast and what stays private.\n\n## How to Try Lily Now\n\nThe [pplx-garden repository](https://github.com/perplexityai/pplx-garden) includes a standalone Lily demo with a minimal OpenAI-compatible HTTP API for greedy text generation. You do not need the Perplexity Mac app. You do need an Apple Silicon Mac with enough unified memory to load a 35B-parameter model — plan for at least 64 GB for comfortable operation.\n\nThe OpenAI-compatible endpoint means existing tooling that already calls a cloud API can point at Lily instead with no code changes beyond the base URL.\n\n## The Specialist Argument\n\nLily is interesting precisely because it refuses to generalize. The trend in open-source inference has been toward universal runtimes — llama.cpp runs hundreds of models, Ollama wraps that into a convenient interface, MLX handles the Apple Silicon side. Each layer of generality comes with a cost.\n\nWhat Lily demonstrates is that when you know the exact model and the exact hardware, you can eliminate that cost entirely. The [Perplexity engineering blog](https://www.perplexity.ai/hub/blog/optimizing-on-device-inference-for-apple-silicon) frames this as “on-device compute shouldn’t bottleneck Computer tasks” — meaning Lily’s job is to be fast enough that the local model never becomes the rate-limiting step in a hybrid workflow.\n\nExpect more of this. As AI companies operationalize local inference at scale, the specialist runtime — tuned for one model, one architecture — will become a standard artifact alongside the model weights themselves. Lily is an early, well-executed example of that pattern. The [benchmark data backs it up](https://alphasignal.ai/news/perplexity-s-lily-beats-mlx-lm-by-1-35x-running-qwen3-6-on-apple-silicon).", "url": "https://wpnews.pro/news/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac", "canonical_source": "https://byteiota.com/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac/", "published_at": "2026-09-08 03:09:11+00:00", "updated_at": "2026-09-08 03:31:07.054146+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-products", "ai-tools"], "entities": ["Perplexity", "Lily", "Apple", "MLX", "Qwen3.6-35B-A3B", "Privacy Gate", "pplx-garden repository", "M5 Max MacBook Pro"], "alternates": {"html": "https://wpnews.pro/news/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac", "markdown": "https://wpnews.pro/news/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac.md", "text": "https://wpnews.pro/news/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac.txt", "jsonld": "https://wpnews.pro/news/perplexity-lily-1-35x-faster-local-ai-than-mlx-on-mac.jsonld"}}