TL;DR — Key Takeaways
- Perplexity’s Lily runtime brings local AI inference to Apple silicon, allowing users to process local files and data without sending every step to the cloud.
- Lily works as part of Perplexity’s Hybrid Compute approach, pairing cloud-based frontier models with local models running directly on a Mac.
- The runtime uses custom Metal kernels rather than MLX or PyTorch, optimizing execution specifically for Qwen-based models and Apple hardware.
AI search company Perplexity has released a runtime engine, called Lily, that allows macOS users to run foundational models against their own files and processes much more quickly–while using fewer cloud tokens and better guarding privacy–than off the entire job to the cloud.
Lily is a local inference engine that moves AI model calculations to Apple silicon. In this setup, the cloud-based frontier models do the high-level reasoning and planning, web searches, and artifact creation, while the local models on the machine work with the local data, such as client records, spreadsheets and legal files.
Currently, Lily works alongside Anthropic’s Claude 3.5 Sonnet or Opus, OpenAI’s GPT-4o or GPT-5, or Google Gemini. The default local model is Qwen3.6-35B-A3B, though users can substitute Gemma 4 E4B or Perplexity’s version of Qwen customized for the Mac.
Lily works with Perplexity’s Hybrid Compute on Apple silicon, a framework for orchestrating a task between cloud frontier models and a local model (the current default) on the Mac.
Local is Key
Local inference is needed to process prompts in a timely fashion, asserted Perplexity engineers in a blog post explaining the technology.
Apple’s GPUs, while not as powerful as NVIDIA’s, feature a unified memory architecture that provides enough shared bandwidth to keep large models resident without copying data back and forth.
And Apple has its own inference stack for its hardware: The MLX ML framework for Apple silicon and MLX-LM for managing language models. Together, MLX and MLX-LM provide an off-the-shelf, general-purpose stack for local LLM inference.
Lily doesn’t use MLX, however. Instead, it uses custom Metal kernels to run Qwen-specific instructions. Metal is Apple’s library and associated API for GPU acceleration.
Nor does Lily use PyTorch, the researchers pointed out, which is better suited for researchers building custom model layers and exacts a heavy performance penalty in the process.
Lily’s runtime, written in Rust, manages the session, running it as a single process, maintaining state and LLM checkpoints. It uses an OpenAI-compatible chat-completions API to field requests and stream tokens.
During processing, Apple’s Neural Accelerators handle the heavy matrix math during the prompt prefill phase, while the GPU’s vector ALUs manage the token-by-token decode phase.
The Evolving World of Inference Engines
Lily is competing with a variety of inference engines designed to run on the user’s own hardware, rather than in the cloud. The C/C++ llama.cpp runtime (and its user-friendly Ollama wrapper) can run on most hardware with CPUs or GPUs. Microsoft’s ONNX Runtime GenAI runs on Windows PCs and leverages the DirectX API for GPU acceleration.
For its own hardware, NVIDIA offers the TensorRT-LLM C++ inference engine, and Meta developed the ExecuTorch lightweight runtime to run PyTorch-trained models on mobile and edge devices. What gives Lily an edge is its architectural specificity. “Lily’s narrower scope allows it to coordinate … around Qwen’s exact architecture and dimensions,” the Perplexity engineers write.
One approach is to match the GPU path to the inference phase: use matrix execution when the prefill phase can reuse weights across multiple rows and use vector execution for decode processes that run one row at a time.
Minimizing data movement was also a priority. Weights are compressed until they are needed. Expert work is routed to the appropriate sub-networks, or “experts,” without unnecessarily round-tripping it back to the CPU. Qwen’s structure is mapped onto the GPU to minimize memory reformatting. Queries are grouped to reuse the KV cache data wherever possible.
And likewise, Metal compute kernels are shaped to the workload. In each phase, tile sizes, execution layouts, and attention paths are shaped according to factors such as available row count, the operation’s dimensions, and the context length.
To try Lily, you can use any Apple computer with macOS 15+ and 24GB of unified memory (though 32GB would be much cooler).