Details
- Perplexity is open-sourcing Lily, its local inference engine built to power hybrid compute in Perplexity Computer.
- Lily is specialized for running the Qwen3.6-35B-A3B model on Apple silicon Macs so on-device compute does not slow down Computer tasks.
- Hybrid compute in Perplexity Computer splits work between frontier models in the cloud and a local model on the Mac, requiring fast, reliable on-device inference.
- Unlike general-purpose frameworks such as MLX-LM, Lily is purpose-built for this specific inference workload and treats prefill and decode as distinct phases with different performance constraints.
- Prefill handles many prompt tokens at once and reuses weights, while decode generates one token at a time with far less reuse, making memory traffic and bandwidth critical.
- Benchmarks on an M5 Max MacBook Pro show Qwen3.6-35B-A3B running in Lily with an average 1.23× higher prefill throughput and 1.35× higher decode throughput than MLX-LM, with output quality essentially unchanged.
- Lily treats Apple silicon as a distinct inference platform, mapping Qwen operations directly to its compute and memory architecture rather than relying on more generic runtime stacks.
- The engine exposes this optimized local performance to Perplexity Computer so hybrid tasks can route sensitive steps to the Mac while keeping overall latency competitive with pure cloud workflows.
- By open-sourcing Lily, Perplexity makes its Apple silicon optimizations available to developers and researchers working with Qwen3.6-35B-A3B and similar open-weight models.
- Lily is now available through Perplexity’s official release page linked from the announcement thread.
Impact
Open-sourcing Lily strengthens Perplexity’s hybrid compute strategy by turning its Apple silicon optimizations into shared infrastructure, narrowing the performance gap between local and cloud inference for large open-weight models. This can encourage wider adoption of on-device LLMs, support privacy-focused deployments, and pressure rival AI assistants to treat local inference as a first-class platform rather than an optional add-on.
