Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon

Perplexity has open sourced Lily, the local inference engine behind Hybrid Compute in Perplexity Computer. It is built for one model, Qwen3.6-35B-A3B, and one hardware family: Apple silicon Macs. That narrow scope is the whole point. Lily is a single-process runtime in Rust, with hand-written Metal kernels doing the heavy lifting and an OpenAI-compatible chat-completions API streaming tokens back out. PyTorch and MLX are not in the execution path.

That design choice is not subtle. Perplexity’s own description of the shipping Hybrid Compute product says macOS 15+ is required, with 24 GB as the minimum and 32 GB recommended for best results. The 4-bit checkpoint weighs 19.4 GB, so memory headroom matters. There is also a public standalone demo in the pplx-garden repository, which makes the engine more than a paper exercise.

The reason Lily can be fast comes down to how Qwen3.6-35B-A3B behaves. It stores 35B parameters but activates roughly 3B per token. A router scores 256 experts and selects eight, plus one shared expert that always runs. The model also mixes 10 full-attention layers, using grouped-query attention, with 30 Gated DeltaNet layers. That produces three very different workloads: expert routing, growing KV-cache attention, and fixed-size recurrence. Lily attacks each one directly instead of trying to stay general.

In prefill, the big trick is to avoid dragging expanded weights through memory. The checkpoint uses groupwise affine 4-bit quantization, and Lily reconstructs the weights tile by tile inside the grouped GEMM, keeping the results in threadgroup memory and accumulating in FP32. Perplexity says that fusion improved end-to-end prefill by 77.4% at a 512-token prompt. Keeping routing, prefix scan, scatter, and block mapping inside one GPU command buffer added another 89% at 512 tokens by cutting CPU synchronization inside each MoE layer.

Decode is about moving as little as possible for each token. One recorded step used 795 kernels across 555 sequential stages, so Lily records dependencies in a concurrent Metal pass so independent work can overlap. It writes the selected token straight into the next step’s GPU-resident input slot, skipping a per-token CPU round trip, and fuses four kernel chains so intermediates stay in registers. On a 40-core, 128 GB M5 Max, Lily averaged 4,156 prefill tokens/s and 170.0 decode tokens/s, ahead of MLX-LM’s 3,388 and 126.4 across ten lengths from 256 to 128K tokens. A teacher-forced check across 192 positions showed perplexity just 0.04% higher, with the same top-ranked token 96.35% of the time.