Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

Perplexity Engineering has published a look inside the machinery behind pplx-embed, the embedding service used across Perplexity Search, Computer and the API Platform. The focus isn’t the model itself. It’s everything around it: how requests get split, scheduled, graphed and pushed through GPUs without wasting time on launch overhead.

The company says embedding serving really comes in two flavors. Batch jobs show up when Perplexity is building or refreshing its vector database, so throughput matters most. Online embedding is the opposite: a short query arrives and has to be turned into vectors quickly. Scoring sits between those poles, ranking document batches after vector search. Instead of building a separate engine for that mix, Perplexity reuses the prefill and decode kernels from its LLM stack, arguing that small embedding Transformers line up neatly with those same compute patterns.

That pipeline has three internal pieces. Ivy is the Rust HTTP gateway, where JSON parsing, tokenization, templating and batch splitting happen before requests are translated into a custom gRPC protocol. Tulip is the gRPC scheduler, built with Rust, tokio and tonic, and it decides when to batch and dispatch work. ROSE, short for Runtime-Optimized Serving Engine, is the inference layer itself. It’s mostly Python, defines the kernels and model code, manages CUDA graphs and hands Tulip a step() function to run.

Tulip’s scheduler is intentionally plain. It uses first-come, first-served ordering while requests build up, because Perplexity says the dense layers dominate the cost for the small embedding models it serves. In that setup, latency tracks token count more than sequence count, and once a batch fills the GPU, around 512 tokens on a sub-billion-parameter model, stuffing in more sequences does not buy much. That is a very unglamorous rule, which is usually a sign it’s doing something useful.

The harder work is on the GPU launch side. Perplexity says CPU-side kernel launches can overwhelm the actual GPU work on small batches, so it captures whole-model CUDA graphs for its embedding models. Because graphs have to be built per configuration, token counts are padded into 64 or 256 token buckets, which creates thousands of graphs and takes minutes per model to capture. The workaround is lazy capture: the first request warms a configuration, the second triggers graph capture and replay. Perplexity pairs that with LazyTensor, which lets Tulip return before device work finishes and lets Rust tasks overlap CPU prep for the next batch with GPU execution for the current one.

On the kernel side, ROSE supports FlashInfer 2, FlashInfer 3 and FlashAttention 4 for ragged inputs. Perplexity says FlashAttention 4 is usually faster, but FlashInfer 3 wins on some Qwen-based models at very long sequence lengths, so backend choice is made case by case. For embeddings, ROSE skips a KV cache entirely and uses ragged attention variants to avoid padding. Perplexity benchmarks this stack against vLLM v0.22.0 in BF16, using real weights and eval-derived inputs, with warmup runs checking cosine similarity divergence stays within 0.1%.