Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

MarkTechPost · 1 month ago · 22

Perplexity Engineering published Fast Embeddings on GPUs, detailing the GPU serving stack (Ivy, Tulip, and ROSE) that powers pplx-embed and related Perplexity search and ranking models. It reports GPU saturation at around 512 tokens on a sub-billion-parameter embedding model. As a result, embedding serving reuses the LLM prefill/decode kernels and uses CUDA graph + lazy capture and a LazyTensor async path so latency scales mainly with token count rather than sequence count.