Details
- Perplexity has released new research detailing the state-of-the-art serving infrastructure behind its embedding and ranking models.
- The system embeds both user queries and documents into a single vector space, enabling nearest-neighbor search over embeddings for relevance ranking.
- Perplexity distinguishes between bulk batch embedding for indexing and scoring, which is optimized for throughput, and per-query online embedding for live search, which is optimized for low latency.
- The serving stack is organized into three main components: Ivy as the Rust HTTP gateway, Tulip as the Rust gRPC inference server, and ROSE as the Python-based model engine.
- Ivy handles CPU-side tasks such as parsing, tokenization, templating, and splitting large batches, then forwards requests via gRPC to Tulip.
- Tulip sits between Ivy and the ROSE engine, aggregating incoming requests, batching them, and dispatching them to GPUs, with performance tuned around token count rather than query count for small embedding models.
- Tulip reduces latency and improves throughput using CUDA graphs to pre-record GPU work for single-call execution and LazyTensors to track results asynchronously so CPU and GPU can operate in parallel.
- ROSE serves as the unified model engine, reusing kernels for both large language models and embedding models while skipping the KV cache for embeddings and using ragged attention in place of paged attention.
- ROSE supports multiple attention backends, allowing kernel selection based on model architecture and sequence length to optimize performance across workloads.
- By combining Ivy, Tulip, and ROSE, Perplexity reports lower latency and higher throughput for both online and batch embedding workloads, achieving faster search at reduced cost compared with off-the-shelf serving solutions.
Impact
By formalizing and publishing the architecture behind Ivy, Tulip, and ROSE, Perplexity signals that custom, vertically integrated inference stacks are becoming a competitive necessity for high-volume AI search and ranking workloads. This approach can narrow cost and latency gaps versus cloud-native, off-the-shelf serving options used by rivals, and may influence other AI providers to invest in specialized batching, CUDA graph, and attention-backend optimizations for both embeddings and LLMs.
