Details
- Perplexity introduces Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard to evaluate retrieval in agentic RAG systems.
- The benchmark measures how embedding models perform on large-scale web search using agent-reformulated queries derived from real production usage.
- Q2D-Web combines a 190 million document web corpus with 69,721 agent-reformulated queries, averaging 99.6 positive relevance judgments per query.
- The dataset is built from 23,000 PII-free production searches across ten languages and many domains, including programming, law, health, science, finance, consumer goods, travel, entertainment, and local information.
- For each user request, agents generate primary and support queries that are independently evaluated, each with its own relevance judgments.
- The corpus includes the top 5,000 production retrieval results per query, deduplicated via MinHash-LSH so every document plausibly matches at least one query, including hard distractors that miss a required date, entity, or version.
- Q2D-Web defines three relevance sets: agent citations, production web rankings, and a combined set expanded with LLM-based judgments, reducing false negatives and dependence on a single labeling pipeline.
- Reciprocal rank fusion subsampling preserves full-corpus rankings on Combined Recall@1000 while using only 31.7% of documents, cutting evaluation for pplx-embed-v1-4b from about 4,608 to roughly 1,500 H200 GPU-hours.
- Perplexity evaluated 13 retrieval models across the three relevance sets, using Recall@1000 as the primary metric; pplx-embed-v1-4b leads Web Ranking (65.73) and Combined (69.11), while Nemotron-3-Embed-8B leads the Citation set (61.68).
- Developers can request Q2D-Web evaluations by submitting publicly available Hugging Face retrieval models via Perplexity’s evaluation form, and a full technical report on methodology and findings is provided.
Impact
Q2D-Web strengthens the emerging ecosystem of robust benchmarks for retrieval in agentic RAG, addressing a key bottleneck for large-scale systems by pairing production-scale queries with deep relevance judgments. By operating a public leaderboard focused on open-weight retrievers and quantifying GPU-hour savings via subsampling, Perplexity narrows the gap between academic IR benchmarks and real-world web search, while giving embedding model providers such as NVIDIA and others a standardized yardstick for first-stage retrieval performance.
