Fast RAG: How Production Systems Cut Response Latency

A RAG answer that takes 12 seconds feels broken even when it’s correct. Production teams obsess over latency as much as accuracy — and the good news is that fast RAG is mostly engineering, built from a handful of well-understood techniques. This page walks the pipeline end to end and shows where the time goes and how modern systems claw it back.

First: know your latency budget

You can’t optimize what you haven’t broken down. A typical naive pipeline:

Two numbers matter, and they’re different: TTFT (time to first token — what feels like responsiveness) and total time. Most techniques below target one or the other; know which you’re buying.

Streaming: the highest-ROI “optimization” isn’t one

Streaming tokens as they’re generated changes nothing about total time and changes everything about perceived speed: a 6-second answer that starts appearing at 600 ms feels fast. Every serious product streams. Push it further:

• Stream the citations first — show “Searching… found 4 sources” while generation starts, so the interface is alive during the silent phase.
• Pipeline the stages — start the reranker as soon as the first vector results arrive; open the LLM request the instant the context is assembled, not after post-processing niceties.

Make retrieval itself fast (the milliseconds still matter at scale)

• ANN indexes. Nobody brute-force scans embeddings in production. HNSW (a navigable-graph index) answers nearest-neighbour queries in ~1–10 ms over millions of vectors at 95–99% recall; IVF variants trade a little recall for far less memory. This is what Qdrant/pgvector/FAISS do under the hood — your job is mostly to not misconfigure it (ef_search, nprobe).
• Quantized & binary embeddings. Full float32 vectors are fat. int8 quantization cuts memory 4× with ~1% recall loss; binary embeddings (1 bit per dimension) cut it 32× and make similarity a hamming distance — XOR + popcount, absurdly fast. The standard trick is binary first pass, float rescore on the top candidates: ~95%+ of full-precision quality at a fraction of the cost.
• Matryoshka embeddings (MRL). Newer embedding models are trained so the first N dimensions form a valid smaller embedding. Search with the first 256 dims, rescore with all 1024 — same funnel logic as rerankers, applied inside the vector itself.
• Keep the index hot. Cold starts (index loading from disk, serverless spin-up) dwarf query time. Pin the index in memory; co-locate it with the app to avoid a cross-region hop that costs more than the search itself.

Caching: the fastest retrieval is none

Three distinct caches, often confused:

1. Exact/normalized cache — same question (after lowercasing, stripping punctuation) → serve the stored answer. Trivial, and in support/FAQ workloads hit rates of 20–40% are common.
2. Semantic cache — embed the incoming query and search a cache of previously answered queries; if similarity clears a threshold (~0.95), serve the cached answer in tens of milliseconds instead of seconds. This is the “latest technique” version of caching (GPTCache and friends) — with one honest caveat: tune the threshold on real traffic, because “how do I delete my account” and “how do I delete my message” embed dangerously close.
3. Prompt / prefix caching — provider-side KV-cache reuse (Anthropic, OpenAI, Gemini all ship it). Structure your prompt so the stable prefix (system prompt, tool definitions, few-shot examples) comes first and the volatile parts (retrieved chunks, user question) come last; the provider skips re-processing the cached prefix, typically cutting TTFT dramatically and input cost by ~50–90% on cache hits. In RAG this is free money — your system prompt rarely changes. It’s also what makes contextual retrieval affordable at index time.

Shrink the work the model does

The purple bar shrinks two ways: fewer tokens, or faster tokens.

• Fewer input tokens. Rerank hard and send 5 great chunks, not 20 decent ones — output quality usually rises while prefill cost falls. Context compression (LLMLingua-style) can squeeze retrieved text 2–5× with minor loss when you must send a lot.
• Fewer output tokens. Latency scales linearly with output length — decoding is sequential, one token at a time. “Answer in 2–4 sentences unless asked for detail” is a genuine latency optimization.
• A faster model. The most honest lever. Route by difficulty: a small fast model (Haiku-class) answers the easy 80%, escalating to a large model only when a cheap classifier — or the small model’s own uncertainty — says so. Model routing is now standard practice in high-volume RAG.
• Speculative decoding. Provider-side: a small draft model proposes several tokens, the big model verifies them in one parallel pass — identical output, 2–3× faster decoding. RAG is especially suited to it, because answers quote retrieved text and quoted spans are easy for the draft model to predict. You get this by flag/default on modern serving stacks (vLLM, TensorRT-LLM, provider APIs) rather than building it.

Parallelize the pipeline

Naive pipelines are sequential by accident:

• Run BM25 and vector search concurrently (you’re fusing them anyway); run multi-query retrievals concurrently.
• Skip what you can. A tiny classifier (or rules) decides “does this need retrieval at all?” — “thanks!” shouldn’t trigger a vector search. Same idea as agentic RAG’s decide-to-retrieve, applied for speed.
• Speculative RAG (research-grade but real): draft an answer with a small model over each retrieved subset in parallel, have the big model only verify/select — cutting big-model tokens and wall-clock time together.

A realistic before/after

Naive:     embed 50ms → search 150ms → 20 chunks, no rerank
           → 6,000-token prompt, big model, no streaming, no cache
           ≈ 8s to first paint. Feels broken.

Tuned:     semantic cache (30% of traffic: ~50ms total)
           hybrid search ∥ 40ms → rerank 120ms → 5 chunks
           cached system prefix → TTFT ~400ms, streaming on
           small model for easy queries, speculative decoding on
           ≈ 0.6s to first token, ~2.5s total. Feels instant.

Same corpus, same questions — the difference is entirely the stack above.

Adoption order

1. Streaming + prompt caching — an afternoon, transforms perceived speed.
2. Send fewer, better chunks (rerank hard) — faster and more accurate.
3. Exact + semantic caching — if traffic repeats, this is your biggest win.
4. Model routing / smaller models — when quality evals say you can.
5. Quantized embeddings, parallel retrieval — at scale, for cost as much as speed.
6. Speculative decoding — turn it on if you self-host; enjoy it silently if you don’t.