Modern RAG: Beyond Naive Retrieval

The basic RAG loop — embed the question, fetch the nearest chunks, stuff them into the prompt — takes an afternoon to build and works impressively in a demo. Then real users arrive, ask questions in ways your chunks never anticipated, and accuracy falls off a cliff. Everything on this page exists because of that cliff. These are the techniques that have become standard in serious pipelines, roughly in the order you should reach for them.

Where naive RAG actually fails

Diagnose before treating. Nearly all RAG failures land in one of four buckets:

1. The answer wasn’t retrieved — it’s in the corpus, but the query’s embedding didn’t land near it (vocabulary mismatch, multi-part question).
2. The answer was retrieved but ranked too low — it’s result #14 and you sent the top 5.
3. The chunk lost its context — “the rate increased to 4.2%” retrieved perfectly, but which rate, when? The chunk no longer says.
4. The question needs synthesis — “compare our Q3 and Q4 churn drivers” has no single chunk to find; it needs many, connected.

Buckets 1–2 are retrieval problems; 3 is a chunking problem; 4 is an architecture problem. Each gets its own fix below.

Hybrid search: dense + keyword, fused

Embeddings are great at meaning and bad at exact strings — product codes, error messages, names, version numbers. Keyword search (BM25) is the exact opposite. Hybrid search runs both in parallel and fuses the results, most commonly with Reciprocal Rank Fusion (RRF), which needs no score calibration — it only uses each document’s rank in each list:

RRF(doc) = Σ over each list L:  1 / (k + rank_L(doc))     # k ≈ 60

A doc ranked #1 by BM25 and #8 by dense search beats a doc
ranked #3 in only one list. Robust, tuning-free, hard to beat.

This is the single highest-value upgrade to a naive pipeline, and every serious vector store (Qdrant, Weaviate, pgvector setups, Elastic) now ships it natively.

Reranking: cheap recall, then expensive precision

Retrieval and ranking want different tools. The fix is a two-stage funnel:

• Stage 1 (fast, broad): hybrid search pulls the top ~50–150 candidates. Bi-encoder embeddings are used here because they’re cheap — the query and documents were embedded independently, so search is just nearest-neighbour.
• Stage 2 (slow, sharp): a cross-encoder reranker (Cohere Rerank, BGE-reranker, Voyage) reads the query and each candidate together and scores actual relevance. It’s far more accurate precisely because it can attend across query and document — and far too slow to run over the whole corpus, which is why it only sees the shortlist.

Send the reranker’s top 5–10 to the model. This two-stage shape — optimize recall first, precision second — fixes bucket #2 almost completely and is standard in every production pipeline I’d call modern.

Query transforms: fix the question, not the index

The user’s phrasing is often the weakest link, so let a model rewrite it before retrieval:

• Query rewriting — strip conversational wrapping, resolve pronouns from chat history (“what about its pricing?” → “what is Acme Cloud’s pricing?”).
• Multi-query — generate 3–5 paraphrases, retrieve for all, union the results (pairs beautifully with RRF).
• Decomposition — split “compare X and Y’s approach to Z” into one retrieval per sub-question, then answer over the combined evidence.
• HyDE (Hypothetical Document Embeddings) — have the model hallucinate a plausible answer, then embed that fake answer and search with it. Sounds absurd, works because an answer-shaped text lands nearer to real answers in embedding space than a question-shaped one does.

Contextual retrieval: fix the chunks

Bucket #3’s fix, popularized by Anthropic (2024): at indexing time, have an LLM prepend each chunk with a sentence or two situating it in its document —

Original chunk:   "The rate increased to 4.2%, up from 3.1%."
Contextualized:   "From Acme Corp's Q2 2024 SEC filing, discussing customer
                   churn. The rate increased to 4.2%, up from 3.1%."

— then embed that. In Anthropic’s benchmarks this cut retrieval failures by ~35%, and combined with hybrid search + reranking by ~67%. It costs one LLM pass per chunk at index time, which prompt caching makes cheap (the full document sits in the cached prefix while each chunk is processed). Related ideas: late chunking (embed the whole document with a long-context embedder, then slice, so each chunk’s vector saw the full context) and parent-document retrieval (search small chunks, but hand the model the larger section around the hit).

GraphRAG: retrieval over relationships

For bucket #4 — questions that span many documents (“what themes recur across all customer complaints?”) — similarity search structurally can’t help, because no single passage contains the answer. GraphRAG (Microsoft, 2024) has an LLM extract an entity–relationship graph from the corpus at index time, clusters it into communities, and writes summaries at each level. Broad questions get answered over community summaries; narrow ones traverse the graph from a matched entity to its neighbours. Expensive to build, and the strongest known answer for global, synthesis-style questions over a corpus.

Agentic RAG: retrieval inside the loop

The current frontier folds retrieval into the agentic loop: instead of one retrieve-then-generate pass, the model gets search as a tool and drives it — reformulating queries, retrieving again when evidence looks thin, deciding whether retrieval is even needed, and stopping when it has enough. Patterns you’ll see named: Self-RAG (the model critiques its own retrievals and drafts), corrective RAG (a grader checks retrieved docs and falls back to web search when they’re irrelevant), and plain retrieval as an MCP tool in a general agent. Costs more tokens and latency; buys the ability to recover from a bad first retrieval — which no single-pass pipeline can do.

Evaluate or fly blind

None of the above is free, so measure before and after. The standard axes (RAGAS and similar frameworks):

Metric	Question it answers
Context recall	Did retrieval find the passages that contain the answer?
Context precision	Are the retrieved passages relevant, or padding?
Faithfulness	Is the answer actually supported by the retrieved text?
Answer relevance	Does the answer address the question asked?

Build a ~50-question test set with known answers from your real corpus. It’s an afternoon of work that turns every technique on this page from a vibe into an A/B result.

The order to adopt them

1. Hybrid search + reranking — biggest wins, lowest risk. Do these first.
2. Query rewriting (with chat history) — trivial to add, fixes real failures.
3. Contextual retrieval — when chunks losing context shows up in your evals.
4. Multi-query / HyDE / decomposition — for complex-question workloads.
5. Agentic RAG — when single-pass accuracy plateaus and latency budget allows.
6. GraphRAG — only for genuinely global questions; costly to maintain.

And once accuracy is solved, the next complaint is always speed — that’s Fast RAG: cutting response latency.