Retrieval-Augmented Generation (RAG)

An LLM only knows what it saw during training — which means it doesn’t know your company’s docs, last week’s news, or the contents of your database, and when asked anyway it will often confidently make something up. Retrieval-Augmented Generation (RAG) fixes this with a simple move: before the model answers, fetch the relevant facts and put them in the prompt. The model then answers from real, current, cited source material instead of fuzzy memory.

The core loop

RAG splits into two phases: an offline indexing phase that prepares your knowledge base, and an online retrieve-then-generate phase that runs on each query.

Phase 1 — Indexing (offline, once)

1. Chunk your documents into passages (a few hundred tokens each). Too big and retrieval is imprecise; too small and it loses context.
2. Embed each chunk with an embedding model — turning text into a vector (a list of numbers) where similar meaning ⇒ nearby vectors.
3. Store the vectors in a vector database (pgvector, Pinecone, Qdrant, FAISS, …) that supports fast nearest-neighbor search.

Phase 2 — Retrieve & generate (per query)

4. Embed the question with the same model.
5. Search the vector store for the top-$k$ nearest chunks (semantic similarity, not keyword match — so “car” finds “automobile”).
6. Augment the prompt: paste those chunks in as context, followed by the question and an instruction to answer only from the provided context.
7. Generate — the LLM writes an answer grounded in real passages, ideally with citations back to the sources.

Why it beats just asking the model

Problem with a bare LLM	How RAG addresses it
Hallucination	Answers are grounded in retrieved text you can cite
Stale knowledge	Update the index, not the model — no retraining
Private/internal data	The model never trained on it; retrieval supplies it
No provenance	Each claim traces back to a source chunk
Cost of fine-tuning	Indexing is far cheaper than training on your corpus

A minimal prompt shape

You are a support assistant. Answer using ONLY the context below.
If the answer isn't in the context, say you don't know.

<context>
[chunk 1 …]   (source: billing-faq.md)
[chunk 2 …]   (source: refunds-policy.md)
</context>

Question: How long do refunds take?

That “only from the context / say you don’t know” instruction is what converts retrieval into trustworthy generation — it’s the difference between grounding and decoration.

Making it actually good

Naive RAG (embed, top-k, done) gets you 70% of the way. The rest is retrieval quality:

• Hybrid search — combine semantic (vector) with keyword (BM25) search; each catches what the other misses (exact IDs, names, and codes especially).
• Reranking — over-fetch, say, 50 candidates, then use a cross-encoder reranker to pick the best 5. Big quality win for modest cost.
• Better chunking — split on semantic boundaries (headings, paragraphs), and attach metadata (title, section) so retrieval has more to grab onto.
• Query rewriting — have the LLM rephrase or decompose the question before retrieving (a first step toward agentic RAG).

RAG vs. the alternatives

• vs. long context — you could stuff the whole corpus into a huge context window, but it’s slow, expensive, and accuracy degrades in the middle. RAG sends only what’s relevant.
• vs. fine-tuning — fine-tuning teaches style and behavior; RAG supplies facts. They’re complementary, not competing.
• Agentic RAG — instead of a single fixed retrieval, let an agent decide what to search for, search multiple times, and refine — retrieval becomes a tool the agent calls, not a fixed pipeline stage. This is where most modern systems are headed.