Retrieval Augmented Generation Explained

Retrieval augmented generation explained in one sentence: it connects a large language model (LLM) to an external knowledge source so the model answers from retrieved facts instead of relying only on what it memorized during training. This guide covers how a RAG pipeline actually works end to end, the four components that decide whether it succeeds or fails, when RAG beats fine-tuning or a long context window, and how to measure quality so you ship something that holds up in production rather than a demo that breaks on the second question.

What retrieval augmented generation is and why it exists

A standard LLM is a closed system. Everything it "knows" was frozen at training time, it can't cite where an answer came from, and it confidently invents details when its parametric memory falls short—a failure mode known as hallucination. For most real applications that need current, private, or domain-specific information, that's a dealbreaker.

Retrieval augmented generation (RAG) fixes this by splitting the problem in two. A retriever finds relevant text from a knowledge source you control—internal docs, a product catalog, support tickets, a wiki. A generator (the LLM) then reads that retrieved text and writes a grounded answer. The pattern was formalized in a 2020 paper from Meta AI researchers (Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"), and it has become the default architecture for question answering over private data.

The payoff is concrete:

• Fresh knowledge without retraining. Update the index, not the model weights.
• Attribution. Because answers trace back to retrieved chunks, you can show citations.
• Lower hallucination rate. Grounding the model in real text constrains what it can say.
• Cost control. You inject only the relevant passages into the prompt instead of paying to fine-tune a model on your whole corpus.

RAG isn't a model—it's a system design. If you're new to wiring LLMs into real software, it sits alongside other core patterns covered in our guide to building production applications on top of LLMs.

How a RAG pipeline works

A RAG system runs in two phases: an offline indexing phase that prepares your knowledge, and an online query phase that runs every time a user asks something.

Phase 1: Indexing (offline)

This happens once per document (and again whenever content changes):

1. Load raw content from PDFs, HTML, databases, or APIs.
2. Chunk each document into smaller passages. You can't embed a 50-page PDF as one unit and expect precise retrieval—you split it into pieces, typically 256–512 tokens each.
3. Embed each chunk. An embedding model converts text into a dense vector (a list of floating-point numbers, often 384 to 3,072 dimensions) that captures meaning. Two passages about the same topic land close together in vector space.
4. Store the vectors, the original text, and metadata (source, date, section) in a vector database.

Phase 2: Retrieval and generation (online)

Every user query runs through these steps:

1. Embed the query with the same embedding model used for indexing. Mixing models here silently destroys retrieval quality—a mistake worth flagging early.
2. Search the vector database for the nearest chunks using a similarity metric (cosine similarity or dot product). This is an approximate nearest neighbor (ANN) search; it returns the top k most relevant passages, where k is usually 3–20.
3. (Optional) Rerank the candidates with a more accurate model to push the truly relevant chunks to the top.
4. Augment the prompt. Insert the retrieved chunks into a prompt template alongside the user's question and an instruction like "Answer using only the context below."
5. Generate. The LLM reads the augmented prompt and produces a grounded answer, ideally with citations back to the source chunks.

The whole online loop typically adds 100–500 ms of retrieval latency before generation starts—usually a worthwhile trade for accuracy.

Building a minimal RAG system: a worked example

Here's a small but complete RAG pipeline in Python using sentence-transformers for embeddings and NumPy for the similarity search. In production you'd swap NumPy for a real vector database, but this shows every moving part with nothing hidden.

import numpy as np
from sentence_transformers import SentenceTransformer

# 1. Your knowledge base, already chunked into passages.
documents = [
    "RAG retrieves relevant text before the LLM generates an answer.",
    "Embeddings map text to vectors so similar meanings sit close together.",
    "HNSW is a graph-based index for fast approximate nearest neighbor search.",
    "Reranking reorders retrieved chunks using a cross-encoder for accuracy.",
    "The capital of France is Paris.",
]

# 2. Index: embed every chunk once. Normalize for cosine similarity.
model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, fast, free
doc_embeddings = model.encode(documents, normalize_embeddings=True)

def retrieve(query, k=2):
    # 3. Embed the query with the SAME model used for indexing.
    q = model.encode([query], normalize_embeddings=True)[0]
    # 4. Cosine similarity = dot product of normalized vectors.
    scores = doc_embeddings @ q
    top_k = np.argsort(scores)[::-1][:k]
    return [(documents[i], float(scores[i])) for i in top_k]

def answer(query):
    hits = retrieve(query)
    context = "\n".join(f"- {text}" for text, _ in hits)
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context doesn't contain the answer, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
    )
    # Send `prompt` to your LLM of choice here.
    return prompt

print(retrieve("How does RAG reduce hallucinations?"))
# Returns the RAG and reranking chunks, not the Paris chunk.

Notice what this captures: chunking is assumed done, embeddings are computed once and reused, the query uses the same model, similarity is cosine, and the final prompt explicitly instructs the model to stay grounded and admit uncertainty. That last instruction is one of the cheapest, highest-leverage moves for reducing hallucinations with RAG.

The four components that make or break RAG quality

Most RAG failures aren't LLM problems—they're retrieval problems. If the right passage never reaches the prompt, no model can answer from it. Four components determine retrieval quality.

Embeddings: the foundation

The embedding model decides what "similar" means. A weak model retrieves topically related but unhelpful text; a strong one captures nuance. Popular choices range from compact open models like all-MiniLM-L6-v2 (384 dimensions, runs on a CPU) and the BGE and E5 families, up to hosted APIs such as OpenAI's text-embedding-3-large (up to 3,072 dimensions) and Cohere's embed models.

Bigger isn't automatically better. Higher dimensions cost more memory and slow search; pick a model that scores well on retrieval benchmarks like MTEB for your domain, then test it on real queries. If embeddings are unfamiliar territory, start with our deeper breakdown of how embeddings turn text into searchable vectors. The non-negotiable rule: embed your documents and your queries with the exact same model.

Chunking: the most underrated lever

How you split documents has an outsized effect on retrieval. Chunks that are too large dilute relevance and bury the answer in noise; chunks that are too small lose context. Common strategies include fixed-size splitting, recursive splitting that respects paragraph and sentence boundaries, and semantic chunking that splits where the topic shifts. A typical starting point is 256–512 tokens per chunk with 10–20% overlap so a sentence cut at a boundary still appears intact in a neighboring chunk.

There's no universal best size—it depends on your content and query style. We compare the approaches in detail in our guide to chunking strategies for RAG, but the practical advice is to treat chunk size as a tunable parameter and measure its effect rather than guessing once.

Vector database: where retrieval scales

For a few thousand chunks, brute-force search in NumPy works fine. Past that, you need a vector database with an ANN index. The dominant index type is HNSW (Hierarchical Navigable Small World), a graph structure that trades a sliver of recall for huge speed gains; IVF with product quantization is another option when memory is tight.

The field is crowded—Pinecone, Weaviate, Qdrant, Milvus, Chroma, and pgvector (a Postgres extension) all see heavy use, each with different tradeoffs in managed-vs-self-hosted operation, metadata filtering, and hybrid search support. We break down the options for different scales and budgets in our roundup of the best vector databases for RAG. One feature to prioritize regardless of choice: metadata filtering, so you can restrict retrieval to, say, the current user's documents or the last 90 days.

Reranking: cheap accuracy

Vector search is fast but coarse. A common upgrade is two-stage retrieval: pull a generous candidate set (say, top 50) with the vector index, then rerank with a cross-encoder that scores each query-document pair directly. Cross-encoders are too slow to run over an entire corpus but excellent at reordering a short list. Adding a reranker like Cohere Rerank or a BGE reranker often improves answer accuracy more than swapping the LLM does. The mechanics and when the latency cost is worth it are covered in our piece on reranking retrieval results for better precision.

A related upgrade is hybrid search, which blends dense vector search with sparse keyword search (BM25) and fuses the rankings—often with Reciprocal Rank Fusion. Hybrid search rescues queries that hinge on exact terms, product codes, or rare names that embeddings tend to smooth over.

When to use RAG—and when not to

RAG is powerful but not the answer to every problem. Two alternatives compete with it, and the right choice depends on what kind of knowledge you're adding.

RAG vs. fine-tuning is the classic confusion. Fine-tuning teaches a model how to behave—a consistent JSON format, a brand voice, a classification task. It's a poor and expensive way to inject facts, which it tends to memorize imperfectly. RAG teaches the model what to reference. When teams complain that fine-tuning "didn't fix hallucinations," it's usually because they reached for the wrong tool. Many production systems use both: fine-tune for behavior, RAG for knowledge.

RAG vs. long context is the newer debate now that models accept hundreds of thousands of tokens. If you only need to reason over a handful of documents per request, stuffing them all into the context window is simpler and skips the retrieval infrastructure entirely. But long context degrades on large corpora: it's slow, expensive per call, and suffers the "lost in the middle" problem where models underweight information buried in the center of a long prompt. RAG stays cheap and fast at scale because it sends only the relevant slice.

Skip RAG when your knowledge fits in the prompt, when the task needs no external facts (summarizing pasted text, reformatting), or when answers require reasoning across the entire corpus at once rather than a few relevant passages—retrieval can't surface what no single chunk contains.

Common mistakes and how to evaluate RAG

The fastest way to a frustrating RAG project is to build the pipeline, eyeball a few answers, and ship. These are the mistakes that show up later, and the metrics that catch them early.

The mistakes that keep recurring

• No evaluation harness. "It looked good in the demo" is not a quality bar. Build a test set of real questions with known good answers before you tune anything.
• Asymmetric embeddings. Different models (or model versions) for documents and queries quietly wrecks recall.
• Wrong k. Too few chunks miss the answer; too many flood the prompt with noise and raise cost. Tune it.
• Treating RAG as a hallucination cure. It reduces hallucination sharply but doesn't eliminate it—models still occasionally contradict or over-extend the retrieved context. Citations and "say you don't know" instructions are essential guardrails.
• A stale index. If content changes and you don't re-index, the model confidently serves outdated facts.
• Ignoring chunk boundaries. Splitting mid-sentence or mid-table strands the answer across two chunks, and neither alone is sufficient.

Measuring quality

Evaluate retrieval and generation separately, because they fail for different reasons.

Retrieval metrics ask whether the right chunks were found:

• Recall@k — did the relevant chunk appear in the top k? The single most important retrieval number.
• Precision@k — what fraction of retrieved chunks were actually relevant?
• MRR / NDCG — reward placing the right chunk near the top, which matters more once you rerank.

Generation metrics ask whether the answer used the context faithfully. The RAGAS framework popularized four widely used measures: faithfulness (does the answer stick to the retrieved context?), answer relevancy, context precision, and context recall. An "LLM-as-judge" setup—using a strong model to score answers against a rubric—has become a practical way to run these at scale.

The discipline of building test sets, tracking these numbers as you change chunk size or swap embedding models, and catching regressions is its own skill; we go deep on it in our guide to evaluating RAG quality with the right metrics. The key habit: change one variable at a time and re-measure.

Frequently asked questions

Is RAG the same as fine-tuning? No. RAG retrieves external text at query time and feeds it to the model, adding knowledge without changing the model. Fine-tuning adjusts the model's weights to change its behavior or style and is a poor way to inject facts. Many systems use both for different jobs.

Does RAG completely stop hallucinations? No, but it reduces them substantially by grounding answers in retrieved facts. Models can still misread or over-extend the context, so citations, an instruction to admit uncertainty, and faithfulness evaluation remain necessary.

How big should my chunks be? A common starting point is 256–512 tokens with 10–20% overlap, but the best size depends on your content and queries. Treat it as a parameter to tune and measure rather than a fixed rule.

Do I need a vector database to build RAG? Not for small prototypes—brute-force similarity search over a few thousand chunks works fine. Once you scale to large corpora or need fast filtered search, a vector database with an ANN index becomes essential.

Why are my answers wrong even though the LLM is good? Almost always a retrieval problem: the relevant chunk never reached the prompt. Check recall@k first, then look at chunking, embedding quality, and whether reranking would help—before blaming the model.

The takeaway

Retrieval augmented generation explained simply: it's the architecture that lets an LLM answer from your data instead of guessing from its training, and it lives or dies on retrieval quality, not on which model you pick. Get the foundations right—sensible chunking, matched embeddings, a vector store that scales, and reranking where precision matters—and back every change with measurement.

Your next step is to build the smallest end-to-end pipeline you can, assemble a test set of real questions, and measure recall@k before you optimize anything else. From there, tune one component at a time. Everything else in a strong RAG system is refinement on top of that loop.