Evaluating RAG Quality

Evaluating RAG quality means measuring two things separately—did retrieval find the right context, and did the model use it faithfully—then tracking both against a fixed test set so you can prove a change helped instead of guessing. This guide covers the metrics that matter, the frameworks that compute them (RAGAS, the RAG Triad, LLM-as-judge), and a workflow that catches regressions before users do. The payoff: you stop shipping on vibes and start improving on evidence.

Why systematic evaluation beats eyeballing the output

Most RAG pipelines pass the demo and fail in production, and the reasons are predictable: answers that sound grounded but aren't, retrieval that returns the right documents in the wrong order, chunks that hold the answer but were cut at a bad boundary. You can't catch any of that by reading a handful of outputs—the failures are subtle, intermittent, and easy to miss when you're hoping it works.

A real evaluation needs three things: a fixed set of representative questions, known-good answers or relevant chunks to compare against, and metrics computed the same way every time. With around 70% of engineers reporting they have RAG in production or plan to ship within a year, evaluation infrastructure is a prerequisite, not an afterthought. The reward is diagnostic power—when quality drops, good metrics tell you which stage broke, so you fix retrieval or generation rather than blindly swapping models.

Evaluate retrieval and generation separately

The single most important principle: a RAG system has two stages that fail for different reasons, so measure them independently before measuring the whole. A faithful answer built on the wrong retrieved context is still wrong, and a perfect retrieval ruined by a sloppy generation step still disappoints. Splitting the metrics tells you where to look.

Retrieval metrics

These ask whether the right context reached the model. They're computed against labeled relevant chunks:

• Recall@k — did the relevant chunk appear in the top k retrieved? The most important single number, because the LLM can't use what was never retrieved.
• Precision@k — what fraction of retrieved chunks were actually relevant? Low precision means noise crowding the context window.
• MRR (Mean Reciprocal Rank) — rewards putting the first relevant result near the top.
• NDCG@k — accounts for both relevance and position. Research indicates it correlates more strongly with end-to-end RAG quality than binary precision/recall because it rewards the right ordering, not just the right documents—which makes it the metric to watch when you add a reranking pass to reorder retrieved results.

Precision and recall trade off against each other. A system at 0.9 precision and 0.5 recall ranks its small set beautifully but leaves half the answer on the floor. Raising k lifts recall but adds noise; empirically, most production systems land at 4–8 chunks, and faithfulness tends to degrade above 8 as the model's attention dilutes across irrelevant text.

Generation metrics

These ask whether the model used the retrieved context well:

• Faithfulness — are the answer's claims supported by the retrieved context? This is the core metric for reducing hallucinations with RAG.
• Answer relevancy — does the answer actually address the question asked?
• Citation coverage — does each claim point to a supporting chunk?

A crucial distinction: faithfulness is not factual correctness. Faithfulness asks whether the answer matches the retrieved source; correctness asks whether it's true in the real world. A high-faithfulness answer can still be wrong if the source document contained an error. Use faithfulness for internal grounding checks and a separate fact-checking layer for external accuracy.

End-to-end metrics

Finally, measure the whole pipeline: answer correctness against known-good answers, plus the operational realities of latency, cost, and safety. These are what your users and your finance team actually feel.

The standard frameworks: RAGAS and the RAG Triad

Two frameworks define how most teams compute these scores.

RAGAS (Retrieval Augmented Generation Assessment) is the open-source standard. It popularized four core metrics—faithfulness, answer relevancy, context precision, and context recall—and most can run reference-free: it decomposes an answer into individual claims with LLM-as-judge calls and verifies each against the retrieved context, so no labeled ground truth is required. The exception is context recall, which needs ground-truth answers to know whether everything relevant was retrieved. For teams without labeled data, RAGAS generates synthetic question-answer pairs from your own documents, and it runs in well under a dozen lines of code with integrations for LangChain, LlamaIndex, Haystack, and DSPy.

The RAG Triad, pioneered by TruLens, frames evaluation as three relationships: context relevance (is retrieved context relevant to the query?), groundedness (is the answer supported by that context?), and answer relevance (does the answer address the query?). It's the same ideas organized as a triangle connecting query, context, and answer.

Beyond these, the tooling has matured: DeepEval, LangSmith, Arize Phoenix, ARES (adversarial retrieval stress-testing), and AWS Bedrock's evaluation (adding citation precision and logical coherence) all occupy this space, and public benchmarks like RAGBench, CRAG, and LegalBench-RAG offer standardized test beds.

LLM-as-judge: how it works and where it misleads

Most generation metrics—faithfulness, answer relevancy—can't be computed by string matching, so they're scored by an LLM-as-judge: a model reads the question, the retrieved context, and the answer, and scores them against a rubric. It's the best available method for nuanced text quality, and it's reasonably good—judges built on strong models exceed 80% accuracy at telling genuinely relevant context from hard negatives designed to look relevant, with moderate-to-substantial agreement with human annotators on groundedness.

But treat the judge as an instrument with known error, not an oracle. Three pitfalls recur. First, cost and latency: every metric is one or more model calls, so evaluating thousands of examples adds up. Second, self-preference bias: a model judging its own generations tends to rate them favorably, so don't use the same model to generate and grade. Third, rubric sensitivity: vague judge prompts produce noisy, irreproducible scores—pin down the rubric and version it. Calibrate your judge against a small human-labeled sample before trusting it at scale.

Building an evaluation workflow that catches regressions

Metrics only help inside a repeatable loop. Here's a practical workflow.

1. Build a test set. Collect 50–200 real questions with known-good answers or labeled relevant chunks. Use RAGAS synthetic generation to bootstrap if you lack labeled data, then have a human review the hardest cases.
2. Pick your metrics. At minimum: recall@k for retrieval, faithfulness and answer relevancy for generation. Add NDCG once you care about ordering.
3. Establish a baseline. Run the current pipeline and record every score. This is the number every future change is measured against.
4. Change one variable at a time. Swap the embedding model, adjust chunk size, add a reranker—then re-run. Changing several at once tells you nothing about which one helped.
5. Gate on regressions. Wire the eval into CI so a pull request that drops faithfulness or recall below a threshold fails automatically, before it ships.
6. Monitor production. Sample live traffic and score it continuously; offline test sets drift from real usage over time.

A minimal RAGAS run looks like this:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

# Each row: the question, the model's answer, the retrieved chunks,
# and (for context_recall) the ground-truth answer.
data = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_chunks,     # list[str] per question
    "ground_truth": gold_answers,
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy, context_recall])
print(result)   # {'faithfulness': 0.91, 'answer_relevancy': 0.88, 'context_recall': 0.79}

Diagnose by stage. Low recall or context recall means fix retrieval—revisit your chunking strategies, your embedding model, or your index among the best vector databases for RAG. High retrieval scores but low faithfulness points at the generation step—the prompt or the model. For the bigger picture of how these stages fit together, our overview of how retrieval augmented generation works maps the full pipeline.

Common mistakes

Measuring end-to-end only. A single "is the answer good" score can't tell you whether retrieval or generation broke. Always split the layers.

No fixed test set. Evaluating on whatever queries come to mind makes runs incomparable. Freeze a versioned test set and reuse it.

Trusting the judge blindly. LLM-as-judge has real error and bias. Calibrate against human labels and never let a model grade its own output.

Confusing faithfulness with correctness. An answer can be perfectly grounded in a wrong source. Track both, and keep a fact-checking path for external accuracy.

Evaluating once and stopping. Quality drifts as documents, models, and usage change. Evaluation is continuous, not a launch checklist item.

Frequently asked questions

What are the most important RAG evaluation metrics? Recall@k for retrieval and faithfulness plus answer relevancy for generation form the minimum viable set. Add NDCG when ordering matters and answer correctness when you have ground-truth answers.

Can I evaluate RAG without labeled data? Largely yes. RAGAS computes faithfulness, answer relevancy, and context precision reference-free via LLM-as-judge, and it can generate a synthetic test set from your documents. Context recall and answer correctness are the metrics that still need ground truth.

What's the difference between faithfulness and answer relevancy? Faithfulness asks whether the answer's claims are supported by the retrieved context. Answer relevancy asks whether the answer actually addresses the question. An answer can be faithful but off-topic, or relevant but ungrounded.

Is LLM-as-judge reliable? Reliable enough to be the standard method, with strong judges exceeding 80% accuracy on hard cases, but it carries cost, self-preference bias, and rubric sensitivity. Calibrate it against human labels and don't grade with the generating model.

How big should my evaluation set be? Start with 50–200 representative questions covering your real query types and edge cases. Quality and coverage matter far more than raw size—a focused, well-labeled set beats thousands of random queries.

The takeaway

Evaluating RAG quality comes down to one habit: measure retrieval and generation separately against a fixed test set, every time you change something. Your next step is to assemble 50 real questions, run RAGAS or a comparable framework to baseline faithfulness and recall, and wire that check into CI so regressions fail loudly instead of silently reaching users. Once measurement is in place, every other improvement in your pipeline—better chunking, embeddings, reranking—becomes provable rather than hopeful.