RAG Evaluation Framework — Measuring Performance with RAGAS

RAG Evaluation Framework — Measuring Performance with RAGAS

Once you've built a RAG pipeline, the next question is: "How well is it actually working?" Human evaluation is slow and expensive. RAGAS (RAG Assessment) is a framework that uses an LLM as a judge to automatically evaluate RAG pipelines.

The Four Core RAGAS Metrics

1. Faithfulness

Measures how faithful the generated answer is to the retrieved context. If the model fabricates content that isn't in the context, the score drops.

score = (number of claims supported by the context) / (total number of claims in the answer)

A low score is a signal that the model is hallucinating.

2. Answer Relevancy

Measures how relevant the generated answer is to the question. Extra unrelated content lowers the score.

score = avg(cosine_similarity(question, reverse-generated questions))

RAGAS reverse-generates several questions from the answer and computes their similarity to the original question.

3. Context Precision

The proportion of retrieved chunks that are actually useful. The more irrelevant documents you retrieve, the lower this score.

4. Context Recall

How much of the information needed to produce the ground-truth answer is present in the retrieved context. This requires a labeled dataset.

Hands-on Code

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

#
eval_data = {
 "question": ["RAG ?", " DB?"],
 "answer": ["RAG ...", " DB..."],
    "contexts": [
 ["RAG Retrieval-Augmented...", " ..."],
 [" ..."],
    ],
 "ground_truth": ["RAG external ...", " DB save..."],
}

dataset = Dataset.from_dict(eval_data)

result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(result)
# {'faithfulness': 0.89, 'answer_relevancy': 0.92, 'context_precision': 0.85, 'context_recall': 0.78}

Interpreting Scores and How to Improve Them

Auto-generating an Evaluation Dataset

Even without labeled Q&A data, you can synthesize a dataset with RAGAS.

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator = TestsetGenerator.from_langchain(
    generator_llm=ChatOpenAI(model="gpt-4o"),
    critic_llm=ChatOpenAI(model="gpt-4o"),
    embeddings=OpenAIEmbeddings(),
)

testset = generator.generate_with_langchain_docs(
    documents=docs,
    test_size=50,
    distributions={simple: 0.5, reasoning: 0.3, multi_context: 0.2},
)

Integrating Evaluation into CI

Every time you change the RAG pipeline, evaluation should run automatically.

# .github/workflows/rag-eval.yml
- name: RAG Evaluation
  run: python eval/run_ragas.py
- name: Check Thresholds
  run: |
    python -c "
    import json
    r = json.load(open('eval/results.json'))
 assert r['faithfulness'] > 0.85, 'Faithfulness '
 assert r['context_recall'] > 0.75, 'Context Recall '
    "

If scores fall below the threshold, you can block the PR.

That concludes the Mastering RAG series. If you've followed along from concepts (EP 01) → choosing a vector DB (EP 02) → chunking strategy (EP 03) → evaluation (EP 04), you're ready to design and measure a production-grade RAG pipeline.

When Scores Wobble Between Runs

RAGAS metrics use an LLM as the judge internally, so rerunning on the same data gives slightly different scores. Judging A/B settings on small differences without knowing this leads to wrong conclusions.

Causes

• The judge LLM's output isn't fully deterministic
• A small evaluation set lets a few items move the average a lot
• The judge model version changes (alias in use)

Check: evaluate the same setting several times and measure the spread

import statistics
runs = [evaluate(dataset, metrics=[faithfulness, answer_relevancy]).to_pandas() for _ in range(5)]
for m in ["faithfulness", "answer_relevancy"]:
    means = [r[m].mean() for r in runs]
    print(m, round(statistics.mean(means), 3), "±", round(statistics.stdev(means), 3))

If the difference between two settings is smaller than this spread, treat it as "no difference."

Fix

• Pin the judge LLM to a snapshot version and set temperature to 0.
• Grow the evaluation set, or compare per-item score differences with a paired test (e.g., Wilcoxon signed-rank).

Prevention

• Set CI gate thresholds with the measured spread in mind, and have a person review results near the threshold.