Complete Guide to RAG System Performance Evaluation: In-Depth Analysis from Faithfulness to Context Relevancy

Complete Guide to RAG System Performance Evaluation: In-Depth Analysis from Faithfulness to Context Relevancy

As application development with LLMs (Large Language Models) has become a mainstream market trend, a subjective judgment of “Does the answer look plausible?” is no longer enough to guarantee service reliability. That is especially true for RAG (Retrieval-Augmented Generation) systems that use external knowledge, where reliability is directly tied to business success.

Evaluating RAG system performance, however, is like dissecting a complex machine inside a black box. Comparing only the final output (Answer) is like measuring a car’s top speed while ignoring engine efficiency or transmission defects.

This guide presents a practical methodology for engineers and data scientists building LLM-based services: breaking RAG system performance down into measurable, reproducible, and improvable metrics.

1. Why Is LLM Performance Evaluation So Hard? (The Trap of Simple Answer Matching)

LLMs are inherently probabilistic models. Even with the same prompt and Context, they can produce slightly different outputs depending on internal state or sampling parameters. That makes traditional NLP evaluation metrics hard to apply to RAG systems.

Limitations of Existing Evaluation Metrics

Metrics we commonly encounter, such as BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation), primarily measure word overlap between a reference answer and a generated answer.

Problems:

1. They ignore semantic differences: If two answers use different words but mean exactly the same thing, these metrics can assign a low score.
2. They cannot verify grounding: These metrics provide no mechanism to check where the answer came from (Source). The core of a RAG system is answering based on retrieved evidence.

We therefore need to shift the paradigm toward evaluating an LLM’s verifiability and transparency.

2. Understanding the Core Axes of RAG System Evaluation

A RAG system is broadly divided into two stages, and each stage requires a different kind of evaluation.

RAG Evaluation Flow (Conceptual Flow)

$$\text{Input (Query)} \xrightarrow{\text{Retriever}} \text{Context (Documents)} \xrightarrow{\text{Generator}} \text{Answer}$$

Around these two axes, we need to focus on three core metrics.

3. In-Depth Analysis of Core RAG Evaluation Metrics (Theory and Definitions)

These three metrics diagnose different kinds of failures. Understanding the conceptual differences among them is the most important part.

📚 Understanding Through Analogy: Recipe vs. Actual Dish

Think of a RAG system as a chef (LLM) looking at a recipe (Context) to make a dish (Answer).

1. Faithfulness:

   • Definition: Are all ingredients and cooking methods in the dish (Answer) the chef made explicitly mentioned in the provided recipe (Context)?
   • Measurement: Verify whether claims in the answer are grounded in the Context. Including content that is not in the Context is a hallucination.
   • Failure scenario: The chef says, “This really needs truffle oil to taste right,” but the recipe never mentions truffle oil.

2. Context Relevancy:

   • Definition: Does the retrieved recipe (Context) itself contain the information actually needed to answer the question (Query)?
   • Measurement: Evaluate whether the retriever fetched documents that are off-topic, too broad, or irrelevant to the question.
   • Failure scenario: The user asks, “What’s the weather like today?” but the retriever suddenly hands the chef a document titled “Company Annual Leave Policy.”

3. Answer Relevancy:

   • Definition: Does the generated answer (Answer) accurately capture the core intent of the question (Query) and address what the user actually wants to know?
   • Measurement: Catch cases where the Context is perfect and the answer is grounded in it, but the response still misses the core of the question or goes off on a tangent.
   • Failure scenario: The user asks, “What’s the difference between A and B?” but the answer only explains A at length and omits the comparison with B.

📊 Scoring Logic for Evaluation Metrics (Conceptual Approach)

These metrics are typically scored from 0.0 (not at all) to 1.0 (perfect).

• Faithfulness Score: $\text{Score} = \frac{\text{Number of claims in Answer supported by Context}}{\text{Total number of claims in Answer}}$
• Context Relevancy Score: $\text{Score} = \frac{\text{Number of retrieved chunks relevant to Query}}{\text{Total number of retrieved chunks}}$
• Answer Relevancy Score: $\text{Score} = \frac{\text{Number of key intents addressed in Answer}}{\text{Total number of key intents in Query}}$

4. Designing Practical Test Cases and Comparative Analysis

To turn theory into actual code, the most important step is designing which failure scenarios to test.

🧪 Three Essential Test-Case Scenarios

The following three cases are representative examples that best expose weaknesses in a RAG system.

Case 1: Faithfulness Failure (Hallucination)

• Scenario: The Context only contains information about A and B, but the model mentions C in its answer.
• Metric: Because it generated information not in the Context, Faithfulness Score = 0 (or very low).

Case 2: Context Blindness Failure (Ignoring Context)

• Scenario: The Context explicitly states “A is red,” but the model answers “A is blue.”
• Metric: Because it ignored a fact stated in the Context, Context Adherence Score = 0.

Case 3: Relevance Failure (Lack of Relevance)

• Scenario: The question is about marketing strategy, but the Context is filled only with HR policy content.
• Metric: Because the Context has low relevance to the question, the model failed to retrieve highly relevant information.

🛠️ Using Evaluation Tools (RAG Evaluation)

To automate this evaluation, it is essential to use RAG evaluation metrics (Faithfulness, Context Relevancy, and others) provided by frameworks such as LangChain or LlamaIndex and assign quantitative scores.

Summary: RAG system performance is not simply about whether the answer “looks plausible.” It depends on whether the system extracted accurate information that matches the question’s intent, based on the provided evidence (Context). Therefore, you must validate all three perspectives mentioned above (Faithfulness, Context Adherence, and Relevance).

Frequently Asked Questions (FAQ)

Q. What does answer_relevancy measure? A. RAGAS’s answer_relevancy measures how focused the generated answer is on the question. It reverse-generates several questions from the answer and scores them by embedding similarity to the original question. Scores drop when the answer contains a lot of unnecessary or off-topic content. Combined with faithfulness, it tells you whether the answer is both grounded in the evidence and focused on the question.

Q. Which should I look at first, faithfulness or answer_relevancy? A. If hallucination is the problem, prioritize faithfulness (whether the answer matches the retrieved evidence). If the answer is verbose or off-topic, prioritize answer_relevancy. If both are low, start by checking retrieval quality (context_precision and recall).