RAG Evaluation Metrics: Assessing Answer Relevancy, Faithfulness, Contextual Relevancy, And More - Confident AI

RAG Evaluation Metrics: Assessing Answer Relevancy, Faithfulness, Contextual Relevancy, And More

Oct 10, 2025·9 min read

Jeffrey Ip

Co-founder @ Confident AI. Creator of DeepEval & DeepTeam. Building an unhealthy LLM evals addiction. Ex-Googler (YouTube), Microsoft AI (Office365).

Building a Retrieval-Augmented Generation (RAG) pipeline isn’t the mission impossible — knowing if it’s actually working is. You might fine-tune prompts, tweak retrievers, and switch models, but if your system is still hallucinating or citing the wrong context, what’s really broken? Without the right RAG evaluation metrics, it’s guesswork.

Sure, 2025 is the year of AI agents, but let’s face it: most agentic system still has a RAG pipeline somewhere in their AI workflow, and it’s vital that you’re about to secure the quality of that too.

So, in this article, we’ll go through everything you’ll need for RAG evaluation:

All of course, this all includes code samples using DeepEval ⭐, an open-source LLM evaluation framework. Let’s get started.

TL;DR

What is RAG Evaluation?

RAG evaluation is the process of using metrics such as answer relevancy, faithfulness, and contextual relevancy to test the quality of a RAG pipeline’s “ retriever” and the “ generator” separately to measure each component’s contribution to the final response quality

To do this, RAG evaluation involves 5 key industry-standard metrics:

For agentic RAG use cases, which we'll cover more in a later section, you might also find it useful to include a task completion metric to evaluate your AI agent RAG pipeline as well.

Before we get too into the metrics though and RAG agents, let’s recap what RAG is. A RAG pipeline is an architecture where an LLM’s output is informed by external data that is retrieved at runtime based on an input. Rather than relying solely on the model’s trained knowledge, a RAG system first:

  1. Searches a knowledge source — like a document database, vector store, or API, then
  2. Feeds the retrieved content into the prompt for the LLM to generate a response.

Here’s a diagram showing how RAG works:

A RAG Pipeline Architecture

You’ll notice that the quality of the final generation is highly dependent on the retriever doing its job well. A RAG pipeline can only produce a helpful, factually correct response if it:

  1. Has access to the right context.
  2. Does not hallucinate, and follows instructions given the right context.

In fact, the quality of your RAG generation is only as strong as its weakest component — retriever or generator. Think of it as a product, not a sum: if either the retriever or the generator performs poorly (or fails entirely), the overall quality of the output can drop to zero, regardless of how well the other performs.

So the question becomes: how do we evaluate both the retriever and the generator independently to understand where the failures are happening in a RAG pipeline?

Ways In Which Your RAG Pipeline Can Fail

You can address this by using retriever-targeted and generator-targeted metrics that evaluate each component separately, focusing on the most common failure modes within each stage of the pipeline.

When the retriever fails, it often due to the following:

This forces the LLM to “fill in the blanks,” leading to hallucinations or confidently wrong answers. Even with good retrieval, the generator can still fail to use the context effectively:

Now with this in mind, it’s time to talk about how retrieval and generation metrics tackle each one of them.

Retrieval Evaluation Metrics

The retrieval step typically consists of three main stages:

  1. Embedding the query: Convert the input into a vector representation using an embedding model of your choice (e.g., OpenAI’s text-embedding-3-large).
  2. Vector search: Use this embedded query to search a vector store containing your pre-embedded knowledge base, retrieving the top-K most similar text chunks.
  3. Reranking: Refine the initial results by reranking the retrieved chunks, since raw vector similarity may not always reflect true relevance for your specific use case.

Retriever in RAG

Which involves several tunable hyperparameters — like the embedding model, chunk size, and top-K. Retrieval metrics help answer key questions such as:

A generic model might miss nuances in specialized fields like medicine.

Initial vector search may not reflect true relevance.

Chunk size and Top-K directly affect how much and what gets passed to the generator.

Retrieval metrics uses a variation of QAG (question-answer-generation) to compute metric scores, which is a great technique used by LLM evaluators to combat arbitrary score calculations.

Contextual Relevancy

The contextual relevancy metric uses LLM-as-a-judge to quantify the proportion of retrieved text chunks that are relevant to the input. It is a measure of how well your top-K and chunk size is configured. Smaller chunks can give you more precise information, but they often require retrieving more of them (higher top-K) to cover enough context, while larger chunks may capture broader meaning but risk including irrelevant details.

You can definitely build something to loop through all your text chunks, handle asynchronously LLM-as-a-judge evals for judging relevancy, and compute the final score, however if you want something that just works out of the box, I’d recommend DeepEval ⭐, the open-source LLM evaluation framework with all the RAG metrics you can imagine:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualRelevancyMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = ContextualRelevancyMetric(threshold=0.7, model="gpt-4")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score, metric.reason)

Note that you should click here to learn what a test case is.

The threshold parameter in the ContextualRelevancyMetric tells DeepEval what is the acceptable proportion of relevant text chunks for a test case to pass, while the model determines which model should be used to determine each text chunk’s relevancy.

For more information, find DeepEval’s full documentation here.

Contextual Recall

The contextual recall metric is a reference-based metric that uses LLM-as-a-judge to quantify the proportion undisputed facts found in the labelled, expected output for a given input can be attributed to the retrieved text chunks. It measures the embedding model parameter.

The whole purpose of contextual recall is to assess whether the retrieval context actually contains all the necessary information to produce the ideal output. Without contextual recall, anyone can achieve perfect contextual relevancy by retrieving as little information as possible.

Here’s the code implementation in DeepEval ( docs here):

from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualRecallMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

metric = ContextualRecallMetric(threshold=0.7, model="gpt-4")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score, metric.reason)

Contextual Precision

The contextual precision metric is a reference-based metric that uses LLM-as-a-judge to quantify whether relevant text chunks are ranked higher than irrelevant ones. It measures the quality of your reranker, and follows a rather complicated equation that you can learn more about here.

The purpose of precision is to make sure that the context you are feeding into your generator is not just relevant, contains all the information, but also in the correct order for an LLM to consider each text chunk’s importance appropriately.

Here’s how you can use it in DeepEval:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualPrecisionMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output of your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

metric = ContextualPrecisionMetric(threshold=0.7, model="gpt-4")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metri.measure(test_case)
print(metric.score, metric.reason)

Full documentation here.

(Note that you should aim to use all three contextual metrics in order to evaluate your retriever fully.)

Generation Evaluation Metrics

The generation step, which follows retrieval, typically involves two key stages:

  1. Prompt construction: Combine the user’s original input with the retrieved context to form a complete prompt.
  2. LLM generation: Send this prompt to your language model (e.g., GPT-4, Claude, or an open-source model) to generate the final output.

An example generator can be as simple as this:

prompt = """You are a helpful assistant. Use the information provided in the context below to answer the user's question. If the answer is not found in the context, say "I don't know based on the provided information."

Context:
{retrieval_context}

Question:
{user_query}

Answer:"""

res = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[\
        {"role": "system", "content": "You are a helpful assistant."},\
        {"role": "user", "content": prompt}\
    ],
    temperature=0.2
)
print(res.choices[0].message["content"])

Compared to retrieval, generation is usually more straightforward — thanks to standardized APIs and model interfaces — but generation metrics still help answer key questions like:

You’ll also likely need custom, task specific metrics to evaluate RAG generation, as we’ll go through why and how later. Similar to retrieval metrics, generation metrics also leverages LLM evaluators with QAG.

Answer Relevancy

The answer relevancy metric quantifies the proportion of the generated output that is relevant to the given input. It is extremely straightforward and a direct measure of how well your model is able to follow instructions in your prompt template.

Here’s how you can use answer relevancy in DeepEval:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the output from your LLM app
    actual_output="We offer a 30-day full refund at no extra cost."
)

metric.measure(test_case)
print(metric.score, metric.reason)

Full documentation here.

Faithfulness

The faithfulness metric quantifies the proportion of undisputed, truthful facts in the retrieval context that were not contradicted in the generated output. In other words, it measures the hallucination rate of your LLM.

Here’s how to use it in DeepEval:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score, metric.reason)

Full documentation here.

Custom G-Eval

Often times your RAG pipeline is not a generic QA, and has some aspects of customizations. It might be that responses has to be generated in a certain format (e.g. bullet points, markdown, JSON, etc.), which the generic RAG metrics can’t account for.

For this reason, you’ll want custom metrics that evaluates based on your unique use case. Also note that a custom RAG metric belongs to the generation category of metrics because a retriever is not where task-specific customizations are done.

There are various ways you can built a custom metric, but here we are going to use one of DeepEval’s most versatile metric, G-Eval. G-Eval (full read here) is almost capable of evaluating any task and you can define its criteria in everyday language.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

metric = GEval(
    name="Format",
    criteria="Determine whether the actual output is in Spanish.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    strict_mode=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="Ofrecemos un reembolso completo de 30 días sin costo adicional."
)

metric.measure(test_case)
print(metric.score, metric.reason)

In this example, we set our criteria as whether our generator is able to generate outputs in spanish consistently. You’ll notice that we set the strict_mode parameter on G-Eval as True . This tells DeepEval that G-Eval should only output a binary score (0 or 1) instead of a continuous one. which makes a lot of sense for this use case.

You can find the documentation for G-Eval here and the most used G-Eval use cases in DeepEval here.

Conclusion

In this article, we talked about how a RAG pipeline is only as good as it’s worst performing component, hence why we separate the evaluation of the retriever and generator.

Retriever has targeted LLM-as-a-judge metrics such as contextual relevancy, recall, and precision, while the generator has answer relevancy, faithfulness, and G-Eval to evaluate it’s performance. We also learnt that these metrics can be applied to evaluate RAG pipelines from an end-to-end perspective, or at a component-level. Both of these modes of evaluation can then be ran using a raw Python script, or in CI/CD pipelines.

Beyond the theory, all of this can be implemented in a few lines of code using DeepEval, as it takes care of a lot of boring repetitive tasks such as running tests async in CI/CD, handling exceptions during evaluation, implementation of metrics, etc.

Don’t forget to give ⭐ DeepEval a star on Github ⭐ if you found this article useful, and as always, till next time.