RAG evals

A RAG eval is an LLM eval for retrieval-augmented generation. Your endpoint returns the passages it retrieved alongside its answer, and the judge scores four things separately: whether the answer sticks to those passages, whether it answers the question, whether the passages were the right ones, and whether they held everything the answer needed. That tells you whether a bad answer is a retrieval problem or a generation problem.

Run one whenever you change chunking, the embedding model, the retriever or re-ranker, top-k, or the prompt that turns passages into answers.

What you need

• A dataset with an input column, and ideally expected (needed for a meaningful context_recall).
• An endpoint that returns both the answer and the retrieved passages in its JSON response.
• The target’s contexts_path: where the passages are in that response. A RAG eval without it stops with RAG evals need the target’s contexts path.

Everything else (body template, headers, sample size, threshold, the run’s guards) works exactly as in an LLM eval.

Point Plainwright at the passages

Given this response from your endpoint:

POST https://rag.example.com/ask → 200

{
  "answer": "Yes, we ship to Canada; orders arrive in 5–8 business days.",
  "sources": [
    { "id": "kb-201", "text": "Returns are accepted within 30 days of delivery.", "score": 0.71 },
    { "id": "kb-112", "text": "We ship to the US and Canada.", "score": 0.69 },
    { "id": "kb-118", "text": "Canadian orders arrive in 5–8 business days.", "score": 0.66 }
  ]
}

set response_path to answer and contexts_path to sources[*].text. The judge then receives the three passages, numbered in the order your retriever returned them.

Response shape	contexts_path
{"sources": [{"text": "…"}, …]}	sources[*].text
{"contexts": ["…", "…"]}	contexts
{"data": {"docs": [{"page_content": "…"}]}}	data.docs[*].page_content
{"result": {"citations": [{"snippet": "…"}]}}	result.citations[*].snippet

A single string becomes one passage, nested lists are flattened, and objects are passed as JSON. The judge sees at most the first 10 passages, each cut to 1,500 characters.

The four metrics

All four come from one judge call per row. The exact wording the judge receives is in the Metrics reference.

Metric	The question it answers	Compares
faithfulness	Is every claim in the answer supported by the retrieved passages?	answer ↔ passages
answer_relevance	Does the answer address the question that was asked, completely and specifically?	answer ↔ question
context_precision	Are the retrieved passages relevant, especially the ones ranked first?	passages ↔ question
context_recall	Do the passages contain what is needed to produce the expected answer?	passages ↔ expected answer

A worked example

Take one row and three ways the system could get it wrong.

• Question: “Do you ship to Canada, and how long does it take?”
• Expected: “Yes, in 5–8 business days.”

1. Faithfulness. Retrieved: We ship to the US and Canada. / Canadian orders arrive in 5–8 business days. Answer: “Yes, delivery takes 5–8 business days and shipping is free.” The free-shipping claim is in no passage, so faithfulness drops even though the rest is right. A hallucination like this is a generation problem.

2. Answer relevance. Same passages, answer: “We ship to the US and Canada.” True and grounded, so faithfulness is high, but the delivery time was never given: answer relevance drops.

3. Context precision. Retrieved, in order: Returns are accepted within 30 days. / Gift cards never expire. / Canadian orders arrive in 5–8 business days. The useful passage is ranked last behind noise, so context precision drops, even if the final answer is fine.

4. Context recall. Retrieved: We ship to the US and Canada. only. Nothing says how long delivery takes, so the expected answer can’t be produced from the passages: context recall drops. The retriever missed a document.

Read the four together

Pattern	Likely cause	What to try
Low faithfulness, high context recall	The right passages were there; the model added or changed facts.	Tell the model to answer only from the sources; lower its temperature; show source ids.
Low answer relevance	Answers dodge, hedge or answer only part of the question.	Prompt for direct, complete answers; check truncation.
Low context precision	Retrieval returns noise, or ranks it above the useful passage.	Add or tune a re-ranker, return fewer passages, improve chunking.
Low context recall	The passage the answer needs is never retrieved.	Check the document is indexed, adjust chunk size and overlap, try another embedding model, raise top-k.

Reference answers and passages in the dataset

Column	How a RAG eval uses it
input	Sent to your endpoint as {{input}}.
expected	The judge’s reference for context_recall (and for correctness, if you add it).
context	A ground-truth passage. When a row has no expected, it is shown to the judge in its place, marked as reference context.

Without either, context_recall is judged against what a complete answer would require, which is less reliable. Add expected for the questions that matter.

Full example

rag-test.json

{
  "module_id": "<module-id>",
  "name": "Help-centre retrieval",
  "kind": "rag",
  "natural_language": "The help-centre assistant retrieves the right articles and answers only from them.",
  "config": {
    "dataset_id": "<dataset-id>",
    "target": {
      "type": "http",
      "url": "https://rag.example.com/ask",
      "headers": { "Authorization": "Bearer {{secrets.RAG_API_KEY}}" },
      "body_template": { "question": "{{input}}", "top_k": 5 },
      "response_path": "answer",
      "contexts_path": "sources[*].text"
    },
    "metrics": ["faithfulness", "answer_relevance", "context_precision", "context_recall"],
    "sample_size": 50,
    "threshold": 0.75
  }
}

In the editor, a RAG eval also offers correctness and latency. Create and run the test as shown in Work from the API.

Read the result

The Results tab lists every row with its answer, score and latency. Open a row to see each metric’s score and reason, and the retrieved contexts exactly as the judge saw them. That is the quickest way to tell a retrieval miss from a hallucination. Overview shows the mean per metric, so you can see at a glance which of the four is dragging the score down.

Set a pass mark per metric family

The run passes on the mean of all four metrics. If retrieval and generation have very different targets, split them into two RAG evals over the same dataset (one scoring context_precision and context_recall, one scoring faithfulness and answer_relevance), each with its own threshold.

Next steps

Metrics referenceThe judge's exact definitions for all eight metrics.

DatasetsAdd expected answers and reference passages to your rows.

LLM evalsEndpoint options, scoring rules and the CI gate.

Trials, thresholds and costOne judge call per row: budget it and pick a cheap judge.