Measure RAG Retrieval Quality With Recall@K and MRR

A fluent answer cannot prove that Retrieval-Augmented Generation found the right evidence. The generator may write confidently from incomplete context, while a weak answer can also waste a perfectly good retrieval result. Measure retrieval before blaming the model.

For an offline regression test, label which document IDs are relevant to each query, freeze the number of ranked results you will evaluate, and calculate both coverage and rank. Recall@K asks how much of the known relevant set appeared in the first K results. Mean Reciprocal Rank (MRR) rewards placing the first relevant result near the top. Neither aggregate is safe by itself: a candidate can raise MRR while losing one business-critical query.

This guide is for a developer who can run Python 3 and export ranked document or chunk IDs from a retriever. No model API, vector database, package install, or production corpus is required for the worked lab. The synthetic fixture exists to make every label and failure inspectable; use reviewed domain queries and stable IDs before applying the gate to a real RAG release.

Label the Retrieval Contract Before Scoring

RAG evaluation contains at least two different systems. Retrieval selects context; generation turns that context into an answer. The current Evidently RAG evaluation guide recommends evaluating those components separately because the corrective action depends on which stage failed. An answer-quality score cannot tell you whether the right source was absent from the prompt.

Ground truth for retrieval is commonly stored as qrels: query-to-relevant-document judgments. Each query needs a stable identity, and each relevant source needs a stable document or chunk ID. Recall cannot be calculated honestly when the complete relevant set is unknown. In that case, manual or model-assisted relevance review may estimate usefulness, but the result is not the same contract as labeled Recall@K.

Fix K before comparing runs. Precision@3 divides relevant results in the first three positions by three. Recall@3 divides those same hits by the total number of labeled relevant results for that query. Reciprocal rank is 1 / position of the first relevant result, or zero when none appears inside the evaluated list.

Pinecone’s metric guide makes the order distinction explicit, while Anyscale’s RAG evaluation reference notes that retrieval sets an upper bound for answer quality. Choose floors from the user cost: missing one compliance or recovery source may matter more than adding a little irrelevant context.

Build a Small Offline Benchmark

Use one shell for the blocks below so LAB remains defined. The bootstrap refuses an existing path and records the exact marker required for cleanup.

set -Eeuo pipefail
LAB=/tmp/voxfor-rag-retrieval-eval-128
MARKER="$LAB/.voxfor-rag-eval-lab"

if [[ -e "$LAB" ]]; then
  printf 'Refusing existing lab path: %s\n' "$LAB" >&2
  exit 1
fi
mkdir -m 700 "$LAB"
printf '%s\n' 'voxfor-rag-eval-128' > "$MARKER"
python3 --version

Create five queries. Two need more than one relevant document, and q-recovery is a declared must-pass query. Labels should be reviewed by people who understand the corpus and user task; synthetic or model-generated labels are starting material, not automatic truth.

cat > "$LAB/qrels.json" <<'JSON'
{
  "q-backup": ["doc-backup-policy", "doc-restore-test"],
  "q-dns": ["doc-dns-negative-cache"],
  "q-gpu": ["doc-gpu-memory", "doc-kv-cache"],
  "q-recovery": ["doc-restore-test"],
  "q-security": ["doc-origin-isolation", "doc-firewall-baseline"]
}
JSON

The ranked-run file contains one baseline, one deliberately unsafe candidate, and one corrected candidate. A production export should record retriever version, embedding or keyword configuration, chunking version, corpus snapshot, filters, reranker, K, and query-set hash beside these IDs.

cat > "$LAB/runs.json" <<'JSON'
{
  "baseline": {
    "q-backup": ["doc-storage-price", "doc-backup-policy", "doc-restore-test"],
    "q-dns": ["doc-dns-negative-cache", "doc-dnssec", "doc-anycast"],
    "q-gpu": ["doc-model-format", "doc-gpu-memory", "doc-kv-cache"],
    "q-recovery": ["doc-restore-test", "doc-snapshot-list", "doc-rpo"],
    "q-security": ["doc-monitoring", "doc-firewall-baseline", "doc-origin-isolation"]
  },
  "candidate_bad": {
    "q-backup": ["doc-backup-policy", "doc-restore-test", "doc-storage-price"],
    "q-dns": ["doc-dns-negative-cache", "doc-dnssec", "doc-anycast"],
    "q-gpu": ["doc-gpu-memory", "doc-kv-cache", "doc-model-format"],
    "q-recovery": ["doc-snapshot-list", "doc-rpo", "doc-storage-price"],
    "q-security": ["doc-origin-isolation", "doc-firewall-baseline", "doc-monitoring"]
  },
  "candidate_good": {
    "q-backup": ["doc-restore-test", "doc-backup-policy", "doc-storage-price"],
    "q-dns": ["doc-dns-negative-cache", "doc-dnssec", "doc-anycast"],
    "q-gpu": ["doc-kv-cache", "doc-gpu-memory", "doc-model-format"],
    "q-recovery": ["doc-restore-test", "doc-rpo", "doc-snapshot-list"],
    "q-security": ["doc-origin-isolation", "doc-firewall-baseline", "doc-monitoring"]
  }
}
JSON

Stable IDs matter when the corpus changes. If rechunking replaces every identity, either map old judgments deliberately or create a new benchmark version. Voxfor’s private RAG and vector-database architecture explains where chunking, embeddings, storage, retrieval, and generation sit; this article owns the acceptance layer after a retriever returns ranked IDs.

Make Every Query Inspectable

The evaluator uses only Python’s standard library. It rejects empty qrels, duplicate labels, missing queries, duplicate ranked IDs, or runs shorter than K before calculating a plausible score.

cat > "$LAB/evaluate.py" <<'PY'
#!/usr/bin/env python3
import argparse, json, pathlib

def load(path):
    return json.loads(pathlib.Path(path).read_text())

def validate(qrels, runs, k):
    if not qrels or not runs:
        raise ValueError("qrels and runs must not be empty")
    expected = set(qrels)
    for qid, relevant in qrels.items():
        if not relevant or len(relevant) != len(set(relevant)):
            raise ValueError(f"{qid}: relevant IDs must be unique and non-empty")
    for run_name, run in runs.items():
        if set(run) != expected:
            raise ValueError(f"{run_name}: query IDs differ from qrels")
        for qid, ranked in run.items():
            if len(ranked) < k or len(ranked) != len(set(ranked)):
                raise ValueError(f"{run_name}/{qid}: need {k} unique ranked IDs")

def score(qrels, run, k):
    rows = []
    for qid in sorted(qrels):
        relevant = set(qrels[qid])
        top = run[qid][:k]
        hits = [doc for doc in top if doc in relevant]
        first = next((i for i, doc in enumerate(top, 1) if doc in relevant), None)
        rows.append({
            "query": qid,
            "hit_at_k": bool(hits),
            "precision_at_k": len(hits) / k,
            "recall_at_k": len(hits) / len(relevant),
            "reciprocal_rank": 0.0 if first is None else 1.0 / first,
            "found": hits,
            "missing": sorted(relevant - set(hits))
        })
    n = len(rows)
    return {
        "queries": rows,
        "macro": {
            "hit_rate_at_k": sum(r["hit_at_k"] for r in rows) / n,
            "precision_at_k": sum(r["precision_at_k"] for r in rows) / n,
            "recall_at_k": sum(r["recall_at_k"] for r in rows) / n,
            "mrr_at_k": sum(r["reciprocal_rank"] for r in rows) / n
        }
    }

parser = argparse.ArgumentParser()
parser.add_argument("--qrels", required=True)
parser.add_argument("--runs", required=True)
parser.add_argument("--run")
parser.add_argument("--baseline")
parser.add_argument("--candidate")
parser.add_argument("--k", type=int, default=3)
parser.add_argument("--must-pass", action="append", default=[])
parser.add_argument("--validate-only", action="store_true")
args = parser.parse_args()
qrels, runs = load(args.qrels), load(args.runs)
validate(qrels, runs, args.k)
if args.validate_only:
    print(json.dumps({"status": "valid", "queries": len(qrels), "runs": sorted(runs), "k": args.k}))
    raise SystemExit(0)
if args.run:
    print(json.dumps({"run": args.run, **score(qrels, runs[args.run], args.k)}, indent=2))
    raise SystemExit(0)
if not (args.baseline and args.candidate):
    parser.error("use --run or --baseline with --candidate")
baseline = score(qrels, runs[args.baseline], args.k)
candidate = score(qrels, runs[args.candidate], args.k)
candidate_rows = {row["query"]: row for row in candidate["queries"]}
checks = {
    "recall_not_lower": candidate["macro"]["recall_at_k"] >= baseline["macro"]["recall_at_k"],
    "mrr_not_lower": candidate["macro"]["mrr_at_k"] >= baseline["macro"]["mrr_at_k"],
    "must_pass_queries_hit": all(candidate_rows[q]["hit_at_k"] for q in args.must_pass)
}
result = {
    "baseline": args.baseline,
    "candidate": args.candidate,
    "k": args.k,
    "baseline_macro": baseline["macro"],
    "candidate_macro": candidate["macro"],
    "failed_queries": [r for r in candidate["queries"] if not r["hit_at_k"]],
    "checks": checks,
    "accepted": all(checks.values())
}
print(json.dumps(result, indent=2))
raise SystemExit(0 if result["accepted"] else 3)
PY

chmod 700 "$LAB/evaluate.py"
python3 -m py_compile "$LAB/evaluate.py"

First, validate identities and length. This is a separate control because an average across different query sets is not a comparable experiment.

python3 "$LAB/evaluate.py" \
  --qrels "$LAB/qrels.json" --runs "$LAB/runs.json" \
  --k 3 --validate-only

Expected validation is {"status": "valid", "queries": 5, "runs": ["baseline", "candidate_bad", "candidate_good"], "k": 3}. Then preserve the full baseline, including per-query found and missing IDs rather than only its macro line.

python3 "$LAB/evaluate.py" \
  --qrels "$LAB/qrels.json" --runs "$LAB/runs.json" \
  --run baseline --k 3 | tee "$LAB/baseline.json"

This baseline has Recall@3 1.0, MRR@3 0.7, and macro Precision@3 about 0.5333. The distinction is useful: every known relevant source is present, but three queries place their first relevant result at rank two.

Let a Better Average Fail

Run the negative candidate through a release gate. The command expects exit code 3; exit zero would mean the deliberately unsafe control was accidentally accepted.

set +e
python3 "$LAB/evaluate.py" \
  --qrels "$LAB/qrels.json" --runs "$LAB/runs.json" \
  --baseline baseline --candidate candidate_bad \
  --must-pass q-recovery --k 3 | tee "$LAB/candidate-bad.json"
bad_rc=${PIPESTATUS[0]}
set -e
test "$bad_rc" -eq 3

The candidate moves relevant items to rank one for four queries, so MRR rises from 0.7 to 0.8. Yet q-recovery returns no relevant result, macro Recall@3 falls to 0.8, and the must-pass check fails. The higher MRR is real and the release is still unsafe.

negative_control:
  exit=3 accepted=false candidate_mrr_at_3=0.8
  failed_queries=[q-recovery]
  recall_not_lower=false mrr_not_lower=true must_pass_queries_hit=false
corrected_candidate:
  exit=0 accepted=true candidate_recall_at_3=1.0 candidate_mrr_at_3=1.0
  recall_not_lower=true mrr_not_lower=true must_pass_queries_hit=true
cleanup=passed

Weaviate’s retrieval-metric tutorial demonstrates standard qrels/run calculations with pytrec_eval, and Deconvolute’s RAG metric review expands from binary relevance into MAP and NDCG. Use those measures when the task needs multiple relevant ranks or graded usefulness. Keep the query-level gate even after adding a richer aggregate.

Promote Only the Corrected Candidate

The corrected run restores doc-restore-test and places a relevant item first for all five queries. It must satisfy the same fixed query set, K, and floors as the baseline.

python3 "$LAB/evaluate.py" \
  --qrels "$LAB/qrels.json" --runs "$LAB/runs.json" \
  --baseline baseline --candidate candidate_good \
  --must-pass q-recovery --k 3 | tee "$LAB/candidate-good.json"

The regression check is complete when the qrels and every run contain the same five unique query IDs, each top-three list contains unique document IDs, the baseline has Recall@3 1.0 and MRR@3 0.7, the negative candidate exits 3 while naming q-recovery, and the corrected candidate exits zero with Recall@3 and MRR@3 both 1.0. Preserve the full JSON receipts and fixture hashes so another reviewer can reproduce the decision.

Real release criteria usually need more than not lower. Define minimum slice sizes and floors for high-value query groups, inspect every newly failed query, and test latency and context size separately. Do not let a large easy-query cohort overwhelm a small critical cohort. A weighted average may express business value, but an explicit must-pass list makes the non-negotiable boundary visible.

Corpus and infrastructure changes also belong in the receipt. Qdrant storage and optimizer sizing shows why index configuration and data movement have physical consequences; retrieval evaluation should version the corpus, collection schema, filters, and candidate index together. Generator capacity is a different decision covered by GPU memory planning for local LLM inference.

What This Receipt Proves—and What It Does Not

The lab proves metric arithmetic, input validation, query-level failure visibility, a negative control, and a corrected acceptance. It does not prove that the labels are complete, that user queries are representative, that chunks contain enough information to answer, or that the generator will cite and use retrieved context correctly.

LangChain’s current LangSmith RAG tutorial goes further by building a dataset, running a complete RAG application, and scoring retrieval and answers. That workflow is appropriate when you need end-to-end traces and model-based evaluators. The small offline gate remains valuable because it is deterministic, cheap, provider-neutral, and easy to run before expensive generation tests.

Build the real query set from support failures, search logs, domain-owner questions, known ambiguous terms, recently changed documents, and safety-critical tasks. Remove secrets and personal data before storing it. Review label disagreements, keep a benchmark changelog, and report results by slice rather than publishing one context-free score.

Observability must remain bounded. If query or document IDs become metric labels, Prometheus cardinality diagnosis explains why unbounded identities can multiply time series. Store detailed per-query receipts in files, traces, or an evaluation database; expose small aggregate and slice labels to monitoring.

Release automation should also protect retries. AI agent action reconciliation demonstrates the wider principle: a repeated automated attempt must preserve one stable decision identity. Bind each evaluation to the same corpus, query-set, run, and configuration hashes so a retry cannot silently compare different experiments.

Keep the Release Evidence, Remove the Fixture

Retain the qrels version, corpus snapshot, ranked-run files, evaluator hash, K, metric definitions, per-query output, slice thresholds, must-pass list, exit code, reviewer, and approval. Those fields turn an aggregate into a release decision that can be audited later.

Remove only the disposable synthetic fixture after verifying the exact path and marker:

test "$LAB" = '/tmp/voxfor-rag-retrieval-eval-128'
test "$(<"$MARKER")" = 'voxfor-rag-eval-128'
rm -rf -- "$LAB"
test ! -e "$LAB"
printf 'cleanup=passed\n'

Cleanup applies only to the marker-bound synthetic lab. If a candidate retriever was already deployed and the gate later fails, keep the evaluation evidence and restore the last approved retriever, index, corpus snapshot, filters, and configuration through the deployment system’s documented rollback. Never delete production qrels, user evidence, or the failed run merely to make the metric dashboard look clean.