Reproducible evaluation

Benchmark retrieval with your questions and your bundle.

The kit provides question, judgment, and result formats plus a dependency-free evaluator. It measures retrieval candidates without presenting sample output as a universal performance claim.

Download benchmark kit Read retrieval guide

Included

Enough structure to make results inspectable.

questions.jsonl

Real questions grouped by lookup, multi-hop, lifecycle, and unanswerable cases.

judgments.csv

One or more acceptable supporting concepts for every answerable question.

evaluate.py

Dependency-free recall, reciprocal-rank, abstention, latency, and token calculations.

RESULTS.md

A record of corpus revision, settings, hardware, metrics, and failure review.

Interpretation

Metrics find failures. Review explains them.

Recall tells you whether the retriever surfaced acceptable evidence. Reciprocal rank tells you how early that evidence appeared. Abstention accuracy measures unanswerable questions separately. Latency and tokens loaded expose the cost of reaching evidence.

None of those metrics proves that a generated answer is correct. Review missed concepts, stale results ranked above current policy, confident answers to unanswerable questions, and cases where graph expansion added noise.

Direct answer

Compare retrieval methods against the same evidence.

Use real questions, mark every acceptable supporting concept, then run each retrieval candidate against the same corpus revision. The evaluator rejects incomplete or malformed result sets before calculating recall at five, mean reciprocal rank, abstention accuracy, average latency, and tokens loaded.

The included results demonstrate the file format only. They are not a performance claim. Record index settings, models, hardware, corpus revision, and missed evidence so another team can reproduce the comparison.

Start with the OKF retrieval guide. Kit guidance updated 2026-08-24.