RAGAS: Reference-Free Metrics, Answer Relevance, Context Recall, and RAGAS Monday
Arjun had been writing two numbers on the whiteboard across four chapters of Book 2. Faithfulness. Context precision. Both from the RAGAS evaluation harness he had built in Book 1. Both had moved in the right direction. Both told an incomplete story.
It took him most of a morning to admit, even to himself, what the incompleteness was.
The two metrics he had been quoting in every status update were precisely the two metrics that did not ask him to write down what "correct" would have looked like.
The Valid Failure: Reference-Free Metrics Grade the System Against Itself
Faithfulness measures whether the model's answer is grounded in the retrieved chunks. It compares the model's claims against the retrieved context — not against what the customer actually needed. It does not require a ground-truth answer. It runs on traffic alone.
Context precision measures whether the retrieved chunks are relevant to the query. It uses the model itself to judge relevance. It also does not require ground-truth answers.
Both are what RAGAS calls reference-free metrics. They run on live traffic without anyone first writing down what the right answer was supposed to be. That was their virtue in Book 1, when the labeled dataset did not exist and shipping was the priority. It was also their limit: reference-free means the system grades itself.
The failure this hides — the one context precision cannot see — is a retriever that returned relevant-seeming but incomplete chunks. The correct answer might require three chunks; the retriever returned two. Faithfulness says: yes, the model spoke from what it received. Context precision says: the chunks it received were relevant. Neither says: it missed the one chunk that would have made the answer complete.
Context recall is the metric that catches this case. It requires ground truth.
The Book 1 threshold retirement. The 0.75 dense-similarity threshold from Book 1 Chapter 6 had been set against 20 hand-picked queries on a dense-only retriever. With hybrid retrieval and cross-encoder reranking now in place, the threshold's role had changed — the cross-encoder produces scores on a different scale; RRF-fused ranks are not similarity scores at all. Running the labeled dataset revealed the correct calibration: a 0.55 cross-encoder confidence floor (if no candidate scores above 0.55, refuse and route to support). This correctly refused 12 queries the previous architecture had been answering from weak evidence.
The Valid Source: What the Labeled Dataset Unlocked
Answer relevance asks whether the model's answer addressed the customer's question — not whether it was grounded, but whether it answered the right thing. Requires ground truth.
Context recall asks whether the retriever retrieved everything needed to answer correctly, judged against a known-correct answer. A score of 0.79 means that for 21% of evaluation queries, the retriever failed to surface at least one chunk the correct answer depended on.
Building the labeled dataset took two days. The 100 queries were stratified deliberately:
- 20 queries ShopBot had answered correctly on the first attempt.
- 20 that eventually succeeded after architectural changes.
- 20 that still failed.
- 20 multi-turn conversations.
- 20 drawn at random from the unsorted production log (to avoid biasing toward problems already known).
For each query: a reference answer (what an ideal ShopBot response would say, drawing only on catalog facts), a list of gold chunk ids (which chunks should have been retrieved), and for ambiguous queries, a list of acceptable alternative answers.
{"id":"q001","query":"K-1299 in mint?","reference_answer":"The cotton kurta (K-1299) is available in mint green among the colours offered, in sizes S to XXL, priced at ₹1,299.","gold_chunk_ids":["K-1299:colours","K-1299:price","K-1299:sizes"],"strata":"sku_lookup"}
{"id":"q014","query":"What's good for a function?","reference_answer":"For a function, the silk saree with zari border (₹4,299) and the Anarkali suit with dupatta (₹3,499) are festive options in the catalog.","gold_chunk_ids":["S-4299:overview","A-3499:overview"],"strata":"vocabulary_gap"}
{"id":"q082","query":"What's your return policy?","reference_answer":"I don't have catalog information about return policies. For policy questions, please contact our team at support@zudyog.com.","gold_chunk_ids":[],"strata":"out_of_scope"}
The dataset must come, in part, from the failed-query log. A dataset built only by the developer grades the developer's blind spots leniently.
The Valid Knowledge: Four Scores and a Weekly Ritual
Running the full RAGAS evaluation against the system at the end of Chapter 6:
| Metric | Score |
|---|---|
| Faithfulness | 0.83 |
| Answer relevance | 0.81 |
| Context precision | 0.88 |
| Context recall | 0.79 |
The first two numbers were familiar — confirming what earlier RAGAS runs had shown. The fourth was new: 21% of evaluation queries were missing at least one chunk the correct answer depended on. The multi-product queries (show me a saree and matching jewellery) were the largest slice of context-recall failures — the retriever had returned one product but missed the second. Neither faithfulness nor context precision had been able to see this.
RAGAS Monday. Every Monday morning, a cron job ran the full evaluation against the production system and appended a row to a shared spreadsheet:
def post_monday_row(notes: str = "") -> None:
scores = run_evaluation("eval/labeled_dataset.jsonl")
sheet = gspread.service_account().open("zUdyog ShopBot").sheet1
sheet.append_row([
date.today().isoformat(),
scores["F"], scores["AR"], scores["CP"], scores["CR"],
notes,
])
The first row, dated the third Monday of January 2026:
F: 0.83 · AR: 0.81 · CP: 0.88 · CR: 0.79 · changes: hybrid + rerank + memory + cache · note: first labeled-dataset run; baseline locked.
Above the headers, in the cell Krishna would see:
RAGAS Monday — every week, same dataset, same questions. The system grades itself against the truth, not against itself.
A system that measures itself this way is a system that can be improved against a standard. A system that does not is a system that drifts in whichever direction is easiest to celebrate. The labelling is the price of trust. Monday is when the price gets paid.
This article covers concepts from Book 2, Chapter 8 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.