RAGAS Explained: How to Measure Whether Your RAG System Is Working

Manual testing has a name: selection bias. You test the queries you expect the system to handle. The system fails on the queries you did not think to test. RAGAS measures four properties of a RAG system automatically — without you choosing the questions.

The Valid Failure

ShopBot's team ran four manual test queries after the first deployment. All four passed. Context Precision: not measured. Faithfulness: not measured. The team said the system was working.

Three weeks later, a customer asked about a product the team had not thought to test. The retriever returned a chunk at 0.61 — below the 0.72 threshold in production but the team had lowered it to 0.65 in dev the week before "to get better results." The model received a borderline chunk and hallucinated a price.

Four test queries is not a test. It is a performance. RAGAS is a test.

What RAGAS Measures

RAGAS evaluates four properties, each measuring a different failure mode:

The critical property: Faithfulness and Context Precision are reference-free — no labeled ground truth needed. Context Recall requires labeled ground truth (what the correct answer should be). In Book 1, the team runs Faithfulness and Context Precision first. Context Recall is added in Book 2.

A Minimal RAGAS Evaluation

from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
from datasets import Dataset

# Build your evaluation dataset
eval_data = {
    "question": [
        "Does the cotton kurta have a lining?",
        "What sizes does the silk saree come in?",
        "How do I wash the linen co-ord?",
        "Is the embroidered lehenga available in red?",
    ],
    "answer": [
        # The answers your system actually produced
        shopbot_answers,
    ],
    "contexts": [
        # The chunks your retriever returned for each question
        shopbot_contexts,
    ],
}

dataset = Dataset.from_dict(eval_data)
result = evaluate(dataset, metrics=[faithfulness, context_precision])
print(result)

To build shopbot_answers and shopbot_contexts, run each question through your pipeline and capture both the answer and the chunks that were used:

def evaluate_pipeline(questions: list[str]) -> dict:
    answers = []
    contexts = []
    for q in questions:
        chunks = retrieve(q)
        answer = generate(q, chunks)
        answers.append(answer)
        contexts.append([c["text"] for c in chunks])
    return {"answer": answers, "contexts": contexts}

What the Scores Tell You

After running RAGAS on 50 queries at the end of Book 1:

Faithfulness:      0.71
Context Precision: 0.82

Faithfulness 0.71 means 29% of claims in the answers were not directly supported by the retrieved context. The retriever was returning relevant chunks (Context Precision 0.82 — 82% of retrieved chunks were relevant). The failure was in the system prompt: the grounding constraint was too soft. The model was adding details from training memory.

This is what RAGAS produces that manual testing does not: a separable diagnosis. The retriever was not the problem. The generation constraint was.

The Evaluation Dataset

50 queries is a minimum. The dataset should include:

• Queries the system handles well (validate that good performance is maintained)
• Queries that test edge cases (partial information, missing products, ambiguous phrasing)
• Queries from real user sessions (if you have production data — these are the most valuable)

Avoid building the dataset from the same product descriptions used to construct the system prompt. That creates artificial recall. Real evaluation uses queries phrased the way users phrase them, not the way the catalog is written.

RAGAS Monday

In Book 2, the team establishes a weekly ritual: every Monday, run RAGAS on the previous week's 200 production queries. The targets:

• Faithfulness ≥ 0.85
• Context Precision ≥ 0.85
• Answer Relevance ≥ 0.80
• Context Recall ≥ 0.80 (added in Book 2 once labeled data exists)

When any metric drops below target, the Monday meeting starts with the RAGAS report. The metric identifies which component failed. The fix is scoped to that component only.

The 4-Book RAGAS Trajectory

Each improvement was identified by a drop in a specific RAGAS metric, diagnosed to a specific component, and fixed with a targeted change. No book improves all metrics by changing everything at once.

The Valid Knowledge

• Faithfulness and Context Precision require no labeled data: you can run them immediately after your first deployment, on real production queries.
• Low Faithfulness → weak system prompt: the model is speaking from training memory, not retrieved context. Tighten the grounding constraint.
• Low Context Precision → poor chunking or low threshold: too many irrelevant chunks are being retrieved. Improve chunking strategy or raise the similarity threshold.
• 50 queries minimum, 200 preferred: the score variance at 4–10 queries is too high to be meaningful.
• RAGAS is not the only metric: it does not measure latency, cost, GDPR compliance, or whether the system handles edge cases gracefully. It measures retrieval and generation quality. Both matter — but RAGAS only sees one dimension.