Per-Route Sliced RAGAS Evaluation

The RAGAS Monday eval ran the same way it had run every week since Book 2 Chapter 8. Same labelled dataset. Same four metrics. Same script.

The numbers came back: F 0.88 / AR 0.88 / CP 0.91 / CR 0.86.

Arjun pasted them into the spreadsheet and was about to close the tab when he noticed something he had been not-quite-noticing for two weeks.

The single number was a blend. And the blend was hiding something.

The Valid Failure: A Blended Score Across Five Different Sub-Systems

Through Book 2, the pipeline had been a chain. Every query went through every step. The aggregate F-score was a fair grade because every query was graded on the same architecture.

Through Book 3, the pipeline had become a graph. Different queries took different routes through seven nodes, two conditional edges, and three terminal paths. A vague query went through HyDE first, then standard retrieval against the hypothetical, then CRAG. A specific query skipped HyDE and went straight to retrieval. A type-mismatch query triggered CRAG re-retrieval with a sub-query. A query with no usable evidence routed to fallback without reaching the model.

Five routes. One blended average score. The average was hiding what the routes were actually doing.

How a blended score can mislead in three different ways:

It moves up if every route improved.

It moves up if a small subset of routes improved sharply while others regressed mildly — the improvement on the smaller subset, weighted up by traffic share, dominates. The system is worse on most queries; the headline says better.

It moves down in the inverse — most routes improved, one regressed sharply, the regression dominated. The system is better on most queries; the headline says worse.

Looking at a single F-score of 0.88, there was no way to know which case was true.

The Valid Source: Tagging Routes and Scoring Each Separately

The LangGraph pipeline already wrote routing decisions into the state object. The is_vague field recorded whether HyDE had fired. The crag_verdict field recorded whether CRAG had passed through, re-retrieved, or routed to fallback. Together those two fields uniquely identified the path each query had taken.

def derive_route(state: PipelineState) -> str:
    if state.get("crag_verdict") == "IRRELEVANT":
        return "E_fallback"
    hyde = state.get("is_vague", False)
    crag = state.get("crag_verdict", "RELEVANT")
    if not hyde and crag == "RELEVANT":
        return "A_standard_relevant"
    if not hyde and crag == "PARTIAL":
        return "B_standard_partial"
    if hyde and crag == "RELEVANT":
        return "C_hyde_relevant"
    if hyde and crag == "PARTIAL":
        return "D_hyde_partial"
    return "X_unknown"

The evaluation script ran each labelled query through the actual pipeline, captured the final state, derived the route, and grouped queries by route before scoring. Routes with fewer than five queries were marked scored=False — RAGAS metrics on three or four samples carry noise too large to act on.

Per-route RAGAS results (week 6, post-INT8):

The blended number was exactly the 0.88 Arjun had pasted into the spreadsheet. The per-route numbers were not.

Route A — specific queries, clean CRAG pass-through — was the strongest: F 0.92, CP 0.94. The model's behaviour on this route was as good as ShopBot had ever been.

Route D — the compound case, vague query through HyDE, then CRAG ruling PARTIAL — was the weakest: F 0.79, CR 0.74. Both meaningfully below the blended number. The blend had hidden it because Route D was only 9 queries out of 90.

Route E — fallback — had no RAGAS scoring because the answer was a deterministic support-routing message.

The Valid Knowledge: Making the Weakness Visible and Targetable

Route D was 9 queries out of 90 in the labelled dataset — which extrapolated to roughly 9–10% of production traffic — about a thousand queries a week at current load, fifty thousand a week at projected load. A 0.79 F-score on five thousand queries a week was a different shape of failure than a 0.79 F-score on nine queries in a test set.

The compound failure mode in Route D: A vague query went through HyDE. HyDE generated a hypothetical. The retriever retrieved against the hypothetical. CRAG judged the result PARTIAL — the retrieved context was close but not quite right. The sub-query for re-retrieval was generated from a HyDE-rewritten anchor that may not have been the right anchor in the first place. Two layers of indirection. The merged final context was sometimes off-target.

RAGAS Monday extended. The shared spreadsheet got a second tab called Per-Route: one row per (Monday × route). The blended row stayed on the first tab as the headline. Per-route rows got watched independently.

The Route D entry got a WATCH annotation and a target: F ≥ 0.85 by week 9.

The fix (week 8, 16 lines across 2 files):

1. HyDE prompt updated: ask for a hypothetical naming one concrete product category, not two — fewer hallucinated specifics, less drift in the resulting embedding.
2. CRAG sub-query prompt updated: when CRAG judged PARTIAL on a HyDE-routed retrieval, give it the original raw customer query as context (not the HyDE-rewritten one) — restoring the original anchor before corrective re-retrieval.

Week-8 per-route results after fix:

The blended F went up one point. The Route D F went up seven points. The headline had moved a little. The route had moved a lot.

The per-route view did not solve the problem. It made the problem visible, named it, and gave it a target. The fix followed from reading the nine Route D queries one by one and finding the pattern. A blended score can never surface what a route-level view surfaces — and at 5× traffic, that difference is measured in thousands of customers per week.

This article covers concepts from Book 3, Chapter 7 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.