Why Your RAG System Retrieves the Wrong Chunks (And How to Fix It)

Wrong retrieval produces wrong answers. The failure is not always obvious — the system answers confidently from the wrong chunk, and the answer often sounds plausible. Here are the five most common retrieval failure modes, how to identify each, and what fixes them.

The Valid Failure

After ShopBot's first production week, RAGAS Context Precision was 0.74. That means 26% of retrieved chunks were not relevant to the query. The model was generating answers from chunks that had been retrieved by accident — not because they were the right chunks, but because they scored above the threshold.

The team looked at five failed queries. Each had a different root cause.

Failure Mode 1: The Blob Problem (Over-Large Chunks)

What it looks like: One chunk covers multiple concerns. A product chunk contains fabric, sizing, care, and price. The query is about care instructions. The chunk's vector averages across all four concerns. The care signal is diluted.

How to identify it: Retrieve the chunk and count how many distinct topics it covers. If more than two, it is a blob.

Fix: Chunk by concern. One chunk per attribute cluster, with a product anchor in each.

# Before (blob)
chunk = f"{product['name']}: {product['fabric']}. {product['care']}. Sizes: {product['sizes']}. Price: ₹{product['price']}."

# After (by concern)
chunks = [
    f"{product['name']} — Fabric: {product['fabric']}",
    f"{product['name']} — Care: {product['care']}",
    f"{product['name']} — Sizing: {', '.join(product['sizes'])}",
]

Failure Mode 2: Context Loss (Over-Small Chunks)

What it looks like: Each sentence is its own chunk. A care instruction chunk says "Hand wash in cold water." It retrieves correctly for washing queries. But the model's answer does not know which product — the product name was in the previous chunk.

How to identify it: Read a retrieved chunk in isolation. Can you answer the user's question from it without needing the surrounding document? If not, it is too small.

Fix: Each chunk must contain the anchor (product name, SKU, document title) plus the specific concern. A chunk without an anchor is a fragment.

Failure Mode 3: Threshold Too Low (Noise Retrieval)

What it looks like: Chunks score 0.55–0.65 but are returned because the threshold is set to 0.50. The model generates from these chunks — and the low-relevance context produces hallucination.

How to identify it: Log retrieval scores for every query. If the minimum score in your results regularly falls below 0.65, your threshold is too permissive.

Fix: Calibrate the threshold by running 50–100 representative queries, logging the scores, and identifying the score below which retrieved chunks are consistently irrelevant. Set the threshold 5 points above that value.

# Audit retrieval scores in production
results = client.search(
    collection_name=COLLECTION,
    query_vector=vector,
    limit=10,  # retrieve more than you return
    score_threshold=0.50,  # temporarily lower to see what's in range
)
for r in results:
    print(f"Score: {r.score:.3f} | Text: {r.payload['text'][:80]}")

Failure Mode 4: Vocabulary Gap (Semantic Miss)

What it looks like: A customer uses a term that does not appear in the catalog and is not in the embedding model's training distribution. "K-1299" (a SKU), "Supplex fabric" (a brand term), "mirrorwork" vs "mirror work" (spacing difference).

How to identify it: RAGAS Context Recall below 0.75 with correct threshold calibration is a signal. Also: identify queries that return zero results — many of them will be exact-identifier lookups.

Fix: Add BM25 sparse retrieval alongside dense retrieval. BM25 token-matches exact identifiers regardless of embedding distribution.

# Hybrid: dense + BM25 sparse, merged with RRF
results = client.query_points(
    collection_name=COLLECTION,
    prefetch=[
        Prefetch(query=dense_vector, using="dense", limit=20),
        Prefetch(query=sparse_vector, using="sparse", limit=20),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=5,
)

Failure Mode 5: Wrong Ranking (Correct Chunks, Wrong Order)

What it looks like: The correct chunk is in the top 5 results. It is ranked 4th. The model receives chunks 1–3 and generates from the wrong evidence. The correct answer was retrieved but not used.

How to identify it: Log the retrieval results and track whether the correct chunk appeared but was not ranked first. If the correct chunk appears in the top 5 but not the top 3 more than 15% of the time, ranking is the problem.

Fix: Add a cross-encoder reranker. It reads each candidate in the context of the query and produces a contextual relevance score that is more accurate than cosine similarity alone.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query: str) -> list[dict]:
    candidates = retrieve(query, top_k=20)
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, _ in ranked[:3]]

Debugging Retrieval Systematically

A retrieval audit for any failing query:

def audit_retrieval(query: str):
    raw = client.search(
        collection_name=COLLECTION,
        query_vector=embed_model.encode(query).tolist(),
        limit=10,
        score_threshold=0.50,  # low threshold to see full range
    )
    print(f"Query: {query}")
    print(f"Results: {len(raw)}")
    for i, r in enumerate(raw):
        print(f"  [{i+1}] score={r.score:.3f} | {r.payload['text'][:100]}")

Run this for every query that produces a wrong or empty answer. The pattern across 10–20 queries will identify which failure mode is dominant.

The Valid Knowledge

• Context Precision is your diagnostic metric: it measures the fraction of retrieved chunks that were relevant. Below 0.80 means more than 20% of chunks passed to the LLM are noise.
• Most retrieval failures have a single root cause: identify the dominant failure mode before fixing anything. Fixing blob chunking when the problem is threshold calibration wastes time and makes diagnosis harder.
• Log retrieval scores in production: you cannot debug what you cannot see. Every production query should log the top-3 retrieval scores alongside the answer. Patterns become visible within days.
• Re-ingestion after chunking fixes is fast: re-chunking and re-embedding an 850-product catalog takes under two minutes. Do not hesitate to re-ingest — the quality improvement is permanent.
• The five failure modes compound: a system with blob chunks, a low threshold, and no reranker may show all five failures simultaneously. Fix chunking first, then threshold, then add hybrid retrieval, then add reranking. Each fix reduces noise in the next diagnostic.