SKU Failure, Exact-Term Lookup, and Silent Retrieval Failure in RAG

Three weeks after ShopBot went live, Arjun pulled the failed-query log. Two hundred and sixteen rows. The first one stopped him.

"Do you have anything in K-1299?"

A customer had typed the four-character SKU of the cotton kurta into the chatbot. ShopBot returned chunks about cotton fabric, machine-washable care instructions, and the colour palette of the linen co-ord. It did not return the cotton kurta. It routed the customer to support.

The system had not hallucinated. It had refused correctly. And the customer had still left empty-handed.

The Valid Failure: 22 of Every 100 Customers

ShopBot was answering 78% of customer queries correctly after three weeks of production traffic. The dashboard was green. The model was not drifting — it was refusing to answer when it had nothing to answer from.

But Krishna stated the problem: "Twenty-two of every hundred. And we do not know which twenty-two."

The failed-query log revealed a pattern. Of the 216 failed queries:

• A customer asked "What's good for a function?" — in Indian English, a function is a wedding or social event. ShopBot returned casual wear. The catalog used festive, occasion, celebration — not function.
• A customer typed "K-1299" — the SKU of the cotton kurta. Dense retrieval found nothing, because the four characters carry almost no semantic meaning in vector space.
• A customer asked for "party-wear" — a phrase absent from the catalog.
• A customer asked for "something traditional in under two thousand" — a compound query with a price constraint.

Roughly half the 216 failures were vocabulary mismatches: the customer's word was not in the catalog, and the semantic distance was not close enough to bridge the gap. The other half were multi-intent queries, comparison questions, and out-of-scope requests.

The Valid Source: Why Dense Retrieval Cannot See Exact Terms

The dense retriever in Book 1 converted every query and every catalog chunk into a 1,536-dimensional embedding. Closeness in that space meant closeness in meaning.

A SKU is not meaning. K-1299 is a pointer — an arbitrary four-character identifier that was assigned to the cotton kurta. In the embedding space, K-1299 sits near nothing. It has no semantic neighbours. A customer typing the SKU produces a query embedding with no strong similarity to any chunk in the catalog. The retriever returns whatever is closest in vector space, which is not the product the customer named.

This is structural — not a failure of the model, not a training gap. Dense retrieval is architecturally blind to exact terms. A retriever that sees only meaning will miss the literal.

The same structural gap explains function, party-wear, and regional vocabulary. The embedding model places function near utility rather than near festive when the catalog has never used function in the context of wedding clothing. The semantic space is shaped by what is written in the corpus, not by what customers mean when they speak.

Silent Retrieval Failure: The Dashboard Is Still Green

This is the failure pattern Book 2 is named for: silent retrieval failure.

The Grounding Layer held. The model was not drifting. ShopBot did exactly what it was trained to do — it refused to answer when the retriever returned nothing relevant, and it routed the customer to support. By every metric Arjun had built — query volume, response time, fallback rate — the system was working.

The failure was earlier. The retriever had reached into the catalog and come back empty, or with the wrong evidence, and there was no signal anywhere in the system that this had happened. The customer left. The dashboard stayed green. The 22% was invisible.

A kept promise — I will not answer without evidence — is not the same as a useful answer. The Grounding Layer, in these cases, grounded the system in nothing.

The Valid Knowledge: A Second Retrieval Signal

The retriever in Book 1 had one signal: semantic similarity. It could not see the SKU. It could not see the exact word. The fix was not a better model. The fix was a second way of seeing.

BM25 (Best Match 25) is a term-frequency model. It does not understand meaning. It cannot match warm fabric to woolen material. But when a customer types K-1299, BM25 finds the chunk where the string K-1299 appears — at rank one.

The insight from Book 2 Chapter 1 is that dense retrieval and BM25 have complementary failure modes. Dense misses the literal; BM25 misses the conceptual. A retriever that uses both — hybrid search — covers the ground that neither covers alone.

The vocabulary mismatch that caused half the 216 failures closes when the retriever has two senses: one for meaning, one for words.

The remaining failures — multi-intent queries (festive and warm), multi-product comparisons, out-of-scope requests — require different fixes. Multi-query retrieval handles compound intent. Conversation memory handles follow-up questions. The 22% is not one problem. It is a taxonomy of four distinct failure modes, each requiring its own chapter of Book 2.

This article covers concepts from Book 2, Chapter 1 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.