Beta Testing, Vocabulary Gaps, and Re-ingestion in RAG

Four days after the Railway URL went live, Arjun sent it to three beta testers with one instruction: use it as a real customer would. Ask what you would actually ask. Do not be kind about what breaks.

Two testers responded with careful queries that ShopBot handled without difficulty.

The third tester was Meera.

Meera runs a boutique in Pune selling ethnic wear to a customer base that knows exactly what it wants and expects to be told clearly whether they can get it. She does not frame problems as suggestions.

Her first message arrived Tuesday evening: "Tested it. One question."

The Query That Started Everything, Asked Again

Meera's query: "Does the Anarkali suit have a cotton lining?"

ShopBot's answer: "I don't have information about the lining material for the Anarkali suit. For specific fabric details, please contact our team at support@zudyog.com."

In Chapter 1, thirty-one support tickets. The first ticket: a customer who asked whether a kurta had a cotton lining. The chatbot replied confidently — "Yes, this kurta features a soft cotton lining for added comfort" — and invented a feature that did not exist. A return filed. A refund.

Meera asked the same shape of question, ten chapters later. ShopBot admitted it did not know. Routed her to support. Did not invent a lining.

The question that broke the old system proved the new one works.

What Real Users Find That Developers Do Not

Two other beta testers found nothing that failed. Meera found three failures.

Failure one — the vocabulary gap.

"What's good for a function?"

In Indian English, function means a social event — a wedding, a reception, a celebration. It is the natural word a Pune customer uses. The catalog uses festive, occasion, celebration — not function.

The retriever does not know this. It converts the query to a vector and searches for the closest semantic match. Function without context pulls toward utility rather than festivity. The retriever returns the linen co-ord set at 0.71, tagged as casual wear.

ShopBot recommends casual summer wear to a customer shopping for a wedding.

Failure two — the multi-intent query.

"Show me something festive and warm."

The retriever converts the entire query into a single vector. The combined vector of festive and warm lands somewhere between the festive chunks and the winter chunks — close to neither. The silk saree returns at 0.79 for festivity. The woolen shawl returns at 0.74 for warmth. But the query asked for both simultaneously. Neither product fully answers it.

Failure three — the out-of-scope slip.

"What jewellery goes with the saree?"

zUdyog does not sell jewellery. The correct answer is the fallback. On Meera's first phrasing, the saree chunk scores 0.62 — below the 0.75 threshold. Correct fallback.

She rephrases: "jewellery for the silk saree". The score rises to 0.77. The retriever returns the saree chunk as if it answers a jewellery question. It does not. The model receives a saree description and attempts to answer a styling question from it.

The Fix for Each

Vocabulary gap: re-ingestion, not a smarter model.

The fix for the function vocabulary gap is not in the model. It is not in the retriever. It is in the chunk metadata.

Adding function as an occasion synonym to the festive-product chunks brings those products into the right semantic neighbourhood:

# data/products.py — update occasion field for festive products
"occasion": "festive, wedding, formal, function",   # p003 Silk Saree
"occasion": "festive, wedding, formal, function",   # p005 Anarkali Suit

Then delete the existing index and rebuild it — re-ingestion:

rm -rf chroma_db/
python src/ingest.py

After re-ingestion, the silk saree and the anarkali suit are in the semantic neighbourhood of function queries. The linen co-ord set drops out of the top results.

A vocabulary gap, closed by data. Not by a smarter model.

This is the re-ingestion pattern: when the data or the structure changes, the vector index is rebuilt from scratch. The existing index holds the old embeddings. There is no partial update — the new chunks get new embeddings, and the index must reflect them completely. Delete, re-embed, re-index.

Multi-intent query: honest acknowledgement until Book 2.

The correct solution — multi-query retrieval, splitting the query into sub-queries and merging results — is Book 2's work. For now, ShopBot handles it with the best single-vector result and acknowledges the limitation: "The silk saree is a festive option, though it is not specifically designed for warmth. For winter events, you may want to layer it — or contact us at support@zudyog.com for styling advice."

Not perfect. Honest.

Out-of-scope slip: one sentence in the system prompt.

One line added inside the "when you do not have the information" block: if the context describes a product but does not address the type of question being asked, say you do not have that information and direct the customer to support.

The model now reads the saree chunk, recognises that the query is about jewellery and the chunk is about a saree, and routes correctly.

Three Failures, Three Lessons

Three failures. Three fixes. None requiring a rewrite of the core architecture. The Grounding Layer held. The failures were at the edges.

The developer's twenty test queries did not surface any of these. Meera found all three on her first session. This is selection bias made concrete — the developer tests what the system was built to handle. The beta tester asks what a real customer would ask.

The vocabulary gap failure is the one that matters most for what comes next. The word function does not appear in the catalog — not because it is wrong, but because the person who wrote the product descriptions did not think in those terms. Real customers think in different terms than catalog writers. This gap — between the vocabulary of the user and the vocabulary of the data — does not close by improving the model. It closes by improving the data.

Every vocabulary gap found in beta testing is a data problem, not a model problem. The fix is to identify the word, add it where it belongs in the catalog, and re-ingest. The index reflects the new data. The retriever finds what it previously could not reach.

This is what beta testing is for. Not to check whether the happy path works — the developer already knows the happy path works. To find the vocabulary your users speak that your data does not.

This article covers concepts from Book 1, Chapter 10 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.