Retrieval Threshold and Calibration in RAG
A customer asks: "Do you have anything for a winter wedding?"
ChromaDB returns the embroidered anarkali suit. Similarity score: 0.51.
ShopBot answers: "The embroidered anarkali suit in teal would be a wonderful choice for a winter wedding, offering a festive and elegant look."
The anarkali suit is a festive garment. It is appropriate for weddings. But the catalog has no winter-wedding product. The anarkali chunk does not mention winter. It does not mention cold weather. The retriever handed the model a chunk that was adjacent to the truth, not the truth itself. The model cannot tell the difference. It answered fluently from a 0.51 similarity score as though it were 0.91.
A retriever that always returns something — regardless of how relevant that something is — is not a Grounding Layer. It is a noise pipe dressed as one. The customer receives Confident Drift, resumed from retrieved noise rather than training data.
The Opposite Failure
The first instinct is to raise the threshold. If 0.51 is too low, set the minimum to 0.90. Only retrieve chunks that are genuinely close.
Testing with the threshold at 0.90:
"Something for a summer wedding." — the silk saree scores 0.87, the anarkali scores 0.84. Both are correct answers. Both are below 0.90. The retriever returns nothing. ShopBot says it does not know.
It does know. The products exist. The catalog has two excellent answers to that query. But the threshold was calibrated for certainty rather than relevance — and certainty, in a 1,536-dimensional embedding space, is rare. A score of 0.87 on a product-query match is strong evidence of relevance. Discarding it because it did not reach 0.90 is a different kind of Grounding Failure: the evidence existed, the retrieval found it, and the system threw it away.
Two failure modes. Opposite directions. Same consequence: the customer does not get the right answer.
- Threshold too low — ShopBot answers from noise. Confident Drift resumes.
- Threshold too high — ShopBot refuses to answer from signal. Evidence is wasted.
The calibration lives between them.
Finding the Right Threshold
Threshold calibration is not a formula. It is a decision made by testing queries against the specific catalog and observing where relevant results appear and irrelevant ones do not.
Testing twenty queries — a mix of product-specific questions, occasion queries, care-instruction questions, and queries that have no answer in the catalog — produces a clean pattern:
- Queries with correct top results score between 0.78 and 0.94
- Queries with incorrect top results score between 0.48 and 0.68
- There is a gap between 0.68 and 0.78 where no result lands
That gap is the threshold. Setting it at 0.75 catches every incorrect result while preserving every correct one — for this catalog, at this stage.
The working number for ShopBot in Book 1 is 0.75. It was arrived at by observation, not assumption. Every catalog has a different similarity distribution. Every deployment needs its own calibration.
Below Threshold: Honest Deflection
When the retriever finds nothing above the threshold, ShopBot does not generate an answer. It routes the customer to support:
"I don't have a product that matches that specifically. Could you describe what you're looking for differently, or contact our team at support@zudyog.com?"
This response is not a failure state. It is the Grounding Layer operating correctly — routing the customer to a human when the catalog cannot serve them. A system that admits gaps is more trustworthy than one that fills them with noise. The support email is not embarrassing. It is the honest answer.
The Retriever Is the Decision Point
Most developers treat retrieval as a search engine. Point it at the vector store, retrieve the top results, pass them to the model. Done.
That framing is wrong.
The retriever is where the Pramana Framework makes its first judgment — not what to say, but what is worth saying from. Every retrieved chunk is a claim: this evidence is relevant enough to ground the model's answer. A retriever without a threshold makes that claim about everything it returns, regardless of score. A retriever with a calibrated threshold makes that claim only when the evidence earns it.
The Grounding Layer does not retrieve everything and let the model decide what is relevant. It decides first. That decision — retrieve or withhold — is where Confident Drift is either stopped or allowed to resume. A model given bad evidence will speak from it fluently. It has no choice. The retriever is the only component in the pipeline that can prevent bad evidence from reaching the model at all.
Threshold Is Not Permanent
This threshold will need recalibration when the catalog grows. A catalog of five products has a different similarity distribution than a catalog of five hundred. As new products are added, some queries that previously returned no results will find relevant matches — and some that returned good results may compete with new, noisier candidates.
Evaluation-driven threshold calibration — using a labelled test set and RAGAS metrics — is covered in Book 2. For a first system, calibration by observation is the correct starting point: run twenty queries, record scores, find the gap, set the threshold there.
This article covers concepts from Book 1, Chapter 6 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.