What is HyDE? Hypothetical Document Embeddings in Advanced RAG

HyDE generates a hypothetical answer to a vague query, then uses that answer as the retrieval query. It works because the answer space — what a good answer to this question looks like — is often far easier to search than the question space. The hypothetical answer lives near the actual answer in the embedding space, even if the original question does not.

The 9% Problem

After implementing hybrid retrieval and re-ranking, ShopBot answered 91 of every 100 customer queries correctly. The remaining 9% were not random — they clustered into four discrete failure types. One of them was vague queries.

A query like "something nice for the weekend" has no strong token to match and no clear semantic anchor. Dense retrieval returns semantically reasonable results — summer casuals, festive wear, comfortable fabrics — but none of them are specifically on-target. The model answers, but the answer is generic. The customer clicks away.

The problem is asymmetry between the question and the answer. The question "something nice for the weekend" is vague. But a good answer — "the linen co-ord set in sage would be ideal for a relaxed weekend outing" — is specific and highly searchable. If we could retrieve against the answer instead of the question, the retrieval would work.

HyDE makes this possible.

How HyDE Works

HyDE adds one step before retrieval:

The user's query arrives: "something nice for the weekend"
An LLM generates a hypothetical answer — not a final response, but a plausible example of what a good answer might look like: "For a weekend, I would recommend something relaxed and breathable — a linen or cotton piece in a neutral or soft colour, something you can wear for brunch or a casual outing without it looking underdressed."
This hypothetical answer is embedded and used as the retrieval query instead of the original question.
The retrieved chunks (actual product entries) are passed to the LLM for final generation.

The hypothetical answer sits closer to the actual product chunks in the embedding space than the vague question does. Retrieval precision improves significantly for queries that would otherwise underperform.

Why It Works

The embedding model places texts with similar meaning near each other. A hypothetical answer describing a relaxed linen outfit is much closer to a product description that says "linen co-ord set, relaxed fit, summer casual" than the question "something nice for the weekend" is.

The LLM generating the hypothetical answer does not need to be correct — it needs to be plausible and on-topic. The hypothetical answer is never shown to the user. It exists only to steer retrieval toward the right region of the semantic space.

When to Use HyDE

HyDE helps when:

Queries are short and vague ("something casual", "good for evenings")
The query does not contain enough specific terminology to anchor dense retrieval
Users describe intent or occasion rather than product attributes

HyDE does not help when:

Queries are already specific ("navy kurta under ₹1500")
The vocabulary gap is the problem — BM25 in hybrid search handles that better
Latency is critical — HyDE adds an extra LLM call before retrieval

HyDE in the Pipeline

In Book 3, HyDE is implemented as one route in a LangGraph pipeline. The router classifies incoming queries: specific queries go through standard hybrid retrieval; vague queries are routed through HyDE. This means the extra LLM cost of HyDE is only paid for the queries that need it.

Query
  → Router (specific / vague / type-mismatch / multi-turn)
  → [if vague] HyDE: generate hypothetical answer → embed → retrieve
  → [if specific] Standard hybrid retrieval
  → Cross-encoder re-rank
  → Top 3 to LLM context

This routing architecture — where different query types take different paths through the pipeline — is what distinguishes a Book 2 system (a chain with better components) from a Book 3 system (a graph that chooses its own path).

Book 3 of the RAG Mastery Series implements HyDE, CRAG (Corrective RAG for type-mismatch failures), LangGraph routing, and fine-tuned embedding models — taking the 9% failure rate down to the sub-5% range where remaining failures are ones the system deliberately routes to human support.