Cross-Encoder vs Bi-Encoder: What's the Difference and When to Rerank

The linen co-ord was the right answer. It ranked third. ShopBot recommended the saree instead. The customer bought the linen co-ord from a competitor. The retriever did not fail — it returned the correct answer in the top 3. The ranker failed.

The Valid Failure

After hybrid search was added in Book 2, retrieval coverage improved significantly. The correct chunk was appearing in the top 5 results for most queries. But "in the top 5" is not the same as "ranked first."

For the query "light cotton outfit for a beach wedding," three chunks scored between 0.78 and 0.82:

1. Silk georgette saree — score 0.82
2. Embroidered cotton kurta — score 0.80
3. Linen co-ord set — score 0.78

The saree was ranked first by similarity score. The linen co-ord — the correct answer for a beach wedding — was third. The similarity scores were close enough that the ranking order was determined by noise, not by the semantic relationship between the query and each individual candidate.

The bi-encoder (the embedding model) encoded the query and each chunk independently. It never read them together.

How Bi-Encoders Work

A bi-encoder (the architecture used by sentence-transformers and most embedding models) encodes the query into a vector and each document into a vector — independently. Similarity is measured as cosine distance between the two vectors.

query → [encoder] → query_vector
doc_1 → [encoder] → doc_1_vector  → cosine(query_vector, doc_1_vector) = 0.82
doc_2 → [encoder] → doc_2_vector  → cosine(query_vector, doc_2_vector) = 0.80
doc_3 → [encoder] → doc_3_vector  → cosine(query_vector, doc_3_vector) = 0.78

This is fast — each vector is computed once and stored. At query time, only the query is encoded; the document vectors come from the index. But the model never sees the query and a document together. It cannot reason about their specific relationship.

How Cross-Encoders Work

A cross-encoder takes the query and one document as a single input and outputs a relevance score. It reads them together.

[query + doc_1] → [cross-encoder] → relevance_score = 0.41
[query + doc_2] → [cross-encoder] → relevance_score = 0.78
[query + doc_3] → [cross-encoder] → relevance_score = 0.86

For the beach wedding query, the cross-encoder read each candidate in the context of the query. It understood that linen is more appropriate for a beach setting than silk georgette. The bi-encoder's similarity scores were global — averaged across all the words in both texts. The cross-encoder's scores were contextual — specific to this query and this candidate.

The Trade-Off

This is why reranking exists as a two-stage pipeline, not as a replacement for bi-encoder retrieval.

Two-Stage Retrieval

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query: str, top_k_retrieve: int = 20, top_k_return: int = 3):
    # Stage 1: fast bi-encoder retrieval (wide net)
    candidates = retrieve(query, top_k=top_k_retrieve)

    # Stage 2: cross-encoder reranking (precise)
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)

    # Sort by cross-encoder score, return top k
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in ranked[:top_k_return]]

Stage 1 retrieves 20 candidates using the fast bi-encoder index. Stage 2 runs the cross-encoder on all 20 pairs — expensive enough to preclude at-scale use, but applied to only 20 candidates, not the full catalog.

What This Fixed in ShopBot

After adding the cross-encoder reranker:

The linen co-ord moved to rank 1. The saree dropped to rank 3 — the cross-encoder correctly identified that silk georgette is a poor match for a beach wedding query.

Context Precision improved from 0.82 to 0.88 across the full RAGAS evaluation set after adding reranking. The retriever was finding the right chunks. The reranker was surfacing them.

Choosing a Cross-Encoder Model

For ShopBot (fashion e-commerce, Indian market, English queries), ms-marco-MiniLM-L-6-v2 produced a consistent improvement with acceptable latency. The reranker added ~80ms to p50 query time — acceptable for a 300ms total budget.

The Valid Knowledge

• The reranker does not replace the retriever: it runs after retrieval on a small candidate set. You need the bi-encoder to narrow the search space; you need the cross-encoder to get the ranking right within that space.
• Retrieve wide, rerank narrow: retrieve 15–25 candidates, rerank, return 3–5. The quality of the top-3 after reranking is significantly better than the top-3 from bi-encoder alone.
• Latency impact is bounded: running a cross-encoder on 20 short passages is ~80–120ms on CPU. This is predictable and measurable.
• Context Precision is the right RAGAS metric to watch: it measures the fraction of retrieved chunks that were relevant. If reranking is working, the top chunks passed to the LLM are higher quality — which is exactly what Context Precision captures.