Hybrid Search & Cross-Encoder Reranking
Vector search alone struggles with acronyms, part numbers, and exact keywords. Sparse search (BM25) struggles with synonyms and intent. Hybrid search + Reranking gives you the best of both worlds.
- •Combine Dense (Vector) and Sparse (BM25) search using Reciprocal Rank Fusion (RRF).
- •Implement Cross-Encoder reranking (Cohere, BGE-Reranker, ColBERT) to eliminate false-positive vector hits.
- •Optimize latency budget for sub-100ms multi-stage retrieval pipelines.
In-Depth Architecture & Mechanics
The Two-Stage Retrieval Funnel
Stage 1: **Candidate Generation (High Recall)** - Fetch Top 50 via Dense Vector Search (semantic similarity). - Fetch Top 50 via BM25 (exact keyword match). - Combine using **Reciprocal Rank Fusion (RRF)**: $$RRF\_Score(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$ where $k \approx 60$.
Stage 2: **Precision Reranking (High Precision)** - Feed (Query, Candidate Doc) pairs into a **Cross-Encoder**. - Unlike Bi-Encoders which compute independent embeddings, Cross-Encoders pass query and document tokens jointly into self-attention. - Select Top 5 reranked docs for final LLM prompt context.
Query ──┬──> [ Dense Vector Search (Milvus/Qdrant) ] ──> Top 50 ──┐
│ ▼
└──> [ Sparse Search (BM25 / Elasticsearch) ] ───> Top 50 ──> [ RRF Fusion ]
│ Top 30
▼
[ Cross-Encoder Reranker ]
│ Top 5
▼
[ LLM Prompt ]def reciprocal_rank_fusion(dense_results: list, sparse_results: list, k: int = 60):
rrf_scores = {}
for rank, doc_id in enumerate(dense_results):
rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k + rank + 1))
for rank, doc_id in enumerate(sparse_results):
rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k + rank + 1))
return sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)