Overview
Level 2: Build · Module 5: Embeddings and Vector Search · 25 min read
Why this matters
Neither keyword search (BM25) nor vector search (dense embeddings) is sufficient on its own for production search engines. Dense vector search excels at conceptual matching, answering queries like "how do I prevent SQL injection in Next.js?" even if the target article uses the term "parameterized prepared statements". However, vector search frequently fails on exact strings, acronyms, product SKUs, error codes (ERR_SOCKET_TIMEOUT), or code function names (useSyncExternalStore), where exact character matches are required. Hybrid Search combines sparse lexical retrieval (BM25 / PostgreSQL full-text search) with dense semantic vector retrieval using Reciprocal Rank Fusion (RRF), delivering state-of-the-art search relevance across every query type.
Learning objectives
- Diagnose why pure vector search fails on exact lexical keywords and technical jargon.
- Implement sparse full-text search alongside dense vector search in PostgreSQL.
- Understand the mathematical mechanics of Reciprocal Rank Fusion (RRF).
- Combine, normalize, and de-duplicate ranked candidate lists in TypeScript.
- Build an end-to-end Hybrid Search pipeline that outperforms single-modality systems.
The Two Modalities Compared
graph TD
UserQuery["User Query: 'fix Next.js CVE-2024-1184 SSR vulnerability'"]
subgraph SparseBranch["Sparse Search Branch (Lexical BM25 / tsvector)"]
UserQuery --> BM25["Full-Text Lexical Index"]
BM25 -->|"Matches exact token 'CVE-2024-1184'"| ResBM25["Rank 1: Security Patch Release Note"]
end
subgraph DenseBranch["Dense Search Branch (Semantic Vector)"]
UserQuery --> VecEmbed["Embedding Model"]
VecEmbed --> VecIndex["Vector Index (pgvector / HNSW)"]
VecIndex -->|"Matches semantic concept 'Server Side Rendering Flaw'"| ResVec["Rank 1: General SSR Best Practices"]
end
ResBM25 --> RRF["Reciprocal Rank Fusion (RRF)"]
ResVec --> RRF
RRF --> Final["Consolidated Winner: Accurate Security Release Note with Context"]The Mathematics of Reciprocal Rank Fusion (RRF)
How do you combine a BM25 relevance score (which ranges from $0$ to $+50$) with a Cosine Similarity score (which ranges from $0.0$ to $1.0$)? You cannot simply add them together, because the differing score distributions will cause one modality to dominate the other.
Reciprocal Rank Fusion (RRF) solves this by disregarding raw scores entirely and operating strictly on ordinal rank positions:
$$RRF_Score(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$
Where:
- $M$ is the set of retrieval systems (e.g., Sparse BM25 and Dense Vector).
- $r_m(d)$ is the 1-based rank position of document $d$ in system $m$.
- $k$ is a smoothing constant, typically set to 60 (empirically derived by Cormack, Clarke, and Büttcher).
Why $k = 60$?
The smoothing factor $k=60$ prevents top-ranked items in one system from overwhelmingly drowning out consistent top-10 appearances in both systems:
- Rank 1 item receives: $\frac{1}{60 + 1} = 0.01639$
- Rank 2 item receives: $\frac{1}{60 + 2} = 0.01612$
- An item ranked #3 in BM25 AND #3 in Vector Search scores: $\frac{1}{63} + \frac{1}{63} = 0.0317$, easily beating an item that was #1 in only one modality!
TypeScript Implementation: Reciprocal Rank Fusion
Here is the exact algorithmic implementation for fusing two search result streams:
// lib/ai/rrf.ts
export interface ScoredDocument {
id: string;
content: string;
metadata?: Record<string, any>;
}
export interface FusedResult extends ScoredDocument {
rrfScore: number;
sparseRank?: number;
denseRank?: number;
}
/**
* Merges sparse (keyword) and dense (vector) search result sets
* using Reciprocal Rank Fusion with smoothing factor k=60.
*/
export function reciprocalRankFusion(
sparseResults: ScoredDocument[],
denseResults: ScoredDocument[],
k: number = 60
): FusedResult[] {
const scoreMap = new Map<string, FusedResult>();
// 1. Process sparse keyword rankings
sparseResults.forEach((doc, index) => {
const rank = index + 1; // 1-indexed
const contribution = 1 / (k + rank);
if (!scoreMap.has(doc.id)) {
scoreMap.set(doc.id, {
...doc,
rrfScore: contribution,
sparseRank: rank,
});
} else {
const existing = scoreMap.get(doc.id)!;
existing.rrfScore += contribution;
existing.sparseRank = rank;
}
});
// 2. Process dense vector rankings
denseResults.forEach((doc, index) => {
const rank = index + 1;
const contribution = 1 / (k + rank);
if (!scoreMap.has(doc.id)) {
scoreMap.set(doc.id, {
...doc,
rrfScore: contribution,
denseRank: rank,
});
} else {
const existing = scoreMap.get(doc.id)!;
existing.rrfScore += contribution;
existing.denseRank = rank;
}
});
// 3. Sort by combined RRF score descending
return Array.from(scoreMap.values()).sort((a, b) => b.rrfScore - a.rrfScore);
}Full Hybrid Search in PostgreSQL (Single SQL Query)
You can execute both sparse full-text search and dense vector search in a single CTE in PostgreSQL:
-- Single SQL Query executing Hybrid Search via RRF in Postgres
WITH
-- 1. Full-text search using English text search vectors
keyword_search AS (
SELECT id, content, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC) AS rank
FROM document_chunks, plainto_tsquery('english', $1) query
WHERE to_tsvector('english', content) @@ query
LIMIT 20
),
-- 2. Semantic vector search using pgvector
vector_search AS (
SELECT id, content, ROW_NUMBER() OVER (ORDER BY embedding <=> $2) AS rank
FROM document_chunks
ORDER BY embedding <=> $2
LIMIT 20
)
-- 3. Reciprocal Rank Fusion merge
SELECT
COALESCE(k.id, v.id) AS id,
COALESCE(k.content, v.content) AS content,
COALESCE(1.0 / (60 + k.rank), 0.0) + COALESCE(1.0 / (60 + v.rank), 0.0) AS rrf_score
FROM keyword_search k
FULL OUTER JOIN vector_search v ON k.id = v.id
ORDER BY rrf_score DESC
LIMIT 5;Interview Angle: How to Answer in System Design
- Question: "How would you design a search engine for a developer documentation portal like Stripe or Next.js to handle both conceptual queries and exact error codes?"
- Target Answer:
"I deploy a Hybrid Search architecture combining sparse lexical search (Postgres
tsvectoror BM25) and dense semantic vector search (pgvectorwith HNSW).When a developer searches for an exact error string like
NEXT_NOT_FOUNDor a specific API parameter, sparse lexical search guarantees an exact 100% recall match. When they ask conceptual queries like 'how do I handle auth across edge middleware', dense vector search captures the semantic intent.We merge both candidate pools using Reciprocal Rank Fusion (RRF) with a smoothing factor of $k=60$. Because RRF operates on normalized ordinal ranks rather than uncalibrated raw distance floats, documents surfaced by both systems rise to the top naturally without manual score weighting."
Quick Check
- Why does pure vector search often fail on exact error codes and package versions?
- Why can't we simply add BM25 scores and Cosine Similarity scores together directly?
- What is the standard smoothing constant $k$ in Reciprocal Rank Fusion?