Overview
Level 2: Build · Module 6: Retrieval Augmented Generation (RAG) · 25 min read
Why this matters
In naive RAG implementations, developers fetch the top-5 nearest vector chunks (LIMIT 5) and immediately dump them into the prompt. This frequently fails: vector similarity only measures high-level topical proximity, not whether a chunk actually answers the user's specific question. A chunk might score a high 0.82 cosine similarity because it talks about "React 19 Server Actions", but may describe form validation rather than security tokens. Cross-Encoder Rerankers (such as Cohere Rerank or BGE-Reranker) revolutionize retrieval quality by performing deep token-to-token cross-attention between the query and candidate passages, reordering candidates so the absolute most informative chunk lands at Rank 1.
Learning objectives
- Understand the difference between Bi-Encoders (fast vector search) and Cross-Encoders (deep reranking).
- Tune Top-K retrieval windows (retrieve top 20 candidates -> rerank down to top 3).
- Apply hard metadata filters (user role, department, recency) in PostgreSQL.
- Integrate the Cohere Rerank API into a Next.js server pipeline.
- Measure retrieval metrics: Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG).
Architectural Anatomy: Bi-Encoder vs Cross-Encoder
graph TD
subgraph Stage1["Stage 1: Fast Candidate Retrieval (Bi-Encoder / ANN)"]
Query1["Query: 'How to invalidate Next.js tags'"] --> Embed["Vector Search (pgvector)"]
Index["1,000,000 Document Chunks"] --> Embed
Embed -->|"Top 25 Candidates (Fast ~8ms)"| Candidates["25 Raw Candidate Chunks"]
end
subgraph Stage2["Stage 2: Deep Precision Reranking (Cross-Encoder)"]
Candidates --> Reranker["Cross-Encoder Reranker<br>(Joint Attention: [Query + Chunk])"]
Query1 -.-> Reranker
Reranker -->|"Compute Relevancy Scores"| Top3["Top 3 Most Informative Chunks"]
end
Top3 --> Generation["Context Injection into LLM Prompt"]| Trait | Bi-Encoder (Dense Vector Search) | Cross-Encoder (Reranker) |
|---|---|---|
| How it works | Query and documents are embedded separately into vectors | Query and document are concatenated and fed jointly through transformer layers |
| Attention Mechanism | Independent vector dot product (shallow) | Full token-to-token cross-attention (deep) |
| Speed | Sub-5ms over 1,000,000 vectors | ~20ms–50ms over 20–50 candidate pairs |
| Scalability | Can search massive datasets via HNSW indices | Cannot scale to entire database; used only on top-K candidates |
| Ranking Accuracy | Good (captures topical similarity) | Exceptional (captures logical alignment) |
The Two-Stage Retrieval Pattern in Code
The industry-standard RAG pipeline uses Stage 1 to cast a wide net (retrieve top-20 chunks) and Stage 2 to distill the top-3 best chunks via a reranker:
// lib/ai/two-stage-retriever.ts
import { prisma } from "@/lib/prisma";
import { CohereClient } from "cohere-ai";
const cohere = new CohereClient({
token: process.env.COHERE_API_KEY,
});
export interface DocumentCandidate {
id: string;
title: string;
content: string;
initialScore: number;
}
export interface RerankedChunk {
id: string;
title: string;
content: string;
relevanceScore: number;
}
/**
* Two-Stage Retrieval:
* 1. Fetch top-25 chunks via pgvector HNSW
* 2. Rerank down to top-3 using Cohere cross-encoder
*/
export async function retrieveAndRerank(
query: string,
queryVector: number[],
topK: number = 3
): Promise<RerankedChunk[]> {
const vectorString = `[${queryVector.join(",")}]`;
// Stage 1: Fast broad retrieval (Fetch 20-30 chunks)
const candidates: DocumentCandidate[] = await prisma.$queryRawUnsafe(`
SELECT
d.id,
d.title,
c.content,
1 - (c.embedding <=> $1::vector) AS "initialScore"
FROM document_chunks c
JOIN documents d ON d.id = c.document_id
ORDER BY c.embedding <=> $1::vector ASC
LIMIT 25;
`, vectorString);
if (candidates.length === 0) return [];
// Stage 2: Deep Cross-Encoder Reranking
const rerankResponse = await cohere.rerank({
model: "rerank-v3.5",
query,
documents: candidates.map((c) => ({
text: `${c.title}\n${c.content}`,
})),
topN: topK,
});
// Map reranked indices back to full candidate objects
return rerankResponse.results.map((result) => {
const original = candidates[result.index];
return {
id: original.id,
title: original.title,
content: original.content,
relevanceScore: result.relevanceScore,
};
});
}Metadata Filtering Before Reranking
Always apply hard business filters (such as tenant organization, role permissions, or publication dates) before or during vector search, rather than filtering post-reranking:
-- Hard filtering: Restrict vector search space by department and active status
SELECT c.id, c.content, 1 - (c.embedding <=> $1) AS similarity
FROM document_chunks c
JOIN documents d ON d.id = c.document_id
WHERE d.department = 'ENGINEERING' -- Metadata Filter
AND d.created_at >= NOW() - INTERVAL '90 days' -- Recency Filter
ORDER BY c.embedding <=> $1 ASC
LIMIT 25;Retrieval Evaluation Metrics: MRR and NDCG
In production AI systems, retrieval quality is measured quantitatively:
- Mean Reciprocal Rank (MRR): Evaluates where the first truly relevant document appears in the ranked list.
- If the ground-truth document is at Rank 1: score = $\frac{1}{1} = 1.0$
- If at Rank 3: score = $\frac{1}{3} = 0.333$
- Target: MRR > 0.85.
- Normalized Discounted Cumulative Gain (NDCG@K): Evaluates whether multiple relevant documents appear near the top with logarithmic penalties for lower ranks. Target: NDCG@5 > 0.80.
Interview Angle: How to Answer in System Design
- Question: "How would you improve RAG retrieval accuracy when users report that relevant documents are frequently missed?"
- Target Answer:
"I implement a Two-Stage Retrieval Architecture with a Cross-Encoder Reranker.
In Stage 1, we widen the initial candidate retrieval pool from top-5 to top-25 using hybrid search (combining pgvector HNSW with BM25 full-text search).
In Stage 2, we pass the 25 candidate passages through a cross-encoder model (such as Cohere Rerank or BGE-Reranker) alongside the original user query. While Bi-Encoders compute isolated vector dot products, Cross-Encoders evaluate full token-to-token cross-attention between the query and text, detecting nuanced logical alignment, negative qualifications, and exact technical conditions.
We retain the top-3 to top-5 reranked passages for context injection, which routinely increases retrieval precision and MRR by 20% to 35% with only ~30ms of added server latency."
Quick Check
- Why can't a Cross-Encoder be used directly over a 1,000,000 document database?
- What is the difference between a Bi-Encoder and a Cross-Encoder in terms of attention mechanism?
- What is the typical candidate ratio for two-stage retrieval (initial fetch vs final reranked count)?