Overview
Level 2: Build ยท Module 6: Retrieval Augmented Generation (RAG) ยท 22 min read ยท Core Track
Why this matters
When a RAG system answers incorrectly in production, inexperienced developers immediately blame the LLM: "The model hallucinated!" In reality, empirical studies show that over 80% of RAG failures originate in the retrieval and ingestion layers, not in the generative model. The model simply generated a plausible answer based on incomplete chunks, outdated documents, or out-of-domain context that the retrieval engine erroneously served. Developing the diagnostic intuition to isolate failuresโdistinguishing between Retrieval Failures, Generation Failures, and Ingestion Failuresโis what transforms an amateur AI builder into a senior AI engineer.
Learning objectives
- Deconstruct the 5 classic failure modes of RAG pipelines.
- Differentiate between Retrieval Precision/Recall failures and Generative Grounding failures.
- Diagnose and fix the "Lost in the Middle" context window attention trap.
- Implement automated cache invalidation to prevent stale chunk poisoning.
- Build an automated diagnostics checklist for RAG production triage.
The 5 Classic RAG Failure Modes
graph TD
subgraph Failures["The Anatomy of a RAG Breakdown"]
F1["Failure 1: Missing Context<br>The answer does not exist in your corpus,<br>or chunking cut the key sentence in half."]
F2["Failure 2: Top-K Miss / Ranking Failure<br>The relevant chunk exists in your database,<br>but was ranked #14 and omitted from top-3."]
F3["Failure 3: Lost in the Middle<br>The chunk was retrieved and injected into prompt,<br>but the model ignored it due to position bias."]
F4["Failure 4: Stale Chunk Poisoning<br>An updated document was inserted, but old<br>deprecated chunks remained in the vector index."]
F5["Failure 5: Hallucination / Over-Extrapolation<br>The model had the context, but ignored it<br>and leaned on pre-trained parametric bias."]
endDeep Dive 1: "Lost in the Middle" (Context Window Attention Degradation)
Groundbreaking research by Stanford (Liu et al.) proved that LLMs do not attend to context windows uniformly:
- Models have highest recall for information placed at the very beginning (primacy bias) and very end (recency bias) of the context window.
- Information placed in the middle of long contexts experiences significant attention degradationโmodels frequently overlook facts positioned between 30% and 70% of the context length!
Attention Recall %
100% โผโโโโฎ โญโโโ
โ โฐโโฎ โญโโโฏ
50% โผ โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โ "Lost in the Middle"
0% โดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโบ Context Depth
Beginning (Top chunks) Middle (Lower ranks) End (User query)The Engineering Fix: Reranked Sorting Order
Instead of placing chunks in descending order [Rank 1, Rank 2, Rank 3, Rank 4, Rank 5]:
Order chunks to place the strongest evidence at the extreme boundaries:
[Rank 1, Rank 3, Rank 5, Rank 4, Rank 2]!
Deep Dive 2: Stale Chunk Poisoning & Document Lifecycle
In many companies, documentation updates regularly:
- On Monday,
v1_auth.mdstates: "Tokens expire in 24 hours." -> Embedded into pgvector. - On Thursday, an engineer updates docs to
v2_auth.md: "Tokens expire in 15 minutes." -> Embedded into pgvector. - The engineer forgets to delete the Monday chunks.
- When a user queries: "What is the token expiration duration?", the vector database returns both chunks. The LLM has conflicting data and hallucinates an average or picks the wrong version!
The Engineering Fix: Parent-Child Document Invalidation
Always associate chunks with an explicit document_id and run deletions in a single transaction:
// lib/ingest/document-sync.ts
import { prisma } from "@/lib/prisma";
export async function upsertDocumentWithChunks(
documentId: string,
title: string,
chunks: Array<{ content: string; embedding: number[] }>
) {
// Execute in a single atomic transaction
await prisma.$transaction(async (tx) => {
// 1. Delete all existing stale chunks for this document
await tx.$executeRawUnsafe(
`DELETE FROM document_chunks WHERE document_id = $1;`,
documentId
);
// 2. Insert new validated chunks
for (const chunk of chunks) {
await tx.$executeRawUnsafe(
`INSERT INTO document_chunks (document_id, content, embedding)
VALUES ($1, $2, $3::vector);`,
documentId,
chunk.content,
`[${chunk.embedding.join(",")}]`
);
}
});
console.log(`[upsertDocument] Purged stale chunks & inserted ${chunks.length} fresh vectors for ${documentId}`);
}The Diagnostic Triage Matrix: Isolating Root Causes
Use this decision tree whenever an incorrect RAG answer is reported:
| Symptom | Probable Root Cause | Verification Step | Fix |
|---|---|---|---|
| Model says "I don't know", but document exists | Ingestion / Chunking cutoff or Low Top-K | Inspect raw SQL query results for candidate similarity scores. | Increase initial Top-K from 5 to 25; add overlap. |
| Model provides outdated or deprecated info | Stale Chunk Poisoning | Inspect chunks in DB matching document_id. | Implement transactional purge on document update. |
| Model gives generic answer without citations | Prompt bleeding / Weak system instructions | Inspect prompt payload; verify <context> XML boundaries. | Tighten system instructions; enforce temperature = 0.1. |
| Model ignores fact in injected chunk #3 | "Lost in the Middle" attention degradation | Check position of chunk in prompt context. | Reduce number of injected chunks; order by primacy/recency. |
Interview Angle: How to Answer in System Design
- Question: "How do you diagnose and debug poor answer quality in a production RAG application?"
- Target Answer:
"I decouple debugging into two distinct inspection gates: Retrieval Diagnostics and Generation Diagnostics.
First, I inspect the retrieved context in our telemetry logs (e.g. LangSmith or OpenTelemetry spans). If the correct answer is completely missing from the top-K chunks, the bug is in the retrieval layer: we have either poor chunk boundary slicing, missing document ingestion, or need a Cross-Encoder reranker.
Second, if the correct answer IS present in the retrieved chunks, but the model fails to output it, the bug is in the generation layer: either we are suffering from 'Lost in the Middle' attention degradation (solved by pruning context and sorting key chunks to boundaries) or weak prompt fence constraints that allow parametric hallucinations."
Quick Check
- Where do over 80% of production RAG errors originate (retrieval vs generative model)?
- What is the "Lost in the Middle" phenomenon in LLM context windows?
- How do you prevent stale, deprecated chunks from polluting vector search results when documentation is updated?