Overview
Level 1: Foundations · Module 15: Transformers Under the Hood · 25 min read · Extended Track
Last updated: November 2026
When you send a prompt to a modern frontier model (such as Claude 3.7 Sonnet or Gemini 2.5 Pro), you are interacting with an assistant trained across three distinct evolutionary stages:
- Pretraining (Self-Supervised Causal Language Modeling): Learning grammar, reasoning patterns, and world knowledge by predicting the next token across 15+ trillion tokens.
- Supervised Fine-Tuning (SFT): Teaching the raw base model how to answer questions, format JSON, and act as a conversational assistant.
- Preference Alignment (RLHF, DPO, and KTO): Aligning the model with human preferences for helpfulness, accuracy, brevity, and safety.
In this lesson, you will understand the exact mathematical objectives and data representations behind each stage.
1. The Three-Stage Training Pipeline
flowchart TD
RawData["15T+ Tokens Web, Books, GitHub"] --> Pretrain["1. Pretraining (Causal LM Loss)"]
Pretrain --> BaseLLM["Base Foundation Model (e.g. Llama-3-Base)<br><i>Raw autocomplete engine</i>"]
InstructData["100k Curated Dialogue & Code Pairs"] --> SFT["2. Supervised Fine-Tuning (SFT)"]
BaseLLM --> SFT
SFT --> InstructModel["Instruct Model<br><i>Follows formatting & answers prompts</i>"]
Pairs["Human/AI Preference Pairs (Chosen vs Rejected)"] --> DPO["3. Direct Preference Optimization (DPO / RLHF)"]
InstructModel --> DPO
DPO --> FrontierAssistant["Aligned Frontier Model (Claude, ChatGPT, etc.)"]2. Stage 1: Causal Language Modeling Objective
During pretraining, the objective is to minimize cross-entropy loss over a sequence of $T$ tokens: $$\mathcal{L}{\text{pretrain}}(\theta) = -\sum{i=1}^T \log P(x_i \mid x_1, x_2, \dots, x_{i-1}; \theta)$$
A base model does not know what a "user" or "assistant" is. If you prompt a base model with:
"What is the capital of France?"
it might autocomplete:
"\nWhat is the capital of Germany?\nWhat is the capital of Italy?"
because it was trained purely on web documents and textbook question lists!
3. Stage 2: Supervised Fine-Tuning (SFT) & Chat Templates
To transform the base model into an assistant, researchers curate 50,000 to 500,000 high-quality multi-turn demonstrations.
Special tokens (e.g., <|im_start|>user, <|im_start|>assistant, <|im_end|>) are introduced to separate roles. During SFT, loss is calculated strictly on the assistant tokens, masking out the prompt tokens:
# Simplified demonstration of SFT Loss Masking
import numpy as np
def compute_masked_sft_loss(
logits: np.ndarray, # Shape: (seq_len, vocab_size)
target_ids: np.ndarray, # Shape: (seq_len,)
loss_mask: np.ndarray # Shape: (seq_len,) 1 for assistant tokens, 0 for user prompt
) -> float:
"""
Computes cross-entropy loss only for assistant response tokens.
"""
# Softmax probabilities over vocabulary
exp_logits = np.exp(logits - np.max(logits, axis=-1, keepdims=True))
probs = exp_logits / np.sum(exp_logits, axis=-1, keepdims=True)
# Extract target token probabilities
seq_len = len(target_ids)
target_probs = probs[np.arange(seq_len), target_ids]
log_probs = np.log(np.clip(target_probs, 1e-12, 1.0))
# Apply loss mask
masked_log_probs = log_probs * loss_mask
total_loss = -np.sum(masked_log_probs) / max(1, np.sum(loss_mask))
return float(total_loss)4. Stage 3: RLHF vs Modern Direct Preference Optimization (DPO)
The Traditional RLHF Paradigm (PPO)
- Train a separate Reward Model ($R_\phi$) on human comparison pairs $(x, y_w, y_l)$, where $y_w$ is the preferred "winning" answer and $y_l$ is the "losing" answer.
- Use Proximal Policy Optimization (PPO) to update the LLM policy $\pi_\theta$ to maximize the scalar reward while applying a KL-divergence penalty against the original reference model $\pi_{\text{ref}}$: $$\max_{\theta} \mathbb{E}\left[ R_\phi(x, y) - \beta \mathbb{D}{\text{KL}}(\pi\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)) \right]$$
The Modern Standard: Direct Preference Optimization (DPO)
PPO is notoriously unstable, requiring four large models loaded into GPU VRAM simultaneously (Policy, Reference, Reward Model, Value Critic).
In 2023, Rafailov et al. proved that the reward model can be mathematically extracted directly from the optimal policy. DPO skips the reward model entirely and optimizes the LLM directly on preference pairs with a simple binary cross-entropy loss:
$$\mathcal{L}{\text{DPO}}(\theta; \pi{\text{ref}}) = -\mathbb{E}{(x, y_w, y_l)}\left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$
If the model assigns a higher relative probability to the winning answer $y_w$ compared to the reference model, loss is small. If it favors $y_l$, loss is high.
5. Interview Angle
Staff Engineer Question: "What is the difference between Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) in an enterprise model customization pipeline?"
Principal Answer: SFT teaches the model domain style and schema adherence (e.g., following your specific SQL dialect or customer support tone) by showing it positive examples of ideal responses. DPO is applied after SFT to teach the model subtle boundary discrimination and negative avoidance—showing it pairs of correct vs subtly hallucinated answers so it learns what not to say without reducing conversational fluency.