Overview
Level 1: Foundations · Module 1: How LLMs Work for Frontend Engineers · 25 min read · Core Track
Last updated: November 2026
Frontend engineers building generative applications often treat the underlying Large Language Model as an opaque API endpoint: you send a string of tokens, wait for HTTP streaming chunks, and render the output. However, designing high-throughput UIs, debugging hallucinatory loops, and choosing the right model tier requires understanding the three fundamental pillars of modern neural language models:
- The Self-Attention Mechanism: How transformers weigh the relationships between tokens across long context windows.
- The 3-Stage Training Pipeline: Pretraining (predicting next tokens) → Instruction Tuning (following conversational prompts) → RLHF/DPO (aligning outputs with human values and safety constraints).
- Inference Characteristics: Why models generate text token-by-token (autoregressive decoding) and why first-token latency differs drastically from streaming throughput.
1. The Transformer Self-Attention Mechanism Explained
In traditional recurrent neural networks (RNNs), text was processed sequentially word-by-word, creating an information bottleneck where early context was lost over long paragraphs. The breakthrough of the Transformer architecture (Vaswani et al.) is Self-Attention: the ability to process all tokens in a prompt in parallel, allowing every token to dynamically attend to every other token regardless of distance.
Token Input: "The bank by the river refused to cash the check"
↑ ↑ ↑ ↑
| | | └── "river" clarifies "bank" is geographical? NO!
| | └──────── "cash" & "check" clarify "bank" is financial!
└─ Attention weights dynamically bind "bank" to "cash" & "check"Inside the transformer layer, each input token vector is projected into three distinct representations:
- Query ($Q$): What the current token is looking for (e.g., "I am the word 'it'; which noun do I refer to?").
- Key ($K$): What the token contains or offers as context (e.g., "I am the noun 'PostgreSQL database'").
- Value ($V$): The actual semantic information transferred if a Query matches this Key.
The attention score between two tokens is computed using scaled dot-product attention:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
Interactive Code: Scaled Dot-Product Attention in TypeScript & Python
2. The 3-Stage Training Pipeline
A raw foundation model is not born a helpful assistant. If you ask a raw base model "How do I debounce an input in React?", it will likely reply with "How do I throttle a click in Vue? How do I memoize in Svelte?" because it is simply continuing the pattern of technical questions found in scraped web text.
Modern assistants undergo three rigorous training stages:
flowchart LR
A[Stage 1: Pretraining<br>Predict Next Token on 15T+ Tokens] --> B[Stage 2: Instruction Tuning SFT<br>High-Quality Dialogue Demonstrations]
B --> C[Stage 3: RLHF & DPO<br>Pairwise Human Preference Optimization]
C --> D[Frontier Assistant<br>Claude 3.7 / o3-mini / Gemini 2.0]Stage 1: Unsupervised Pretraining
- Goal: Ingest 10 to 20 trillion tokens of diverse text (Common Crawl, GitHub, Wikipedia, books, academic papers).
- Objective: Self-supervised causal language modeling—given $t_1, t_2, \dots, t_{n-1}$, predict token $t_n$.
- Cost: Millions of GPU-hours ($10M - $100M+ per training run).
- Result: A base model with broad linguistic capabilities and world knowledge, but no concept of instructions or conversational boundaries.
Stage 2: Supervised Fine-Tuning (SFT / Instruction Tuning)
- Goal: Teach the model to adopt the role of a helpful, concise assistant.
- Dataset: 50,000 to 500,000 curated, human-crafted multi-turn conversations featuring instructions, code solutions, summaries, and tool calls.
- Format: Roles are explicitly tagged using special delimiter tokens:
<|im_start|>user Write a TypeScript function to reverse a string.<|im_end|> <|im_start|>assistant export function reverseString(str: string): string { return str.split('').reverse().join(''); }<|im_end|> - Result: An instruction-tuned model that answers questions rather than auto-completing prose.
Stage 3: Alignment (RLHF and DPO)
Even instruction-tuned models can generate dangerous code, exhibit toxic bias, or hallucinate non-existent API methods. The alignment stage guides model behavior toward human preferences:
- RLHF (Reinforcement Learning from Human Feedback):
- Generate multiple candidate completions ($A$ and $B$) for a user prompt.
- Human evaluators rank which answer is more accurate, helpful, and safe.
- Train a separate Reward Model to score responses.
- Optimize the policy model using PPO (Proximal Policy Optimization) against the reward model with a KL-divergence penalty to prevent mode collapse.
- DPO (Direct Preference Optimization):
- Formulated in 2023–2024 as a simpler, more stable alternative to PPO. DPO eliminates the separate reward model entirely and directly optimizes the policy using an implicit mathematical closed-form reward derived from pairwise preferences $(x, y_w, y_l)$ (where $y_w$ is the winning completion and $y_l$ is the losing completion).
Interactive Code: DPO Pairwise Loss Formulation in TypeScript & Python
3. Frontend Implications: TTFT vs Output Streaming Speed
Understanding transformers directly explains production UX bottlenecks:
| Stage | What Happens on GPUs | UX Impact on Web Apps |
|---|---|---|
| Prefill Phase | The entire input context (prompt, retrieved chunks, conversation history) is processed in parallel through matrix multiplications. | Drives Time to First Token (TTFT). TTFT grows with prompt length. |
| Decode Phase | Generates tokens one by one (autoregressive generation). Each new token requires reloading weights through GPU memory bandwidth. | Drives Tokens Per Second (TPS). Streaming throughput is largely constant regardless of prompt length. |
Senior Engineer Rule of Thumb:
When users perceive your app as "slow", the culprit is almost always prefill latency caused by bloated system prompts or unranked RAG chunks. Keeping your input token budget trimmed to under 1,500 tokens ensures sub-250ms TTFT across modern frontier providers.