Overview
Level 1: Foundations · Module 15: Transformers Under the Hood · 25 min read · Core Track
Last updated: November 2026
At the heart of every modern frontier model—from open-weights clusters to Claude 3.7 and Gemini 2.5 Pro—lies a single mathematical operation: Multi-Head Scaled Dot-Product Attention.
While academic papers often overwhelm engineers with dense tensor calculus, self-attention can be understood cleanly through linear algebra and dictionary lookups:
- Every token asks a question (Query $Q$)
- Compares it against what other tokens contain (Key $K$)
- Scales and normalizes the affinity via Softmax
- Extracts the weighted contextual payload (Value $V$)
In this lesson, you will build an interactive, vectorized Self-Attention simulator with light math in both TypeScript and Python NumPy.
1. The Core Equation Broken Down
The famous Vaswani et al. equation defines attention across token matrices:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
Matrix Dimensionality:
Tokens: N = 4 (e.g., ["The", "bank", "of", "rivers"])
Embedding Dim: d_model = 8, Key Dim: d_k = 4
Q: (N × d_k) ──┐ Matrix Multiply
K^T: (d_k × N) ──┴──► Score Matrix: (N × N)
Scale by 1 / sqrt(d_k)
Apply Softmax along rows ──► Attention Weights Matrix: (N × N)
V: (N × d_v) ────────────────────────────────► Multiply with Weights ──► Output: (N × d_v)Why Scale by $\frac{1}{\sqrt{d_k}}$?
As the vector dimension $d_k$ grows (e.g. $d_k = 128$ in modern models), the dot product magnitudes scale proportionally. Without dividing by $\sqrt{d_k}$, the dot products become extremely large numbers (e.g. $+80$).
When passed into the $\text{softmax}(z) = \frac{e^{z_i}}{\sum e^{z_j}}$ function, large values push gradients into regions where the derivative is practically zero ($\approx 0$), leading to catastrophic vanishing gradients during training. Scaling by $\sqrt{d_k}$ keeps variance at $1.0$.
2. Multi-Head Attention: Splitting Perspectives
Single-head attention forces the model to average all semantic relationships into a single probability distribution. A word like "apple" would have to attend simultaneously to its fruit identity, company identity, and grammatical role in one pass.
Multi-Head Attention splits the model's hidden representation into $h$ distinct subspace heads (e.g., $h=32$ heads in Llama 3):
- Head 1 learns grammatical dependencies (subject $\rightarrow$ verb).
- Head 2 learns coreference resolution ("it" $\rightarrow$ "the server").
- Head 3 learns semantic category matching ("refund" $\rightarrow$ "transaction").
$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$$
3. Vectorized Scaled Dot-Product Attention Simulator
Below is an interactive implementation of scaled dot-product attention that computes raw scores, applies softmax, and prints the token-to-token attention weight matrix.
Key Takeaways for Frontend Engineers
- Quadratic Complexity ($O(N^2)$): Notice the $N \times N$ weight matrix! If you double your context window from 32,000 to 64,000 tokens, the attention matrix requires $4\times$ the memory and FLOPs.
- Dynamic Contextualization: Static dictionary words like "bank" change their vector representations after layer 1 because they absorb the weighted Value vectors ($V$) of surrounding words like "river" or "deposit".
- KV Caching Preserves $K$ and $V$: During autoregressive generation, past tokens do not need their $K$ and $V$ tensors recalculated. Only the new token's $Q$ vector is multiplied against the cached $K$ matrix.