Overview
Level 1: Foundations · Module 15: Transformers Under the Hood · 20 min read · Extended Track
Last updated: November 2026
A common misconception among developers is that self-attention stores all the knowledge in a Large Language Model.
In reality, self-attention is a routing mechanism: it moves information between tokens in a sequence. The actual factual memory—the encyclopedic knowledge of history, programming syntax, geography, and codebases—is stored in the Feed-Forward Network (FFN) blocks that make up roughly two-thirds of the model's total parameter count!
In this lesson, you will inspect:
- The Architecture of a Transformer Block: How attention and FFN layers interleave.
- Residual Connections (Skip Connections): Why deep 80-layer networks don't suffer from vanishing gradients.
- Modern Activation Functions: The evolution from ReLU $\rightarrow$ GeLU $\rightarrow$ SwiGLU.
- Mixture-of-Experts (MoE): How sparse FFNs (like DeepSeek V3 and Mixtral) activate only a fraction of parameters per token.
1. Anatomy of a Modern Transformer Block
Each decoder layer in a modern transformer (e.g., Llama 3 or Qwen 2.5) consists of two main sub-layers wrapped in pre-layer normalization (RMSNorm) and residual connections:
flowchart TD
Input["Input Tensor x_(l-1)"] --> Norm1["RMSNorm"]
Norm1 --> Attention["Grouped-Query Multi-Head Attention (GQA)"]
Attention --> Add1["Residual Add (+)"]
Input --> Add1
Add1 --> Intermediate["Tensor x_attn"]
Intermediate --> Norm2["RMSNorm"]
Norm2 --> SwiGLU["Feed-Forward Network (SwiGLU)"]
SwiGLU --> Add2["Residual Add (+)"]
Intermediate --> Add2
Add2 --> Output["Output Tensor x_l"]2. Residual Connections: The Highway of Gradient Flow
Why do we add the input back to the sub-layer output ($x_{l} = x_{l-1} + \text{SubLayer}(\text{RMSNorm}(x_{l-1}))$)?
Without residual connections, backpropagating gradients through 32 to 128 layers of matrix multiplications causes the gradient to decay exponentially toward zero (the vanishing gradient problem).
With residual connections: $$\frac{\partial \mathcal{L}}{\partial x_{l-1}} = \frac{\partial \mathcal{L}}{\partial x_l} \cdot \left( I + \frac{\partial \text{SubLayer}}{\partial x_{l-1}} \right)$$
The identity matrix $I$ ensures that gradients can flow backwards directly from the loss function to the earliest embedding layer without obstruction!
3. SwiGLU: The Modern Activation Standard
Early neural networks used simple ReLU activations ($\max(0, x)$). Modern frontier transformers universally use SwiGLU (Swish-Gated Linear Unit), introduced by Noam Shazeer.
A SwiGLU block uses three weight matrices ($W_{\text{gate}}$, $W_{\text{up}}$, and $W_{\text{down}}$): $$\text{SwiGLU}(x) = \left( \text{Swish}(x W_{\text{gate}}) \odot (x W_{\text{up}}) \right) W_{\text{down}}$$ where $\text{Swish}(z) = z \cdot \sigma(\beta z)$ (also known as SiLU).
import numpy as np
def sigmoid(x: np.ndarray) -> np.ndarray:
return 1.0 / (1.0 + np.exp(-np.clip(x, -500, 500)))
def swish(x: np.ndarray) -> np.ndarray:
return x * sigmoid(x)
def swiglu_ffn(
x: np.ndarray,
w_gate: np.ndarray,
w_up: np.ndarray,
w_down: np.ndarray
) -> np.ndarray:
"""
Simulates a vectorized SwiGLU Feed-Forward Network layer.
x shape: (batch_size, seq_len, d_model)
"""
# 1. Gate branch: Projected into intermediate dimension and activated with Swish
gate = swish(np.matmul(x, w_gate))
# 2. Up-projection branch: Linear projection without activation
up = np.matmul(x, w_up)
# 3. Element-wise multiplicative gating
gated_features = gate * up
# 4. Down-projection back to d_model
output = np.matmul(gated_features, w_down)
return output
# Dimensionality: d_model = 64, d_ffn = 172
d_model, d_ffn = 64, 172
np.random.seed(42)
w_gate = np.random.randn(d_model, d_ffn) * 0.02
w_up = np.random.randn(d_model, d_ffn) * 0.02
w_down = np.random.randn(d_ffn, d_model) * 0.02
x_input = np.random.randn(2, 4, d_model) # 2 batches, 4 tokens
ffn_out = swiglu_ffn(x_input, w_gate, w_up, w_down)
print("Input shape:", x_input.shape)
print("Output shape:", ffn_out.shape)The element-wise multiplication ($\odot$) acts as a soft information filter: the gate branch decides which features learned in the up-projection are allowed to pass through into the token state.
4. Mixture-of-Experts (MoE): Sparse Computation
In dense models (like Llama 3 8B), every token triggers 100% of the FFN weights.
In Mixture-of-Experts (MoE) models (such as Mixtral 8x7B or DeepSeek V3):
- The single FFN is replaced by $E$ distinct expert FFN sub-networks (e.g., 64 or 256 experts).
- A lightweight Router computes a softmax score for each token over all experts.
- Only the Top-$k$ experts (e.g., Top-2 or Top-8) are activated for that token.
$$\text{MoE}(x) = \sum_{i \in \text{Top-k}} g_i(x) \cdot \text{FFN}_i(x)$$
This allows a model to possess 671 billion total parameters on disk, but execute with the latency and computational cost of only 37 billion active parameters per token!
5. Interview Angle
Staff Engineer Question: "Why do Mixture-of-Experts models experience high throughput during inference but require disproportionately massive GPU VRAM during deployment?"
Principal Answer: During inference, each token only executes the matrix multiplications of the Top-$k$ active experts, keeping Floating Point Operations (FLOPs) and compute latency low. However, all experts must remain resident in GPU High-Bandwidth Memory (HBM) simultaneously because adjacent tokens in a sequence may route to completely different experts. An MoE model with 671B parameters still demands ~1.3 Terabytes of VRAM to host its FP16/FP8 weights, even though each token only uses ~37B parameters of compute.