Overview
Level 4: Production & Interviews · Module 18: Fine-Tuning and Self-Hosted Models · 20 min read · Extended Track
Last updated: November 2026
When open-weights foundation models are released by Meta, Alibaba, or DeepSeek, they are published in full 16-bit floating-point precision (FP16 or BF16).
Hosting a 70B model in FP16 requires: $$\text{Memory} \approx 70 \times 10^9 \text{ parameters} \times 2 \text{ bytes} \approx 140 \text{ GB of VRAM}$$ This requires multiple high-end enterprise GPUs ($30,000+ hardware cost).
Quantization maps 16-bit floating point numbers to low-bit integer representations (8-bit, 4-bit, or even 2-bit), shrinking memory footprints by up to 75% with negligible degradation in model reasoning and perplexity.
In this lesson, you will master the four primary modern quantization standards and know exactly which format to deploy for CPU, Mac Metal, and NVIDIA production environments.
1. Comparative Matrix: Modern Quantization Formats
| Format | Target Hardware | Engine | Quantization Method | Best Use Case |
|---|---|---|---|---|
| GGUF | CPU, Apple Silicon (Metal), Single GPU | llama.cpp, Ollama | Post-Training (K-quants: Q4_K_M, Q8_0) | Local dev, edge devices, MacBook Pro M3/M4 |
| AWQ (Activation-Aware) | NVIDIA GPUs (Turing, Ampere, Ada, Hopper) | vLLM, TGI, SGLang | Weights protected based on activation magnitude | High-concurrency production serving on NVIDIA |
| GPTQ | NVIDIA / AMD GPUs | AutoGPTQ, vLLM | Second-order Taylor series error compensation | Fast batch inference, broad framework compatibility |
| EXL2 | NVIDIA GPUs | ExLlamaV2 | Mixed-precision variable bitrates per layer | Ultra-fast single-user interactive latency |
2. How AWQ (Activation-Aware Weight Quantization) Works
Traditional naive quantization scales all weights uniformly. However, researchers discovered that in language models, only 1% of weights (salient channels) account for 99% of output accuracy:
flowchart LR
OriginalWeights["FP16 Weights Matrix"] --> MeasureActivations["Run Calibration Dataset (Measure Activation Norms)"]
MeasureActivations --> IdentifySalient["Identify 1% Salient Outlier Weights"]
IdentifySalient --> Scale["Multiply Salient Weights by Optimal Scale Factor s"]
Scale --> Quantize["Round non-salient weights to INT4"]
Quantize --> Dequantize["Fast GEMM Kernel on Tensor Cores"]By multiplying salient weight channels by a scale factor $s > 1$ before round-to-nearest integer quantization, the relative rounding error on critical features is suppressed, preserving reasoning benchmarks within 0.5% of uncompressed FP16 models.
3. Calculating VRAM Requirements for Quantized Deployments
To calculate the exact GPU memory required to host a model with context length $C$:
$$\text{Total VRAM} = \text{Model Weight Memory} + \text{KV-Cache Memory} + \text{CUDA Activation Overhead}$$
1. Model Weights Memory:
$$\text{Weight VRAM (GB)} = \frac{\text{Parameters (Billions)} \times \text{Bits per Weight}}{8} \times 1.15 \text{ (Layer Norm & Embedding overhead)}$$
- 70B Model in FP16 (16-bit): $\approx 140 \text{ GB}$ (Needs 2x A100 80GB)
- 70B Model in AWQ INT4 (4-bit): $\approx 38 \text{ GB}$ (Fits comfortably on a single 48GB A6000 or 2x 24GB RTX 4090 GPUs!)
- 8B Model in GGUF Q4_K_M: $\approx 4.8 \text{ GB}$ (Runs effortlessly on a base 8GB Apple M2/M3 MacBook Air!)
4. Quantization Recipe Guide for Engineers
- For Team Local Workstations: Use GGUF with
Q4_K_M(Medium 4-bit) orQ5_K_M. This offers the optimal Pareto curve between generation speed and response quality. - For Cloud Production APIs (vLLM): Always standardize on AWQ (e.g.
casperhansen/llama-3.1-70b-instruct-awq). AWQ leverages native INT4 Tensor Core matrix kernels on Ampere/Hopper architectures, yielding higher throughput than GPTQ under heavy concurrent loads. - Avoid Sub-3-Bit Quantization: Quantizing below 3.5 bits per parameter triggers steep degradation in multi-turn reasoning and tool-calling JSON schema compliance.
5. Interview Angle
Staff Engineer Question: "Why does 4-bit weight quantization often increase inference tokens-per-second on memory-bandwidth-bound GPUs, even though dequantizing INT4 back to FP16 adds extra mathematical operations?"
Principal Answer: During the autoregressive token generation phase (decoding), inference is strictly memory bandwidth bound, not compute bound. The GPU spends $>90%$ of its cycle time simply fetching model weights from High-Bandwidth Memory (HBM) into registers. Because 4-bit weights are 4x smaller than 16-bit floats, the GPU transfers 75% fewer bytes over the memory bus per generated token. Modern Tensor Cores execute on-the-fly INT4 $\rightarrow$ FP16 dequantization in fractional picoseconds, allowing the memory bus speedup to far outweigh the trivial arithmetic cost.