Overview
Level 4: Production & Interviews · Module 18: Fine-Tuning and Self-Hosted Models · 20 min read · Extended Track
Last updated: November 2026
While closed-source APIs (Claude, OpenAI) are convenient, many organizations deploy self-hosted open-weights models (Llama 3.3, Qwen 2.5, DeepSeek R1) for data residency, zero-cost development, or private VPC architectures.
In the 2026 engineering stack, two tools dominate self-hosted inference:
- Ollama: Designed for local workstation development. Runs on macOS Metal, Windows, and Linux; packages quantized GGUF weights into single Docker-like commands; exposes an OpenAI-compatible REST API at
http://localhost:11434. - vLLM: The industry gold standard for high-throughput production serving. Invented PagedAttention, reducing KV-cache memory waste from 80% to under 4%, and delivers 10x-24x higher concurrent request throughput than naive HuggingFace pipelines.
1. Developer Workstation (Ollama) vs Production Cluster (vLLM)
graph TD
subgraph DevMachine["Local Developer Workstation (macOS / Linux)"]
DevNext["Next.js App (Localhost:3000)"] -->|"POST http://localhost:11434/v1"| Ollama["Ollama Daemon (Metal / GGUF)<br><i>Zero GPU cloud bills</i>"]
Ollama --> LocalWeights["Qwen 2.5 / Llama 3 8B"]
end
subgraph ProdCloud["Production Kubernetes Cluster (NVIDIA H100 / A100)"]
ProdNext["Next.js Edge Cluster"] --> Gateway["LiteLLM Load Balancer"]
Gateway --> vLLM1["vLLM Pod 1 (PagedAttention + Continuous Batching)"]
Gateway --> vLLM2["vLLM Pod 2 (Tensor Parallelism TP=4)"]
vLLM1 & vLLM2 --> SharedWeights["High-Speed NVMe Network Storage"]
end2. Local Development with Ollama in Next.js
Running an open-weights model locally requires a single terminal command:
# Pull and start serving Qwen 2.5 7B with 4-bit quantization
ollama run qwen2.5:7bBecause Ollama provides an OpenAI-compatible endpoint, your existing Next.js AI code requires zero refactoring—only an environment variable change:
// app/api/chat/route.ts
import { OpenAI } from "openai";
// Points to local Ollama instance during development!
const client = new OpenAI({
baseURL: process.env.LOCAL_AI_URL || "http://localhost:11434/v1",
apiKey: "ollama", // dummy key required by SDK
});
export async function POST(req: Request) {
const { messages } = await req.json();
const response = await client.chat.completions.create({
model: "qwen2.5:7b",
messages,
stream: true,
});
// Convert response to Web ReadableStream...
}3. Production Serving with vLLM: PagedAttention Mechanics
In naive transformer serving, GPU memory must be allocated contiguously for each request up to max_model_len (e.g. 8,192 tokens). If a user only generates 200 tokens, 95% of that allocated VRAM is wasted due to internal fragmentation.
PagedAttention borrows the classic virtual memory paging concept from operating systems:
- Key-Value attention tensors are divided into fixed-size blocks (e.g. 16 tokens per block).
- Physical blocks are allocated non-contiguously in GPU HBM memory.
- A virtual block table maps logical token positions to physical blocks.
- Multiple requests (such as parallel branches or beam search) share physical blocks via copy-on-write!
# Production launch command for vLLM with OpenAI API compatibility
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--port 80004. Continuous Batching: Squeezing Maximum GPU Utilization
In traditional deep learning static batching, an inference batch can only finish when the slowest request finishes (if Request A needs 10 tokens and Request B needs 1,000 tokens, the GPU remains 50% idle for 990 tokens).
vLLM implements Continuous Batching (Iteration-Level Scheduling):
- As soon as a request completes generating its
<|eot_id|>token, it is immediately evicted from the batch. - A newly arrived request from the queue enters the batch on the very next token iteration!
- GPU Tensor Cores remain at $>90%$ saturation continuously.
5. Interview Angle
Staff Engineer Question: "How does Tensor Parallelism differ from Pipeline Parallelism when serving a 70B parameter model across multiple GPUs?"
Principal Answer: Tensor Parallelism (TP) splits individual weight matrices (such as attention projections and FFN layers) across multiple GPUs within the same physical node, executing parts of each matrix multiplication concurrently via NVLink high-speed interconnects. It delivers ultra-low latency for single requests. Pipeline Parallelism (PP) partitions entire sequential layers across different nodes (e.g. Layers 1-20 on Node 1, Layers 21-40 on Node 2), streaming activation tensors across network boundaries. PP is used for models that exceed the capacity of a single physical server chassis.