Overview
Level 1: Foundations · Module 14: Python for TypeScript Engineers · 25 min read · Core Track
Last updated: November 2026
In modern full-stack generative AI applications, decoupling the frontend client from the AI inference engine is standard architectural best practice:
- Frontend / Edge Layer (Next.js & React Server Components): Delivers instant optimistic UI transitions, handles user authentication, renders client-side Markdown streaming, and enforces responsive mobile layouts.
- AI Microservice Layer (Python & FastAPI): Executes LangGraph state machine graphs, orchestrates vector database pipelines, evaluates guardrails, and connects to vLLM GPU clusters.
The industry-standard protocol bridging these two distributed tiers in real time is Server-Sent Events (SSE).
In this comprehensive lesson, you will master:
- The Anatomy of the SSE Protocol: Why SSE beats WebSockets for LLM text streaming.
- FastAPI Asynchronous Generators: Constructing high-throughput generators that yield non-blocking SSE frames.
- Overcoming Reverse Proxy Buffering: Configuring NGINX, Cloudflare, and AWS ALBs with
X-Accel-Bufferingand keep-alive heartbeats. - Client-Side Consumption in Next.js: Reading streams via
fetch()andReadableStreamDefaultReaderwith fullAbortControllercancellation support.
1. Protocol Comparison: SSE vs WebSockets
Why do OpenAI, Anthropic, and enterprise AI gateways standardize on Server-Sent Events instead of bidirectional WebSockets for conversational streaming?
| Metric | Server-Sent Events (SSE) | WebSockets |
|---|---|---|
| Transport | Standard HTTP/1.1 or HTTP/2 | Bi-directional TCP Upgrade handshake |
| Directionality | Unidirectional (Server $\longrightarrow$ Client) | Full Duplex (Server $\longleftrightarrow$ Client) |
| Multiplexing | Native HTTP/2 multiplexing over a single connection | Requires custom connection pooling |
| Firewall & Proxy Traversal | 100% standard HTTP (Zero special firewall rules) | Often blocked or intercepted by corporate TLS proxies |
| Reconnection | Built-in native browser reconnect with Last-Event-ID | Requires manual client heartbeat & reconnect logic |
| Use Case Fit | Ideal for token streaming, agent progress, and tool events | Ideal for multi-player real-time gaming or collaborative canvases |
2. The End-to-End Streaming Architecture
sequenceDiagram
autonumber
actor User as Browser (React UI)
participant Next as Next.js Route Handler / Edge Proxy
participant FastAPI as FastAPI Microservice (Python)
participant LLM as Frontier Model (Claude 3.7 / o3-mini)
User->>Next: POST /api/chat { prompt: "Analyze architecture" }
Next->>FastAPI: POST /agent/stream { prompt: "..." }
FastAPI->>LLM: Stream inference chunks
loop Stream Tokens
LLM-->>FastAPI: Async token chunk
FastAPI-->>Next: HTTP 200 chunk: data: {"token": "..."}\n\n
Next-->>User: Forward SSE chunk to ReadableStream
Note over User: React UI appends token without re-render lag!
end
FastAPI-->>Next: data: [DONE]\n\n
Next-->>User: Close Stream3. FastAPI Backend Implementation: StreamingResponse & Async Generator
In FastAPI, streaming requires three essential components:
- An Asynchronous Generator (
async def generate()) yielding string frames. - The standard Starlette
StreamingResponse. - Anti-buffering HTTP headers:
Cache-Control: no-cacheandX-Accel-Buffering: no(preventing NGINX reverse proxies from hoarding chunks in memory).
import asyncio
import json
import time
from typing import AsyncGenerator
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from anthropic import AsyncAnthropic
app = FastAPI(title="AI Streaming Microservice", version="2026.1")
anthropic_client = AsyncAnthropic()
class ChatRequest(BaseModel):
prompt: str
model: str = "claude-3-7-sonnet"
async def llm_token_generator(prompt: str, model: str) -> AsyncGenerator[str, None]:
"""
Asynchronously streams token frames formatted as valid SSE packets:
'data: <json_string>\n\n'
"""
try:
# Initial metadata event
init_frame = {"type": "start", "timestamp": time.time()}
yield f"data: {json.dumps(init_frame)}\n\n"
# Stream from Anthropic Claude
async with anthropic_client.messages.stream(
model=model,
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
) as stream:
async for text_delta in stream.text_stream:
chunk_payload = {
"type": "token",
"text": text_delta,
}
yield f"data: {json.dumps(chunk_payload)}\n\n"
# Micro-yield ensures Starlette flushes buffer immediately to socket
await asyncio.sleep(0)
# Final completion event
yield f"data: {json.dumps({'type': 'done'})}\n\n"
except Exception as exc:
error_payload = {"type": "error", "message": str(exc)}
yield f"data: {json.dumps(error_payload)}\n\n"
@app.post("/api/stream-agent")
async def stream_agent_endpoint(request: ChatRequest):
return StreamingResponse(
llm_token_generator(request.prompt, request.model),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no", # Critical for NGINX/Cloudflare!
"Content-Encoding": "none",
},
)4. Next.js App Router Edge Proxy: Forwarding and Decoding Streams
In your Next.js application, route handlers act as secure edge proxies: they authenticate the user's session cookie, hide your internal Python microservice URL from public exposure, and forward the raw stream directly to the browser.
// app/api/chat/route.ts (Next.js Route Handler)
import { NextRequest, NextResponse } from "next/server";
export const runtime = "nodejs"; // or "edge"
export const dynamic = "force-dynamic";
export async function POST(req: NextRequest) {
const { prompt } = await req.json();
// 1. Dispatch request to internal Python FastAPI container
const fastApiResponse = await fetch("http://fastapi-backend:8000/api/stream-agent", {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Internal-Secret": process.env.INTERNAL_SERVICE_SECRET || "",
},
body: JSON.stringify({ prompt, model: "claude-3-7-sonnet" }),
});
if (!fastApiResponse.ok || !fastApiResponse.body) {
return NextResponse.json(
{ error: "FastAPI upstream streaming service unavailable" },
{ status: 502 }
);
}
// 2. Return the raw ReadableStream directly to client with SSE headers
return new Response(fastApiResponse.body, {
headers: {
"Content-Type": "text/event-stream; charset=utf-8",
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
});
}5. Client-Side React Hook: Handling Reconnects and Cancellations
Standard browser EventSource only supports HTTP GET requests without custom body payloads or authorization headers. In production frontend engineering, we use fetch() with the ReadableStream API and an AbortController:
// hooks/useStreamingChat.ts
import { useState, useRef, useCallback } from "react";
export function useStreamingChat() {
const [messages, setMessages] = useState<string>("");
const [isStreaming, setIsStreaming] = useState(false);
const abortControllerRef = useRef<AbortController | null>(null);
const sendMessage = useCallback(async (prompt: string) => {
setIsStreaming(true);
setMessages("");
// Create fresh AbortController for cancelability
abortControllerRef.current = new AbortController();
try {
const response = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt }),
signal: abortControllerRef.current.signal,
});
if (!response.body) throw new Error("No response stream available");
const reader = response.body.getReader();
const decoder = new TextDecoder("utf-8");
let buffer = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n\n");
// Keep unfinished trailing chunk in buffer
buffer = lines.pop() || "";
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed.startsWith("data: ")) continue;
const rawData = trimmed.slice(6);
if (rawData === "[DONE]") break;
try {
const parsed = JSON.parse(rawData);
if (parsed.type === "token" && parsed.text) {
setMessages((prev) => prev + parsed.text);
}
} catch {
// Partial packet: continue buffering
}
}
}
} catch (err: unknown) {
if ((err as Error)?.name === "AbortError") {
console.log("Streaming cancelled by user.");
} else {
console.error("Streaming error:", err);
}
} finally {
setIsStreaming(false);
abortControllerRef.current = null;
}
}, []);
const cancelStreaming = useCallback(() => {
if (abortControllerRef.current) {
abortControllerRef.current.abort();
}
}, []);
return { messages, isStreaming, sendMessage, cancelStreaming };
}6. Avoiding the "Buffer Trap": NGINX, Cloudflare & Keep-Alive Pings
If your streamed tokens arrive in sudden 1,000-character bursts rather than smooth word-by-word characters, your stream is being buffered by intermediate network proxies.
Essential Production Checklist:
- Disable NGINX Buffering: Ensure your NGINX proxy passes
proxy_buffering off;and respect the upstreamX-Accel-Buffering: noheader. - Cloudflare Buffering: If using Cloudflare proxy (orange cloud), ensure you do not have Polish or Auto Minify enabled on API streaming routes, as they buffer whole responses to calculate compression ratios.
- Keep-Alive Heartbeat Pings: Long agent deliberations (e.g. 15 seconds spent executing Python code or fetching vector DB records) can trigger aggressive load balancer idle connection timeouts (AWS ALB default: 60s). Emit an empty SSE comment frame every 15 seconds:
# SSE Comment Frame (Ignored by client parsers, keeps TCP socket warm) yield ": ping\n\n"
7. Staff Engineer Interview Angle
Staff Engineer Question: "How do you guarantee that a user closing their browser tab or clicking 'Stop Generating' actually stops expensive token generation on the Python backend?"
Principal Answer: We implement end-to-end signal propagation:
- In the browser, clicking 'Stop' triggers
abortController.abort(), which terminates the HTTP request from the client to Next.js.- Next.js detects the client disconnect via
req.signal.on("abort")and aborts its upstream fetch to FastAPI.- In FastAPI, we wrap the generator in
request.is_disconnected()checks. When Starlette detects the broken TCP socket, it throws anasyncio.CancelledErrorinside the coroutine, which triggers thefinallyblock and immediately cancels the Anthropic / OpenAI async stream context, stopping billable token generation instantly.