Overview
Level 1: Foundations · Module 3: Calling LLM APIs from Next.js · 25 min read
Why this matters
LLM endpoints are among the most volatile HTTP services in modern cloud architectures. They suffer from high p99 latency spikes, sudden rate-limiting (429 Too Many Requests), capacity overloads (503 Service Unavailable), and transient socket resets. If your server retries naively without backoff or jitter, it risks creating a "thundering herd" catastrophe that locks your team out of the provider API. Conversely, if your application lacks client-directed cancellation, users navigating away or hitting "Stop" leave expensive GPU tokens running on your corporate bill. Mastering resilient retries and end-to-end AbortController cancellation is essential production hygiene.
Learning objectives
- Implement full-jitter exponential backoff algorithms for transient HTTP failures.
- Differentiate retryable errors (
429,500,503, timeouts) from fatal errors (400,401,422). - Chain and propagate
AbortSignalacross Next.js Route Handlers and upstream LLM providers. - Implement timeout deadlines using
AbortSignal.timeout(). - Design user-facing cancellation states in the React frontend.
Mechanical Architecture: The Resilience Loop
sequenceDiagram
autonumber
actor Client as Browser / User
participant Next as Next.js Route Handler
participant LLM as Provider API (OpenAI/Anthropic)
Client->>Next: POST /api/chat (signal_1)
Next->>LLM: fetch(model, signal_merged)
Note over LLM: Latency Spike / 429 Rate Limit
LLM-->>Next: 429 Too Many Requests (Retry-After: 2s)
Note over Next: Exponential Backoff + Full Jitter Delay
Next->>LLM: Retry 1: fetch(model, signal_merged)
alt User clicks "Stop" or navigates away
Client->>Next: Abort signal_1 triggered
Next->>LLM: Socket cancelled immediately!
Note over LLM: Inference halted, billing stopped
Next-->>Client: 499 Client Closed Connection
else Generation Succeeds
LLM-->>Next: 200 OK (Stream begins)
Next-->>Client: 200 OK (SSE Stream)
endImplementation: Resilient Fetch with Jittered Backoff
Here is a battle-tested utility function for calling LLM APIs with retry logic:
// lib/ai/resilient-fetch.ts
export interface RetryOptions {
maxRetries?: number;
initialDelayMs?: number;
maxDelayMs?: number;
backoffFactor?: number;
signal?: AbortSignal;
}
/**
* Calculates exponential backoff with full jitter:
* delay = random_between(0, min(maxDelay, initialDelay * (factor ^ attempt)))
*/
function calculateJitterDelay(
attempt: number,
initialDelayMs: number,
maxDelayMs: number,
factor: number
): number {
const exponential = initialDelayMs * Math.pow(factor, attempt);
const capped = Math.min(maxDelayMs, exponential);
// Full jitter prevents thundering herd synchronization
return Math.floor(Math.random() * capped);
}
const RETRYABLE_STATUS_CODES = new Set([408, 429, 500, 502, 503, 504]);
export async function fetchWithRetry(
url: string,
init: RequestInit,
options: RetryOptions = {}
): Promise<Response> {
const {
maxRetries = 3,
initialDelayMs = 800,
maxDelayMs = 10000,
backoffFactor = 2,
signal,
} = options;
let attempt = 0;
while (true) {
try {
if (signal?.aborted) {
throw new DOMException("Request aborted by caller", "AbortError");
}
const response = await fetch(url, { ...init, signal });
// If successful, or if client returned a non-retryable error (e.g. 400 Bad Request, 401 Invalid Key)
if (response.ok || !RETRYABLE_STATUS_CODES.has(response.status)) {
return response;
}
// Check for upstream provider 'Retry-After' header (in seconds)
const retryAfterHeader = response.headers.get("retry-after");
let waitMs = 0;
if (retryAfterHeader) {
const seconds = parseInt(retryAfterHeader, 10);
waitMs = !isNaN(seconds) ? seconds * 1000 : calculateJitterDelay(attempt, initialDelayMs, maxDelayMs, backoffFactor);
} else {
waitMs = calculateJitterDelay(attempt, initialDelayMs, maxDelayMs, backoffFactor);
}
attempt++;
if (attempt > maxRetries) {
return response; // Return the failing response for caller handling
}
console.warn(`[fetchWithRetry] Received status ${response.status}. Retrying in ${waitMs}ms (attempt ${attempt}/${maxRetries})...`);
// Sleep with abort listener so cancellation cuts sleep immediately
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener("abort", abortHandler);
resolve();
}, waitMs);
const abortHandler = () => {
clearTimeout(timer);
reject(new DOMException("Aborted during retry backoff", "AbortError"));
};
signal?.addEventListener("abort", abortHandler, { once: true });
});
} catch (err: any) {
if (err.name === "AbortError" || signal?.aborted) {
throw err; // Do not retry intentional cancellations
}
attempt++;
if (attempt > maxRetries) {
throw err;
}
const waitMs = calculateJitterDelay(attempt, initialDelayMs, maxDelayMs, backoffFactor);
await new Promise((r) => setTimeout(r, waitMs));
}
}
}Cancellation with AbortController in Route Handlers
A common bug in Next.js is failing to connect the browser's incoming request abort signal with the outgoing upstream fetch call. If a user hits "Stop", Next.js knows the browser hung up, but without explicit signal chaining, the Node runtime keeps reading from the OpenAI socket:
// app/api/chat/route.ts
import { NextRequest } from "next/server";
import { fetchWithRetry } from "@/lib/ai/resilient-fetch";
export async function POST(req: NextRequest) {
// 1. Create a timeout signal (hard limit of 45s)
const timeoutSignal = AbortSignal.timeout(45_000);
// 2. Compose incoming client abort signal with hard timeout signal
// AbortSignal.any() fires if EITHER client disconnects OR 45s elapses
const combinedSignal = AbortSignal.any([req.signal, timeoutSignal]);
try {
const { messages } = await req.json();
const response = await fetchWithRetry(
"https://api.openai.com/v1/chat/completions",
{
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
},
body: JSON.stringify({
model: "gpt-4o",
messages,
stream: true,
}),
},
{
maxRetries: 2,
signal: combinedSignal,
}
);
return new Response(response.body, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
},
});
} catch (err: any) {
if (err.name === "AbortError") {
if (timeoutSignal.aborted) {
return new Response("AI generation timed out after 45s", { status: 504 });
}
return new Response("Generation cancelled by client", { status: 499 });
}
return new Response("Internal Server Error", { status: 500 });
}
}Client-Side Stop Controller in React
In your React UI, trigger cancellation cleanly:
// components/PromptInput.tsx
"use client";
import { useState, useRef } from "react";
export function PromptInput() {
const [isStreaming, setIsStreaming] = useState(false);
const [output, setOutput] = useState("");
const abortControllerRef = useRef<AbortController | null>(null);
const startStream = async (prompt: string) => {
// 1. Cancel any prior running request
abortControllerRef.current?.abort();
abortControllerRef.current = new AbortController();
setIsStreaming(true);
setOutput("");
try {
const res = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messages: [{ role: "user", content: prompt }] }),
signal: abortControllerRef.current.signal,
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const reader = res.body?.getReader();
const decoder = new TextDecoder();
if (reader) {
while (true) {
const { done, value } = await reader.read();
if (done) break;
setOutput((prev) => prev + decoder.decode(value, { stream: true }));
}
}
} catch (err: any) {
if (err.name === "AbortError") {
console.log("Stream successfully stopped by user.");
} else {
console.error("Stream failed:", err);
}
} finally {
setIsStreaming(false);
}
};
const handleStop = () => {
abortControllerRef.current?.abort();
};
return (
<div>
<div className="flex gap-2">
<button
onClick={() => startStream("Explain React 19 Action hooks")}
disabled={isStreaming}
className="px-4 py-2 bg-blue-600 text-white rounded"
>
Generate
</button>
{isStreaming && (
<button
onClick={handleStop}
className="px-4 py-2 bg-red-600 text-white rounded font-medium"
>
Stop Generation
</button>
)}
</div>
<pre className="mt-4 p-4 bg-gray-100 dark:bg-gray-800 rounded">{output}</pre>
</div>
);
}Common Pitfalls to Avoid
- Retrying on 401 or 400: Never retry authentication errors (
401) or invalid JSON payloads (400). You will burn rate limits and delay client error notifications for no reason. - Exponential backoff without Jitter: If 50 users hit a rate limit simultaneously and your backoff delay is exactly
2^attempt * 1000, all 50 clients will retry at precisely 2000ms, immediately triggering a second 429 burst. - Leaving zombie background streams: Always clean up
AbortControllerinstances when components unmount usinguseEffectcleanup handlers.
Interview Angle: How to Answer in System Design
- Question: "How do you handle rate-limiting and cancellation when proxying AI streams in a serverless Next.js architecture?"
- Target Answer:
"For rate-limiting and transient 5xx errors, I wrap the upstream fetch in an exponential backoff loop with full jitter to avoid the thundering herd problem. I respect the upstream
Retry-Afterheader when provided.For cancellation, I compose the browser's incoming
req.signalwith a defensive serverless deadline usingAbortSignal.any([req.signal, AbortSignal.timeout(45_000)]). When the client disconnects or clicks 'Stop', the abort event cascades directly to the provider's HTTP socket, terminating GPU token generation on the provider side and halting cloud billing."
Quick Check
- Why is 'full jitter' critical when implementing exponential backoff for AI APIs?
- Which modern Web API allows composing multiple
AbortSignalinstances into a single signal? - Should HTTP 401 Unauthorized errors be retried?