Overview
Level 1: Foundations · Module 3: Calling LLM APIs from Next.js · 25 min read · Core Track
Why this matters
Traditional REST APIs are largely deterministic: they return in 50ms, or they throw a 404 or 500. LLM APIs, by contrast, are inherently fragile: provider GPU clusters experience sudden concurrency throttling (429 RateLimitError), complex reasoning queries can exceed gateway timeouts (504 Gateway Timeout), and streaming connections can drop halfway through a response due to transient socket resets. If your frontend only handles the "happy path," your users will be greeted with frozen loading spinners, blank screens, or silent data truncation.
Learning objectives
- Catalog the primary failure modes of commercial AI providers (Anthropic, OpenAI, Google).
- Implement robust exponential backoff with full jitter in TypeScript.
- Handle partial stream interruptions without losing already-rendered tokens.
- Design user-centric UI fallback states for rate limit and quota exhaustion errors.
Concept explanation: The Hierarchy of AI Failure Modes
graph TD
A["Frontend AI Request"] --> B{"Failure Point?"}
B -- "Provider Throttled" --> C["429 RateLimit / Overloaded<br>Action: Bounded exponential backoff + jitter"]
B -- "Network / Proxy Timeout" --> D["504 Gateway Timeout<br>Action: Fallback to faster, smaller model tier"]
B -- "Mid-Stream Socket Drop" --> E["ReadableStream Broken<br>Action: Preserve partial tokens + show 'Retry' button"]
B -- "Malformed Model Output" --> F["JSON Parse / Schema Failure<br>Action: Automated self-healing retry turn"]1. Rate Limits (429 Too Many Requests)
Providers enforce two separate rate limits:
- RPM (Requests Per Minute): Absolute count of HTTP calls.
- TPM (Tokens Per Minute): Total sum of input and output tokens consumed across all concurrent requests.
When your cluster hits a TPM spike, the provider returns a
429with aretry-afterheader.
2. Timeouts (504 / ECONNRESET)
Because output token generation takes 20–50ms per token, generating a 2,000-token response takes 40 to 100 seconds. Standard CDN reverse proxies (Cloudflare, Vercel, AWS ALB) often terminate connections at 15–30 seconds if headers or initial bytes are not flushed immediately.
3. Mid-Stream Socket Drops
In streaming architectures, the HTTP response starts with 200 OK and sends chunks over time. If the provider's GPU cluster crashes at token 350, the HTTP status is already sent. The client experiences a broken ReadableStream instead of a clean HTTP error code!
Code walkthrough: Resilient Fetch with Exponential Backoff and Jitter
Never retry immediately in a tight loop—doing so exacerbates provider throttling. Always use Full Jitter:
// lib/ai/resilient-client.ts
export interface RetryOptions {
maxRetries?: number;
initialDelayMs?: number;
maxDelayMs?: number;
}
/**
* Calculates exponential backoff with full jitter to avoid thundering herds.
*/
function getJitterDelay(attempt: number, initialMs: number, maxMs: number): number {
const exponential = Math.min(maxMs, initialMs * Math.pow(2, attempt));
// Randomize between 0 and exponential ceiling (Full Jitter)
return Math.floor(Math.random() * exponential);
}
export async function fetchWithResilientRetry(
url: string,
options: RequestInit,
retryOpts: RetryOptions = {}
): Promise<Response> {
const { maxRetries = 3, initialDelayMs = 500, maxDelayMs = 8000 } = retryOpts;
let attempt = 0;
while (true) {
try {
const response = await fetch(url, options);
// Return immediately on success
if (response.ok) {
return response;
}
// 429 Rate Limit or 503 Provider Overloaded: eligible for backoff
const isRetryable = response.status === 429 || response.status === 503 || response.status === 504;
if (!isRetryable || attempt >= maxRetries) {
return response; // Return error response for caller to handle
}
// Check for provider 'retry-after' header
const retryAfterHeader = response.headers.get("retry-after");
const delayMs = retryAfterHeader
? parseInt(retryAfterHeader, 10) * 1000
: getJitterDelay(attempt, initialDelayMs, maxDelayMs);
attempt++;
console.warn(`[AI Client] HTTP ${response.status}. Retrying attempt ${attempt}/${maxRetries} after ${delayMs}ms...`);
await new Promise((res) => setTimeout(res, delayMs));
} catch (err: any) {
// Network drops / connection errors
if (attempt >= maxRetries) throw err;
attempt++;
const delayMs = getJitterDelay(attempt, initialDelayMs, maxDelayMs);
await new Promise((res) => setTimeout(res, delayMs));
}
}
}Handling Partial Stream Interruptions in React
When consuming a stream in React, wrap the reader loop so that if the stream aborts halfway, the user does not lose what was already read:
// hooks/useResilientStream.ts
import { useState } from "react";
export function useResilientStream() {
const [content, setContent] = useState<string>("");
const [isInterrupted, setIsInterrupted] = useState<boolean>(false);
async function consumeStream(response: Response) {
setIsInterrupted(false);
const reader = response.body?.getReader();
if (!reader) return;
const decoder = new TextDecoder();
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
const textChunk = decoder.decode(value, { stream: true });
setContent((prev) => prev + textChunk);
}
} catch (streamErr) {
console.warn("Stream interrupted prematurely:", streamErr);
// DO NOT clear content! Keep what was generated and show retry badge
setIsInterrupted(true);
} finally {
reader.releaseLock();
}
}
return { content, isInterrupted, consumeStream };
}Common mistakes frontend developers make
- Wiping the text state on stream error: If a 500-word essay drops at word 480, clearing the state forces the user to start over. Preserve the 480 words and show a "Stream paused. Click to continue" button.
- Infinite retry loops on client 400 errors: Retrying a
400 Bad Requestor401 Unauthorizedwill never succeed; only retry429and5xxerrors. - Not providing an Abort button: If an LLM starts streaming a 4,000-token runaway monologue, users must be able to cancel generation immediately to save tokens.
Interview angle: How this appears in interviews
Interviewer: "An LLM stream fails at token 250 out of 500 due to an edge network glitch. How does your frontend architecture handle this?"
Winning Answer:
"I treat streaming failures as partial successes rather than fatal crashes.
- State Preservation: The React state preserves all 250 previously rendered tokens.
- Visual Indicator: The UI replaces the streaming cursor with an inline warning badge ('Stream disconnected').
- Resume Action: We provide a 'Continue generating' action. When clicked, the client sends the original prompt PLUS the 250 previously generated tokens as assistant context, asking the model to resume from where it left off.
- Backpressure & AbortController: The client manages an active AbortController so stale sockets are immediately garbage collected."
Quick check
- What is the difference between RPM and TPM rate limits?
(Answer: RPM limits requests per minute; TPM limits total tokens (input + output) consumed per minute.) - Why is 'Full Jitter' critical when retrying after a 429 rate limit?
(Answer: It randomizes retry intervals so concurrent clients do not collide in a synchronized thundering herd.) - Can an HTTP 500 status code be sent after a stream has already begun streaming tokens?
(Answer: No, HTTP headers and status are committed on the first chunk; failures after that appear as stream reader errors.)
Summary
- AI APIs fail via rate limits (429), gateway timeouts (504), and socket drops mid-stream.
- Use exponential backoff with full jitter; never retry in a tight loop.
- Preserve partial stream tokens when network interruptions occur.
- Always wire an
AbortControllerto give users instant control. - Next up: We will build Retries, Exponential Backoff, and Cancellation with AbortController.