Overview
Level 4: Production & Interviews · Module 17: LLMOps, Gateways, and Production Deployment · 20 min read · Extended Track
Last updated: November 2026
A common architectural disaster in early AI engineering is attempting to run multi-minute autonomous agent loops directly inside synchronous HTTP requests.
When an agent needs to perform deep multi-page research, crawl 20 websites, and synthesize a 10-page report, standard HTTP connections fail:
- Cloudflare terminates connections after 100 seconds (
Error 524: A Timeout Occurred). - Vercel Serverless functions timeout after 15–60 seconds.
- If the user closes their browser tab, the synchronous connection drops, wasting the expensive tokens generated up to that point.
The solution is an Asynchronous Job Queue Architecture using BullMQ (TypeScript) or Celery / ARQ (Python) backed by Redis.
1. Synchronous vs Asynchronous Job Architecture
sequenceDiagram
actor Client as User Browser
participant API as Next.js API / Edge Route
participant Queue as Redis Job Queue (BullMQ)
participant Worker as Background Worker (Python / Node.js)
participant DB as Postgres (Job State & Events)
Client->>API: POST /api/agent/run { prompt: "Analyze 20 competitors" }
API->>Queue: Enqueue Task (jobId="job_789")
API->>DB: INSERT INTO jobs (id, status) VALUES ('job_789', 'queued')
API-->>Client: HTTP 202 Accepted { jobId: "job_789", status: "queued" }
Note over Client: Client polls or connects via WebSocket / SSE!
Queue->>Worker: Dequeue Task
Worker->>Worker: Execute 5-Minute Agent Loop
Worker->>DB: UPDATE jobs SET status = 'completed', result = {...}
Worker-->>Client: WebSocket Notification: "Job Finished!"2. Implementation with BullMQ and Next.js (TypeScript)
1. Enqueueing Route Handler (/api/jobs/route.ts)
import { Queue } from "bullmq";
import Redis from "ioredis";
import { NextResponse } from "next/server";
const connection = new Redis(process.env.REDIS_URL || "redis://localhost:6379");
const agentQueue = new Queue("ai-agent-tasks", { connection });
export async function POST(req: Request) {
const { taskDescription, tenantId } = await req.json();
// Enqueue job with exponential backoff retries and timeout guardrails
const job = await agentQueue.add(
"deep-research-task",
{ taskDescription, tenantId },
{
attempts: 3,
backoff: { type: "exponential", delay: 2000 },
removeOnComplete: { age: 3600 * 24 }, // retain completed logs for 24h
removeOnFail: { age: 3600 * 48 },
}
);
return NextResponse.json(
{
jobId: job.id,
status: "queued",
checkStatusUrl: `/api/jobs/${job.id}`,
},
{ status: 202 }
);
}2. Background Worker Process (worker.ts)
import { Worker, Job } from "bullmq";
import Redis from "ioredis";
const connection = new Redis(process.env.REDIS_URL || "redis://localhost:6379");
const worker = new Worker(
"ai-agent-tasks",
async (job: Job) => {
console.log(`[Worker] Started processing agent job: ${job.id}`);
// Update intermediate progress observable by frontend UI
await job.updateProgress(25);
// Execute sub-agent research step...
await job.updateProgress(75);
// Execute synthesis step...
return {
status: "success",
reportUrl: `https://storage.deepfrontend.net/reports/${job.id}.pdf`,
totalTokensUsed: 14200,
};
},
{
connection,
concurrency: 5, // Run up to 5 concurrent agent workflows per worker pod
}
);
worker.on("completed", (job) => {
console.log(`[Worker] Job ${job.id} completed successfully!`);
});
worker.on("failed", (job, err) => {
console.error(`[Worker] Job ${job?.id} failed:`, err);
});3. Progressive SSE Streaming from Asynchronous Jobs
How can the frontend display live streaming tokens while a long job runs inside a decoupled background worker?
- Redis Pub/Sub Channel: The worker publishes token chunks to a channel keyed by job ID:
channel:job_789. - Next.js SSE Route: When the user's browser opens
/api/jobs/job_789/stream, Next.js subscribes tochannel:job_789and streams chunks directly over Server-Sent Events! - Resilience: If the user temporarily loses Wi-Fi, the background job never stops. When the client reconnects, it retrieves buffered chunks from Redis and resumes the stream smoothly.
4. Interview Angle
Staff Engineer Question: "How do you guarantee idempotency and prevent duplicate billing when an agent task fails halfway through a 10-step workflow?"
Principal Answer: We implement a two-level idempotency strategy:
- Deduplication Key at Ingestion: When enqueueing, we generate a SHA-256 hash of the
(tenantId, requestParams, clientNonce)and assign it as BullMQ'sjobId. BullMQ automatically rejects duplicate enqueue requests within the deduplication window.- Checkpoint State Journaling: Every completed step in the agent workflow (e.g.
step_1_search_done,step_2_extract_done) is committed to a Postgres state table with an optimistic concurrency version. If a worker pod crashes and the job is retried, the newly spawned worker inspects the journal and resumes from Step 3 without repeating expensive prior LLM calls.