Production Tracing: Spans, Token Accounting, Latency Breakdown, and Cost Logs
Prerequisites: Route Handlers and Server Actions for AI Calls.
Key Takeaway: You cannot optimize or debug what you cannot measure. When an AI feature fails or feels sluggish in production, structured OpenTelemetry spans and token accounting allow you to dissect every millisecond: isolating whether delay stemmed from database retrieval, provider network queuing, or token generation.
The Black Box Problem of Production AI
In traditional web development, a slow HTTP request is easy to trace using tools like Datadog or Sentry: you inspect the database query span, identify a missing SQL index, and fix it.
In AI-powered applications, a single user click triggers a complex distributed sequence:
graph TD
Client["Client Browser"] --> Handler["Next.js Route Handler"]
Handler --> Step1["Span 1: Query Rewriter (gpt-4o-mini: 180ms)"]
Handler --> Step2["Span 2: Embedding Generation (text-embedding-3-small: 65ms)"]
Handler --> Step3["Span 3: pgvector HNSW Vector Search (12ms)"]
Handler --> Step4["Span 4: Cohere Cross-Encoder Rerank (95ms)"]
Handler --> Step5["Span 5: Provider Time to First Token TTFT (380ms)"]
Handler --> Step6["Span 6: Streaming Generation (2,400ms @ 35 tok/sec)"]If the total response took 3.1 seconds, where did the time and money go?
Without granular tracing spans, engineering teams waste hours arguing whether the vector database is slow, whether the LLM provider is having an outage, or whether prompt tokens are bloated.
Key Production AI Observability Metrics
| Metric | Measurement | Why It Matters |
|---|---|---|
ai.tokens.prompt | Integer count of input tokens | Directly dictates request cost; spikes reveal prompt injection or unpruned history. |
ai.tokens.completion | Integer count of output tokens | Dictates generation duration and output costs. |
ai.cost.usd | Calculated dollar cost per invocation | Enables per-customer unit economic tracking and anomaly alerting. |
ai.latency.ttft | Time to First Token (TTFT) in ms | The primary metric governing perceived responsiveness for end users. |
ai.latency.total | Wall-clock start-to-finish duration | Total session duration. |
ai.finish_reason | stop, length, content_filter, error | Catastrophic spikes in length mean users are getting truncated answers. |
The OpenTelemetry GenAI Semantic Conventions
The OpenTelemetry (OTel) standard defines specific, vendor-neutral semantic attributes for generative AI operations:
gen_ai.system: e.g."openai","anthropic","local_webgpu"gen_ai.request.model: e.g."gpt-4o-2024-08-06","claude-3-5-sonnet-20241022"gen_ai.request.temperature: floatgen_ai.usage.prompt_tokens: intgen_ai.usage.completion_tokens: intgen_ai.response.finish_reasons: string array
By conforming to OTel conventions, you can send traces to any backend (Langfuse, Arize Phoenix, Datadog, Honeycomb) without changing your application code.
Concrete TypeScript Implementation: Next.js AI Tracing Wrapper
Here is an end-to-end, zero-dependency OpenTelemetry-compatible tracing wrapper for Next.js Route Handlers:
// lib/telemetry/ai-tracer.ts
import { v4 as uuidv4 } from "uuid";
export interface AITraceSpan {
traceId: string;
spanId: string;
parentSpanId?: string;
name: string;
model: string;
provider: string;
startTime: number;
endTime?: number;
durationMs?: number;
ttftMs?: number;
promptTokens: number;
completionTokens: number;
costUSD: number;
attributes: Record<string, any>;
finishReason?: string;
error?: string;
}
// Pricing table (USD per 1,000,000 tokens)
const PRICING_PER_MILLION: Record<string, { prompt: number; completion: number }> = {
"gpt-4o": { prompt: 2.50, completion: 10.00 },
"gpt-4o-mini": { prompt: 0.15, completion: 0.60 },
"claude-3-5-sonnet": { prompt: 3.00, completion: 15.00 },
"text-embedding-3-small": { prompt: 0.02, completion: 0.00 },
};
export class AITracer {
public static startTrace(name: string, model: string, provider: string, parentSpanId?: string): AITraceSpan {
return {
traceId: uuidv4(),
spanId: uuidv4(),
parentSpanId,
name,
model,
provider,
startTime: performance.now(),
promptTokens: 0,
completionTokens: 0,
costUSD: 0,
attributes: {},
};
}
public static recordFirstToken(span: AITraceSpan): void {
if (!span.ttftMs) {
span.ttftMs = Math.round(performance.now() - span.startTime);
}
}
public static endTrace(
span: AITraceSpan,
promptTokens: number,
completionTokens: number,
finishReason: string = "stop",
error?: string
): AITraceSpan {
span.endTime = performance.now();
span.durationMs = Math.round(span.endTime - span.startTime);
span.promptTokens = promptTokens;
span.completionTokens = completionTokens;
span.finishReason = finishReason;
span.error = error;
// Calculate dollar cost
const rates = PRICING_PER_MILLION[span.model] || { prompt: 2.5, completion: 10 };
const promptCost = (promptTokens / 1_000_000) * rates.prompt;
const completionCost = (completionTokens / 1_000_000) * rates.completion;
span.costUSD = Math.round((promptCost + completionCost) * 100000) / 100000;
// Dispatch span to central logging sink asynchronously (non-blocking)
this.flushSpan(span);
return span;
}
private static flushSpan(span: AITraceSpan): void {
// In production, dispatch to Langfuse / Datadog / OTel Collector
if (process.env.NODE_ENV === "production") {
// Background HTTP fetch to telemetry endpoint
fetch(process.env.TELEMETRY_INGEST_URL || "https://telemetry.internal/spans", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(span),
}).catch((err) => console.error("[AITracer] Failed to flush span:", err));
} else {
console.log(
`[Trace ${span.name}] Model: ${span.model} | TTFT: ${span.ttftMs}ms | Latency: ${span.durationMs}ms | Tokens: ${span.promptTokens}+${span.completionTokens} | Cost: $${span.costUSD}`
);
}
}
}Route Handler Integration Example
// app/api/chat/route.ts
import { NextRequest } from "next/server";
import { AITracer } from "@/lib/telemetry/ai-tracer";
export async function POST(req: NextRequest) {
const { messages } = await req.json();
const span = AITracer.startTrace("chat-completion", "gpt-4o", "openai");
try {
// Invoke OpenAI streaming
const response = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "gpt-4o",
messages,
stream: true,
}),
});
// Create TransformStream to intercept tokens and record TTFT
const stream = new ReadableStream({
async start(controller) {
const reader = response.body!.getReader();
let promptTokensEst = Math.ceil(JSON.stringify(messages).length / 4);
let completionTokensCount = 0;
while (true) {
const { done, value } = await reader.read();
if (done) break;
AITracer.recordFirstToken(span);
completionTokensCount++; // Approximate token chunk
controller.enqueue(value);
}
AITracer.endTrace(span, promptTokensEst, completionTokensCount, "stop");
controller.close();
},
});
return new Response(stream, {
headers: { "Content-Type": "text/event-stream" },
});
} catch (error: any) {
AITracer.endTrace(span, 0, 0, "error", error.message);
throw error;
}
}Common Pitfalls & Anti-Patterns
- Logging Plain-Text User Secrets in Trace Sinks: Never dump unredacted user inputs into centralized observability tools without PII filtering. An end-user typing their credit card into chat will pollute your internal logs and violate PCI-DSS/GDPR.
- Blocking HTTP Responses on Telemetry Network Calls: Never
await sendTraceToDatadog()before returning the streaming response to the browser. Telemetry ingestion must be asynchronous and fire-and-forget. - Ignoring
finish_reason: length: When models hit max output token caps, they stop mid-sentence. If telemetry does not alert on sudden surges infinish_reason === 'length', users will suffer broken code and half-finished answers.
Senior Engineer Interview Angle
Staff Engineer Question: "Our TTFT degraded from 350ms to 1,400ms over the past week, but OpenAI's status page reports 100% operational health. How do you isolate the root cause?"
Exemplary Answer:
*"We inspect our distributed OpenTelemetry span tree:
- Span Breakdown: We isolate the parent Route Handler span and inspect child spans:
query-rewrite,embedding-generation,pgvector-query, andllm-provider-request.- If the
pgvector-queryspan jumped from 15ms to 950ms, the issue is database index bloat or table lock contention rather than the LLM.- If the database was fast, we inspect
ai.tokens.prompt. A common root cause is prompt bloat: an engineer deployed a prompt update that injected 20 historical chat turns instead of 3, tripling prompt tokens. Because provider TTFT scales with input prefill length, bloated prompts inflate TTFT even during nominal provider health."*
Quick Self-Check
- What is the difference between TTFT and total generation latency?
- What does the OpenTelemetry GenAI semantic convention define?
- Why must telemetry dispatching never block the Next.js streaming response?