Overview
Level 4: Production & Interviews · Module 17: LLMOps, Gateways, and Production Deployment · 20 min read · Extended Track
Last updated: November 2026
In traditional cloud engineering, a 10x traffic spike increases server CPU and network egress bills by a few hundred dollars.
In AI engineering, an uncapped customer or a buggy while-loop in an agent can consume $10,000 in frontier API tokens in a single afternoon. FinOps and Real-Time Token Observability are not optional post-launch features—they are core engineering requirements from Day 1.
In this lesson, you will learn how to:
- Meter Token Usage Across Tenancy Tiers: Attributing input, output, and cached tokens to specific users and organizations.
- Implement Hard and Soft Budget Caps: Throttling or gracefully downgrading model tiers before budgets expire.
- Build Real-Time FinOps Dashboards: Streaming OpenTelemetry metrics to ClickHouse, Langfuse, or Datadog.
1. Token Metering & FinOps Data Pipeline
flowchart LR
App["Next.js / FastAPI App"] --> Gateway["AI Gateway (LiteLLM / Custom)"]
Gateway --> Vendor["LLM Provider (Claude / OpenAI)"]
Gateway --> AsyncLogger["Async Fire-and-Forget Logger"]
AsyncLogger --> ClickHouse["ClickHouse / Postgres Metering Table"]
ClickHouse --> Dashboards["FinOps Dashboard (Grafana / React)"]
ClickHouse --> BudgetWatcher["Budget Enforcer (Redis Token Bucket)"]
BudgetWatcher -.->|"Enforce Quota Exceeded (HTTP 429)"| Gateway2. Multi-Tenant Token Metering Middleware (TypeScript)
Here is a Next.js / Node.js middleware logging granular token economics to a persistent ledger:
import { NextResponse } from "next/server";
import { getModel } from "@/lib/ai-course/models";
interface UsageRecord {
tenantId: string;
userId: string;
modelId: string;
inputTokens: number;
outputTokens: number;
cachedTokens: number;
estimatedCostUSD: number;
timestamp: string;
}
export function calculateInvocationCost(
modelId: string,
inputTokens: number,
outputTokens: number,
cachedTokens: number = 0
): number {
const spec = getModel(modelId);
const regularInput = Math.max(0, inputTokens - cachedTokens);
// Cached tokens typically receive an 80-90% discount
const inputCost = (regularInput / 1_000_000) * spec.inputPerMillion;
const cachedCost = (cachedTokens / 1_000_000) * (spec.inputPerMillion * 0.1);
const outputCost = (outputTokens / 1_000_000) * spec.outputPerMillion;
return inputCost + cachedCost + outputCost;
}
export async function logTokenUsage(record: UsageRecord) {
// Asynchronously flush to time-series store (ClickHouse / TimescaleDB / Datadog)
console.log(`[FinOps Metering] Tenant: ${record.tenantId} | Model: ${record.modelId} | Cost: $${record.estimatedCostUSD.toFixed(5)}`);
// In production, execute batch INSERT via worker queue or HTTP beacon
}3. Rate-Limiting & Budget Tiers in Redis
To protect against runaway agent loops, enforce spend caps using Redis atomic decrements:
import redis
from fastapi import HTTPException, status
r = redis.Redis(host="localhost", port=6379, db=0)
def verify_tenant_budget(tenant_id: str, estimated_cost_usd: float):
"""
Enforces a strict $100/day soft-cap and $250/day hard-cap per tenant.
"""
key = f"spend:daily:{tenant_id}"
current_spend = float(r.get(key) or 0.0)
if current_spend + estimated_cost_usd > 250.0:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="Hard daily spend quota exceeded ($250.00). Contact support to raise quota."
)
if current_spend > 100.0:
# Soft-cap triggered: Transparently downgrade to fast lightweight tier (e.g. Gemini 2.0 Flash)
print(f"[Warning] Tenant {tenant_id} exceeded soft-cap. Downgrading to lightweight model tier.")
# Increment spend atomically
r.incrbyfloat(key, estimated_cost_usd)4. Key Metrics for AI FinOps Dashboards
When presenting cost analytics to engineering leadership and CFOs, monitor these four primary Key Performance Indicators (KPIs):
- Cache Hit Ratio: $\frac{\text{Cached Input Tokens}}{\text{Total Input Tokens}}$. A well-designed prompt caching system should maintain $>75%$ cache hit rates, saving thousands of dollars monthly.
- Cost per Successful Task: Dividing total API expenditure by completed business goals (e.g. "Cost per resolved support ticket" or "Cost per merged pull request").
- P95 Latency vs Cost Correlation: Identifying if routing requests to frontier reasoning models actually yields measurable user satisfaction improvements compared to fast edge models.
- Token Waste Ratio: Unused output tokens generated before client cancellation or abort signals.
5. Interview Angle
Staff Engineer Question: "How would you architect a real-time FinOps alerting system that catches runaway agent loops within 60 seconds without adding latency to customer API requests?"
Principal Answer: We decouple telemetry logging from the critical path using an asynchronous fire-and-forget message broker (e.g., Apache Kafka or AWS Kinesis). Every LLM response chunk updates an in-memory sliding-window counter in Redis (
tenant:{id}:spend_last_60s). A background stream consumer evaluates rate-of-spend anomalies. If a single tenant generates $> $5.00$ within any 60-second window, an automated circuit breaker trips in Redis, pausing that tenant's active agent threads and firing an urgent PagerDuty alert to the on-call engineer.