Overview
Level 4: Production & Interviews · Module 17: LLMOps, Gateways, and Production Deployment · 20 min read · Extended Track
Last updated: November 2026
While consumer AI products often use direct vendor API keys (e.g. api.anthropic.com or api.openai.com), enterprise organizations in finance, healthcare, and defence cannot allow proprietary customer data to traverse the public internet or rely on shared API tokens stored in environment variables.
Enterprises host foundation models inside managed hyperscaler platforms:
- AWS Bedrock: Accessing Claude 3.7 Sonnet, Llama 3, and Amazon Titan within existing AWS Virtual Private Clouds (VPC) authenticated via IAM Roles (zero static API keys).
- Azure AI Foundry (formerly Azure OpenAI Service): Hosting dedicated o3-mini and enterprise Claude/GPT deployments within private Azure VNets with HIPAA/SOC2 Type II compliance and zero data retention agreements.
1. Public Vendor API vs Enterprise VPC Cloud AI
graph TD
subgraph Insecure["Public SaaS Approach (Risky)"]
App1["Next.js App"] -->|"Public Internet (Static API Key)"| PublicVendor["Vendor Public Endpoint (api.openai.com)"]
end
subgraph Enterprise["Enterprise Cloud VPC (Zero-Trust)"]
App2["Next.js Container in Private Subnet"] -->|"Private Link (No Internet Gateway)"| VPC["AWS VPC Endpoint"]
VPC -->|"IAM Role Assumption (STS Temporary Credentials)"| Bedrock["AWS Bedrock Service (Private Claude Instance)"]
Bedrock --> KMS["Customer-Managed KMS Encryption Key"]
end2. Calling AWS Bedrock with Converse API and IAM Authentication (TypeScript & Python)
Modern Bedrock integrations utilize the standardized Converse API (or ConverseStream), providing a unified interface across Claude, Mistral, Llama, and Command models:
// Next.js Route Handler calling AWS Bedrock via ConverseStream API
import { BedrockRuntimeClient, ConverseStreamCommand } from "@aws-sdk/client-bedrock-runtime";
// In AWS ECS/EKS, credentials are automatically resolved from IAM Task Role (no static keys!)
const bedrockClient = new BedrockRuntimeClient({
region: process.env.AWS_REGION || "us-east-1",
});
export async function POST(req: Request) {
const { prompt } = await req.json();
const command = new ConverseStreamCommand({
// Bedrock Model ID for Claude 3.7 Sonnet
modelId: "anthropic.claude-3-7-sonnet-20250219-v1:0",
messages: [
{
role: "user",
content: [{ text: prompt }],
},
],
inferenceConfig: {
maxTokens: 1024,
temperature: 0.2,
},
});
const response = await bedrockClient.send(command);
// Stream chunks back to client as Server-Sent Events
const stream = new ReadableStream({
async start(controller) {
if (!response.stream) {
controller.close();
return;
}
for await (const chunk of response.stream) {
if (chunk.contentBlockDelta?.delta?.text) {
const text = chunk.contentBlockDelta.delta.text;
controller.enqueue(new TextEncoder().encode(`data: ${JSON.stringify({ text })}\n\n`));
}
}
controller.close();
},
});
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
},
});
}And in Python:
import boto3
# Authenticates via local AWS profile or AWS STS Assumed Role
bedrock = boto3.client(service_name="bedrock-runtime", region_name="us-east-1")
def invoke_enterprise_bedrock(prompt: str) -> str:
response = bedrock.converse(
modelId="anthropic.claude-3-7-sonnet-20250219-v1:0",
messages=[{"role": "user", "content": [{"text": prompt}]}],
inferenceConfig={"maxTokens": 1000, "temperature": 0.1}
)
return response["output"]["message"]["content"][0]["text"]3. Azure AI Foundry: Private Endpoints & Entra ID
For organizations standardized on Microsoft Azure:
- Azure Private Link: Private IP routing ensures all prompts remain within the enterprise Virtual Network (VNet) without transiting public DNS.
- Microsoft Entra ID (Azure AD): Next.js or Python pods authenticate using Managed Identities (
DefaultAzureCredential), completely eliminating staticAZURE_OPENAI_API_KEYsecrets from git repositories. - Provisioned Throughput Units (PTU): Enterprises reserve dedicated GPU compute capacity guaranteeing predictable latencies (e.g. 50ms TTFT) without noisy-neighbor rate-limit throttling during peak traffic.
4. Interview Angle
Staff Engineer Question: "How do you design a high-availability multi-region failover strategy for AWS Bedrock without duplicating static credentials or risking compliance violations?"
Principal Answer: We implement Cross-Region Inference Profiles (e.g.
us.anthropic.claude-3-7-sonnet-20250219-v1:0). AWS Bedrock automatically routes requests across multiple US regions (us-east-1,us-west-2,us-east-2) based on real-time GPU cluster health and quota availability. All traffic is bound to Customer-Managed KMS keys (CMK) configured across all target regions, maintaining end-to-end data boundary compliance without requiring custom client-side retry routing.