Overview
Level 3: Advanced Interactions · Module 19: Voice and Multimodal AI Agents · 20 min read · Extended Track
Last updated: November 2026
Building a conversational voice agent is fundamentally different from building a text chat interface.
In text chat, a 1.5-second delay to the first token feels acceptable. In human speech, any latency over 500 milliseconds feels awkward and disjointed, and latencies over 1,000ms cause users to talk over the bot.
To build a magical, sub-400ms voice experience in the browser, frontend engineers must master:
- The Sub-400ms Latency Budget: Breaking down audio capture, transport, transcription, LLM TTFT, and speech synthesis.
- Client-Side Voice Activity Detection (VAD): Detecting when a user starts and stops talking without sending constant silence streams over the wire.
- Turn-Taking & Barge-In (Interruption Handling): Instantly stopping audio playback and canceling server LLM generation the microsecond the user speaks!
1. The Sub-400ms Speech-to-Speech Latency Budget
flowchart LR
Mic["User Speaks"] --> AudioBuffer["Audio Slice (20-40ms)"]
AudioBuffer --> VAD["Client-Side VAD (Silero / WebRTC)"]
VAD --> Transport["WebSocket / WebRTC (30-50ms)"]
Transport --> ServerSTT["Speech-to-Text / Native Audio Model (100ms)"]
ServerSTT --> LLM["LLM TTFT (Claude 3.7 / Gemini 2.0 / OpenAI Realtime) (120ms)"]
LLM --> TTS["Streaming Audio Chunk (80ms)"]
TTS --> Speaker["Browser Audio Playback (Total: ~370ms)"]2. Client-Side Voice Activity Detection (VAD) with Web Audio API
Running VAD locally in the browser prevents sending gigabytes of dead microphone silence to cloud servers and enables instantaneous detection of user speech onsets:
// Hook: Client-Side Audio Level VAD & Interruption Detection
export class BrowserVoiceDetector {
private audioCtx: AudioContext | null = null;
private analyser: AnalyserNode | null = null;
private isSpeaking = false;
private silenceTimer: NodeJS.Timeout | null = null;
constructor(
private onSpeechStart: () => void,
private onSpeechEnd: () => void,
private silenceThresholdMs = 600,
private volumeThreshold = -45 // dB threshold
) {}
async startListening() {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
this.audioCtx = new AudioContext();
const source = this.audioCtx.createMediaStreamSource(stream);
this.analyser = this.audioCtx.createAnalyser();
this.analyser.fftSize = 512;
source.connect(this.analyser);
this.monitorAudio();
}
private monitorAudio() {
if (!this.analyser) return;
const dataArray = new Uint8Array(this.analyser.frequencyBinCount);
this.analyser.getByteFrequencyData(dataArray);
// Compute average decibels
let sum = 0;
for (let i = 0; i < dataArray.length; i++) {
sum += dataArray[i];
}
const average = sum / dataArray.length;
const dB = 20 * Math.log10(average / 255 || 0.0001);
if (dB > this.volumeThreshold) {
if (!this.isSpeaking) {
this.isSpeaking = true;
// User started speaking! Trigger immediate interruption!
this.onSpeechStart();
}
if (this.silenceTimer) {
clearTimeout(this.silenceTimer);
this.silenceTimer = null;
}
} else if (this.isSpeaking && !this.silenceTimer) {
// User might have finished sentence; wait for silence threshold
this.silenceTimer = setTimeout(() => {
this.isSpeaking = false;
this.onSpeechEnd();
this.silenceTimer = null;
}, this.silenceThresholdMs);
}
requestAnimationFrame(() => this.monitorAudio());
}
}3. Handling Interruptions (Barge-In) Cleanly
What happens when the AI is in the middle of speaking a long sentence and the user says, "Wait, stop!"?
If not handled properly:
- The browser continues playing the queued audio buffer for several seconds.
- The user gets frustrated and speaks again.
- The server gets two overlapping speech turns and hallucinates.
The Three-Step Barge-In Protocol:
- Mute Audio Immediately on Client: The instant VAD detects
onSpeechStart(), pause the HTML5<audio>element or clear the Web Audio buffer queue (audioSource.stop()). - Send Cancellation Signal over WebSocket:
{ "type": "response.cancel" } - Server AbortController: The server immediately terminates token generation at the LLM provider, saving tokens and stopping upstream audio encoding.
4. WebSocket vs WebRTC for Production Voice Agents
| Dimension | WebSocket (TCP) | WebRTC DataChannel (UDP) |
|---|---|---|
| Transport Layer | TCP (Guaranteed delivery, Head-of-line blocking) | UDP (Unordered, zero retransmission delay) |
| Network Jitter Handling | Packets queue up during packet loss, causing sudden audio speedups | Drops late packets smoothly, maintaining sub-300ms real-time audio |
| Implementation Complexity | Simple: Standard ws library and JSON messages | Complex: Requires STUN/TURN servers and ICE candidate negotiation |
| Recommendation | Ideal for prototyping and text-fallback voice apps | Mandatory for production call centers and human-like natural conversation |
5. Interview Angle
Staff Engineer Question: "How do you solve the echo cancellation problem when a user's microphone picks up the AI's own voice playing out of the laptop speakers?"
Principal Answer: We implement Acoustic Echo Cancellation (AEC) at two boundaries:
- Browser Native Constraint: In
getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true } }), modern browsers leverage OS-level hardware DSP to cancel out speaker audio.- Client-Side Cross-Correlation Filter: During AI audio playback, our Web Audio analyzer subtracts the known reference output waveform from the incoming microphone stream. If the incoming frequency signature matches the current playback frame within a high correlation threshold, the VAD ignores it as speaker bleed and avoids triggering false barge-in interruptions.