Overview
Level 3: Advanced Β· Module 19: Voice and Multimodal AI Agents Β· 25 min read Β· Core Track
Last updated: November 2026
The transition from text-based chatbots to conversational voice agents is one of the most exciting shifts in frontend engineering. Traditional conversational voice pipelines were clunky and slow: $$\text{User Audio} \xrightarrow{\text{STT (Whisper)}} \text{Text} \xrightarrow{\text{LLM Generation}} \text{Text} \xrightarrow{\text{TTS (ElevenLabs)}} \text{Audio Stream}$$ This cascading architecture introduced 1,500ms to 3,000ms of latencyβfar too slow for natural human conversation, where natural pauses average between 200ms and 400ms.
Modern Speech-to-Speech Realtime APIs (like OpenAI Realtime and LiveKit Agents) eliminate this transcription bottleneck by processing native audio tokens directly in a unified neural model over low-latency WebRTC and WebSocket data channels.
In this lesson, you will master the transport protocols, Voice Activity Detection (VAD), and user interruption handling required for production voice interfaces.
1. WebRTC vs WebSockets: Choosing the Right Transport
Traditional WebSockets:
Client Audio (PCM16) βββΊ TCP Socket βββΊ Server βββΊ TCP Socket βββΊ Client Audio
- TCP guarantees delivery: re-transmits dropped packets, creating audio jitter / stalling.
- Latency: 500ms - 900ms.
Modern WebRTC Data & Media Channels:
Client Audio βββΊ UDP SRTP Media Stream βββΊ Realtime Model βββΊ UDP SRTP Audio
- UDP drops lost packets instead of stalling, preserving instant conversational cadence.
- Latency: 250ms - 400ms (Human conversation grade!).Protocol Comparison
| Criteria | WebSockets | WebRTC (LiveKit / OpenAI Realtime) |
|---|---|---|
| Transport Layer | TCP (Guaranteed delivery, Head-of-Line blocking) | UDP (Media Streams: RTP / SRTP, peer-to-peer / SFU) |
| Typical Audio Latency | 500ms β 1,000ms | 200ms β 400ms |
| Firewall Traversal | Standard HTTP port 443 | Requires STUN / TURN servers |
| Interruption Handling | Application-level cancel frames | Native RTP silence / truncation signals |
| Best For | Server-to-server pipelines, fallback environments | Direct browser / mobile interactive voice experiences |
2. Audio Agent UX: Voice Activity Detection (VAD) & Barge-In
The biggest UX challenge in voice applications is Barge-In (User Interruption). If the AI agent is in the middle of speaking and the user says "Wait, stop!", continuing to blare synthetic audio while the user is talking creates a disastrous user experience.
The Interruption Lifecycle
- Continuous Audio Capture: The browser's
AudioWorkletrecords raw audio at 24kHz / 16-bit mono. - Server-Side VAD Trigger: When the user begins speaking, the model's Voice Activity Detection detects human speech within 100ms.
- Truncation & Cancellation: The server immediately sends an
item.truncateevent to the client with the exact millisecond offset where the user interrupted. - Immediate Client Mute: The React client immediately silences the active Web Audio playback buffer and clears queued audio frames.
3. Production Code: Realtime Voice Session Controller
Below is an implementation of a WebSocket voice session controller connecting to the OpenAI Realtime API with voice activity detection and user interruption cancellation.
Key Takeaways for Frontend Engineers
- Sub-400ms Latency Requires Speech-to-Speech: Traditional STT $\rightarrow$ LLM $\rightarrow$ TTS waterfalls are too slow for conversational voice. Use native speech-to-speech models over WebRTC or WebSockets.
- Prioritize Barge-In Handling: User satisfaction in voice interfaces depends on natural turn-taking. Immediately truncate agent audio when client-side or server-side VAD detects speech.
- Web Audio Sample Rate Standardization: Ensure your Web Audio context is configured for 24,000 Hz (24kHz) mono 16-bit PCM to match model expectations and avoid expensive resampling.