Design a realtime voice assistant. Walk through the latency budget from the end of user speech to the first audio back.
- 5 min — scope: channel (app vs phone), languages, concurrency, consent
- 10 min — cascaded vs speech-to-speech decision
- 10 min — latency budget per stage and how to cut it
- 10 min — barge-in, tool latency, failure modes
- 5 min — capacity, cost per minute, ownership
Two architectures as of Oct 2026. Cascaded — streaming ASR → LLM → streaming TTS — gives control, swappable parts and a text transcript for guardrails and logs. Speech-to-speech models (OpenAI's Realtime API with gpt-realtime, Gemini Live) give lower latency and better prosody with less control.
The metric is voice-to-voice latency from the end of user speech. Natural conversational gaps are a few hundred milliseconds; target p50 ≤ 800 ms and p95 ≤ 1.2 s. The largest single item is usually endpointing — deciding the user has finished — not the model.
Clinic appointment line. WebRTC from the app (a SIP gateway for phone calls adds 100–200 ms). VAD plus a semantic endpointing model predicts turn end from the words, letting the silence threshold drop from ~700 ms to ~300 ms. Streaming ASR emits partial transcripts so the LLM can start prefill early (prefix-cached system prompt). The LLM streams; TTS starts on the first clause (~8–12 tokens). Barge-in: when the caller speaks, stop TTS within 100 ms and truncate the assistant turn in context to what was actually heard.
- Budget per stage, enforced. Per-stage p95 timers in traces; ASR, LLM and TTS colocated in one zone (each cross-region hop adds 60–150 ms); a small fast model for the first response, with spoken filler ('let me check that') while a tool runs.
- Failure modes: false endpointing cuts callers off (track interruption rate); echo triggers self-barge-in (acoustic echo cancellation); tool calls over 2 s cause dead air; PSTN jitter.
- Consent varies by region: recording rules differ (all-party consent in some US states, GDPR in the EU) — region config, not code.
- Org scope: a voice platform team owns media, endpointing and the latency budget; product owns prompts and flows. Buy telephony and possibly TTS; build-vs-buy ASR depends on the languages required.
Assumptions: cascaded pipeline in one region; 10k peak concurrent calls, 4k average; a user turn every ~6 s; ~3k-token cached prompt + ~200 new tokens per turn, ~80 output tokens; 70B-class FP8 LLM at a low-latency operating point of ~600 output tok/s per H100; streaming ASR ~100 streams per L4; TTS ~50 streams per L4 with ~40% of calls speaking at once; illustrative $2.50/H100-h, $0.80/L4-h.
- Latency (p50): network 120 ms (uplink + downlink + jitter buffer) + endpointing 250 + ASR final 100 + LLM TTFT 250 + first clause 100 + TTS first audio 100 ≈ 920 ms. Semantic endpointing and partial-transcript prefill bring it to ~750 ms.
- LLM: 10k / 6 s ≈ 1,670 turns/s × 80 tokens ≈ 133k output tok/s → ~222 H100 ≈ $405k/month sized for peak.
- ASR + TTS: 10k / 100 = 100 L4 plus 4k / 50 = 80 L4 → 180 L4 ≈ $105k/month.
- Per minute: ≈ $510k/month over 4k × 60 × 730 ≈ 175M call-minutes → ≈ $0.003 per minute of compute, before telephony, which usually costs several times more.
What the answer sounds like at each level
Chains speech-to-text, an LLM and text-to-speech with streaming.
Builds a per-stage budget, adds endpointing, partial-transcript prefill, barge-in and colocation.
Chooses cascaded vs speech-to-speech per flow, sizes GPU pools per stage, prices per call-minute, and sets regional consent policy as config.
Follow-ups the interviewer will push
1Should you move to a speech-to-speech model?
Yes for open-ended conversation where latency and naturalness dominate; keep cascaded for regulated flows that need deterministic guardrails on text. Hybrid: speech-to-speech with a transcript side channel feeding logging and a guardrail model.
2Concurrency goes from 10k to 100k calls.
ASR, TTS and LLM pools scale roughly linearly; per-call KV is small. SIP trunk capacity and regional GPU availability become the limits — reserve both ahead of launch.
3A tool call takes 3 s.
Speak a filler within 500 ms, stream partial results, or prefetch likely tool data when intent is detected from partial ASR.
Red flags interviewers listen for
- Optimises model TTFT while ignoring endpointing.
- No barge-in handling.
- Components spread across regions.
- Measures latency from 'request received' rather than end of user speech.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Latency budget | Stage-by-stage numbers summing to a voice-to-voice target; endpointing identified as the largest item. |
| Conversation mechanics | Barge-in with context truncation, fillers for tools, partial-transcript prefill. |
| Capacity | Separate pool sizing for ASR, LLM and TTS; cost per call-minute. |
| Compliance | Regional recording consent as config. |
Sources
- Robust Speech Recognition via Large-Scale Weak Supervision — Radford et al., 2022paper
- Realtime API guide — OpenAI, 2025docs