AI Simultaneous Interpretation: The 2026 Playbook For Video Conference Translation

Why Fora Soft wrote this playbook

We’ve been building real-time video products since 2005. Most of what we ship touches live communication somewhere — video conferencing, broadcast platforms, tele-education, telemedicine, hybrid events. Over the past three years we’ve added live translation to a growing share of those projects. Captions in 30 languages for a global webinar series. Voice dubbing for a HIPAA-compliant cross-border telehealth app. Real-time interpretation in a corporate town hall that used to hire six interpreters and now hires none.

This playbook is the working document we use internally to scope those projects. It covers what “AI simultaneous interpretation” actually means in 2026, what the pipeline looks like, which vendors and models are viable, where latency and cost go, and when not to use AI at all. If you’re evaluating Zoom AI Companion against a custom build, or choosing between Wordly, KUDO, Interprefy, or a DIY cascade, or just trying to understand why your human interpreters cost $13 k/day — this is for you.

Related reading from our team: the AI voice recognition playbook for intercom systems (which shares half the voice stack), voice in mobile apps (for the client side), AI streaming platforms (for the delivery side), and AI translation companies (for the vendor landscape).

Agent Engineering and modern dev tooling have compressed our timelines roughly 40% over the past 18 months. What used to be a 16-week integration is now 10–12. We still do the hard work — the pipeline tuning, the glossary work, the compliance — but we ship faster.

What “AI simultaneous interpretation” actually means in 2026

The term gets used loosely. It’s worth being precise, because the buying decisions are different.

Translated captions (or “live captions”): Speaker audio → transcript in source language → translated text overlay. Latency <1 second is achievable. Zoom, Teams, Google Meet, and Webex all ship this. No speaker voice replacement. Accessible, cheap, dominant in enterprise.

Simultaneous interpretation (voice-to-voice): Speaker audio → translated audio output in another language, delivered in real time. This is what human interpreters do. AI pipelines now handle it 2–4 seconds behind the speaker. Wordly, KUDO AI, Interprefy’s AI mode, and purpose-built deployments run this architecture.

Consecutive interpretation: Speaker pauses; interpreter translates; speaker resumes. Used for small-group meetings, depositions, medical consultations. Less latency-critical; AI handles it well because there’s no race against a live speaker.

Whisper interpretation (chuchotage): Interpreter whispers to one or two listeners in a meeting. Mobile app use case: the listener holds a phone or headset. Google Meet’s speech translation (GA January 2026) and Apple’s real-time translation in AirPods occupy this space.

Pre-recorded dubbing / subtitling: Not simultaneous. Post-production. Excluded from this playbook — though it shares a lot of stack with live dubbing.

The market: two curves compounding into each other

AI simultaneous interpretation lives at the intersection of two fast-growing markets.

Business Research Insights pegs the AI simultaneous interpreting market at $0.5 B in 2023, growing to $2.3 B by 2032 — a 19.1% CAGR. Fortune Business Insights has the broader video-conferencing market at $41.6 B in 2026, on track to $65.7 B by 2034 (5.9% CAGR). The interpretation share is the fastest-growing slice inside that.

Slator and CSA Research have tracked remote simultaneous interpretation (RSI) adoption since 2020. The pattern is consistent: human-only RSI is growing ~8%/year; human+AI hybrid is ~25%/year; AI-only is up ~40%/year off a smaller base. KUDO’s own numbers show 200% YoY growth in pure-AI sessions from 2024 to 2026.

The demand driver that matters most in 2026 isn’t cost savings — it’s scope. Human interpretation was rationed. A company would hire interpreters for the annual kickoff, not for the weekly town hall. AI makes interpretation cheap enough to turn on for every meeting with international participants. That’s a 10–100× increase in usage, not a unit-price compression.

The interpretation pipeline: six stages, end to end

Almost every deployment — whether it’s Zoom AI Companion or a custom WebRTC build — uses the same six-stage cascade. The interesting decisions are what runs where, and how tightly the stages are coupled.

1. Audio capture. Microphone input from the speaker’s client. WebRTC, SIP, or raw RTP. Sample rate 16 kHz minimum (48 kHz preferred for TTS output). Noise suppression (RNNoise, Krisp, or platform-native) runs here.

2. Voice activity detection and turn segmentation. VAD isolates speech from silence. Turn detection predicts when the speaker has finished a thought. Deepgram’s Flux integrates both directly, emitting start-of-turn and end-of-turn events without a separate model. Silero VAD and WebRTC VAD remain the open-source defaults. Silence threshold: 300–500 ms is typical for conversational turns.

3. Streaming ASR. Speech to text, incrementally. Target P50 latency under 500 ms from audio arrival to first token. Deepgram Nova-3 Multilingual, Google Chirp 3, AssemblyAI Universal-Streaming, Microsoft Azure Speech, and Whisper-v3 (self-hosted via faster-whisper or whisper.cpp) are the 2026 options.

4. Machine translation. Source text to target text. Modern LLMs (GPT-5, Claude Sonnet 4.6, Gemini 2.5) have overtaken dedicated MT engines (DeepL, Google Translate) on COMET scores for most top-30 language pairs. DeepL and Google still win on raw latency (50–100 ms) and predictability. For 100+ languages, Meta NLLB-200 is the open-source fallback.

5. Text-to-speech. Translated text to audio. ElevenLabs Flash v2.5 (~75 ms TTFB, 32 languages) and OpenAI gpt-realtime (speech-to-speech bidirectional) are the low-latency leaders. Google Chirp 3 HD, Amazon Polly Neural with bidirectional streaming, and Microsoft Azure Neural TTS round out the cloud options. For voice cloning — so the output sounds like the original speaker — ElevenLabs Voice Design and Meta SeamlessExpressive.

6. Delivery. Caption overlay (WebVTT, EBU-TT-D for broadcast) or audio replacement/mix. WebRTC SFUs (LiveKit, mediasoup, Janus, ion-sfu, Amazon Chime SDK) fan out per-language streams to participants. Each listener selects their language client-side.

Cascade vs direct speech-to-speech

The theoretically cleaner approach is end-to-end: audio in, audio out, no text intermediate. Meta SeamlessM4T v2 (and its streaming variant, SeamlessStreaming) can do this for ~100 input and 36 output languages. OpenAI gpt-realtime can do it across its supported set. Google Translatotron 3 is in research preview.

In practice, in 2026, 85–90% of production deployments still use the cascade. Here’s why:

We pick direct S2S for use cases where prosody and voice preservation dominate — media dubbing, high-end executive communications, creative content. We pick cascade for everything else, including almost every enterprise deployment.

Model landscape: who ships what in 2026

The vendor list is long. The short answer is: Deepgram or AssemblyAI for ASR, an LLM (Claude, GPT-5, Gemini) or DeepL for MT, ElevenLabs or gpt-realtime for TTS, and Meta Seamless for the rare direct-S2S use case. Everything else is a detail.

Streaming ASR

• Deepgram Nova-3 Multilingual — 45+ languages, speaker diarization, smart formatting, auto language detection. $0.0092/min multilingual streaming. The default for mid-to-large events.
• AssemblyAI Universal-Streaming — ~300 ms P50 latency, $0.15/hour. Universal-3 Pro drops P50 to ~150 ms (purpose-built for voice agents).
• Google Chirp 3 — multilingual ASR, ~$0.016/min streaming. Strong on non-English, regional availability still mostly US.
• Microsoft Azure Speech — enterprise-grade, custom models, regional deployment. Pairs naturally with Teams.
• OpenAI Whisper v3 / Whisper Large v3 — self-hosted via faster-whisper, WhisperX, whisper.cpp. Free if you eat the compute; robust on 99+ languages.
• Meta SeamlessM4T v2 ASR head — 100 input languages. Used as a single-model alternative to the cascade.

Machine translation

• DeepL — BLEU 0.53, TER 19.6 in comparative studies. Fastest MT on the market for high-resource pairs. ~50 ms per sentence.
• Google Translate — broadest language coverage (130+ pairs). BLEU ~0.45–0.50. 50–80 ms.
• Claude Sonnet 4.6 / GPT-5 / Gemini 2.5 — LLMs now beat dedicated MT on COMET for most top-30 pairs, especially with context and glossary prompting. Latency 200–500 ms (slower than dedicated MT, but better quality). Used via streaming completions.
• Meta NLLB-200 — 200 languages, open-source, self-hostable. 75% of supported languages are low-resource. Quality drops hard outside top 100.
• Amazon Translate, Microsoft Translator — enterprise MT with custom terminology support.
• Llama 3.3, Mistral Small 3 — open-weight LLMs for self-hosted translation. Useful when data cannot leave your infrastructure.

Streaming TTS

• ElevenLabs Flash v2.5 — 32 languages, ~75 ms TTFB, ~$0.10/1k chars. The current leader for low-latency multilingual TTS.
• OpenAI gpt-realtime — direct speech-to-speech bidirectional. Audio input ~$32/M tokens, audio output ~$64/M tokens. Latency under 300 ms model-inference.
• Google Chirp 3 HD — streaming TTS paired with Chirp 3 ASR. Good prosody, strong on non-English.
• Amazon Polly Neural (bidirectional streaming) — 2026 added bidirectional streaming; integrates with Amazon Chime SDK for full event pipelines.
• Microsoft Azure Neural TTS — wide voice catalog, streaming API, emotional prosody tuning.
• Hume AI EVI 3 — emotional prosody (sighs, laughs, emphasis). 11 languages. 1.2 s practical end-to-end in a conversational loop.
• PlayHT — voice cloning, real-time streaming. Useful for speaker voice preservation.

Direct speech-to-speech

• Meta SeamlessM4T v2, SeamlessStreaming, SeamlessExpressive — 100 input / 36 output, preserves prosody, preserves speaker voice. Open-source. The best-in-class option for preserving emotion.
• OpenAI gpt-realtime (cross-lingual mode) — speech-to-speech across its supported language set. Prompt-driven instruction makes it easy to build a custom simultaneous-interpretation agent.
• Google Translatotron 3 — research preview. Limited deployment.

Platform landscape: what’s actually shipping

Conferencing suites with built-in AI interpretation

Most enterprise buyers don’t need to build. The big four have meaningful offerings in 2026.

If you’re already on one of these and the built-in captions cover your languages — you’re done. Don’t build. The incremental value from a dedicated platform is in quality, terminology control, voice cloning, large-event reliability, and on-prem / EU-residency compliance.

Dedicated AI interpretation platforms

Latency budget: where the milliseconds go

Human simultaneous interpreters run at a 3–5 second ear-voice span (EVS). That’s not their reaction time — it’s the time they need to hear enough content to translate meaningfully. AI can, in principle, run tighter. In practice, 2–4 seconds is the sweet spot.

Two optimizations matter most. First, pick ASR and MT providers in the same cloud region (co-locate Deepgram + Claude or Chirp + Gemini in us-east1). Cross-region hops cost 40–150 ms. Second, start TTS speculatively — as soon as the MT emits the first clause, don’t wait for the full sentence. ElevenLabs Flash supports incremental input. Parallelizing TTS with the tail of the MT shaves 200–400 ms off perceived latency.

Cost model: AI vs human, per hour

This is where the buying case usually closes. Professional simultaneous interpretation requires two interpreters per language (they rotate every 20–30 minutes to prevent fatigue errors). For a full-day event in three languages, you’re hiring six interpreters.

“DIY cascade” costs assume Deepgram Nova-3 Multilingual ($0.55/hour), Claude Sonnet 4.6 MT (~$0.40/hour at typical speaker word rates), ElevenLabs Flash v2.5 (~$0.80/hour per output language). Scale linearly with languages; one speaker fans out to N languages, so ASR+MT cost scales with speaker hours, TTS scales with speaker-hours × output-languages.

The operational cost that DIY numbers hide: engineering time, infrastructure, SFU operations, caption UX, glossary management, vendor integrations, on-call. Realistically, a DIY interpretation layer costs a small engineering team ~$200 k to build and ~$80 k/year to run. That’s only worth it if you have high-volume usage (thousands of hours/year) or differentiated product needs (custom voices, proprietary glossaries, EU data residency, broadcast-grade reliability).

Compliance: GDPR, EU AI Act, HIPAA, ADA, broadcast

Interpretation sits on top of voice data, which is one of the most regulated data categories on the planet. Most AI interpretation projects hit at least two of these.

• GDPR (EU) — voice is personal data; voiceprints are Article 9 special category (biometric). You need a Data Processing Agreement with every ASR/MT/TTS vendor, lawful basis, explicit consent for biometric features, DPIA, EU data residency, and right-to-delete tooling. EDPB 2024–2025 guidance tightens enforcement.
• EU AI Act Article 50 — enters force June 2026. Any AI system that interacts with humans must disclose it clearly. Translated captions powered by AI must be labeled as such; voice replacement needs an opening disclosure. Non-compliance risk: administrative fines up to €15 M or 3% global turnover.
• HIPAA (US healthcare) — cross-border telehealth uses interpretation. Voice transcripts often contain PHI. Translation vendors must sign Business Associate Agreements; encryption at rest and in transit; 6-year retention; audit logging. Most commercial ASR SDKs are not HIPAA-eligible by default.
• ADA Title III / Section 508 (US) — meetings, webinars, and public-facing events require real-time captions. 95%+ accuracy target. Section 508 extends this to government systems.
• FCC CVAA (US broadcast) — live captions on video content delivered online. Applies to any “video programming distributor.”
• Court / legal interpreting — AI is not accepted in most US and EU courts. Depositions, immigration hearings, asylum interviews still require certified human interpreters. AI can support (prep, glossary, transcription) but cannot substitute.
• Medical interpreting standards (ATA, IMIA, NCIHC) — AI adoption remains controversial. High-risk scenarios (consent, diagnosis, medication) require certified human interpreters. AI is accepted for routine / administrative communication.
• BIPA (Illinois) — any voiceprint processing on Illinois residents needs written consent, published retention, destruction timeline. Active class-action venue.

Integration patterns: where AI interpretation plugs into your stack

Four patterns dominate. Pick based on how much of the media stack you already own.

Pattern A: SaaS overlay on Zoom / Teams / Meet. Wordly, KUDO, Interprefy join the meeting as a participant, receive the speaker audio, and publish captions to a side panel or a dubbed audio output via the platform’s interpretation channels. Zero integration work. Buy, turn on, done. Fits 60–70% of enterprise buyers.

Pattern B: Conferencing platform built on WebRTC SFU (LiveKit, mediasoup, Janus, ion-sfu). You add an interpretation server as an SFU participant. It subscribes to the speaker track, runs the ASR→MT→TTS cascade, and publishes N output tracks (one per language). Client selects its language via track subscription. Clean, scalable, used by most custom video platforms we build.

Pattern C: Broadcast / streaming pipeline with caption-first delivery. HLS or DASH output with separate subtitle tracks per language (WebVTT or EBU-TT-D). The translation layer produces captions only; voice dubbing happens post-production for recorded content. Standard for live news, sports, conferences streamed to public audiences.

Pattern D: Mobile whisper-interpretation apps. Single-user, client-side or API-backed. Apple Translate (AirPods integration), Google Meet Speech Translation, and purpose-built apps like Google Translate Conversation mode. The mobile stack we cover in the mobile voice playbook.

Mini case: interpretation layer on a hybrid-event platform

An enterprise-events client ran 40+ multi-language webinars a year on a custom LiveKit platform. Pre-2026, they hired 4–6 human interpreters per event and paid $60 k/year for the service plus $180 k/year for the platform team. AI interpretation was clearly cheaper, but they’d tried Zoom captions and found the quality insufficient for pharma and medical audiences with heavy terminology.

We built a cascade layer inside their existing LiveKit cluster:

• ASR: Deepgram Nova-3 Multilingual, with a custom medical/pharma keyword boost list of ~2 500 terms.
• MT: Claude Sonnet 4.6 with a glossary prompt pattern (per-event custom glossaries loaded as system context). DeepL fallback for latency-critical paths.
• TTS: ElevenLabs Flash v2.5 for five target languages (Spanish, French, German, Japanese, Portuguese).
• Captions: LiveKit data channels, WebVTT rendering client-side.
• Compliance: EU data residency (Deepgram EU region, Claude via AWS Bedrock eu-central-1, ElevenLabs enterprise with DPA), GDPR DPA, EU AI Act Article 50 disclosure at event start.

Result after three months: P50 latency 1.4 s, P95 2.3 s. Caption accuracy against human transcription ≈96% on source, ≈92% on translation targets. Event cost dropped from ≈$1 500/event (human interpreters) to ≈$80/event (AI). The pharma customer accepted AI-only captions after 30-day pilot. The client kept human interpreters on standby for their highest-stakes regulatory briefings. Total year-one savings ≈$55 k; integration cost $140 k one-time + $18 k/year ops.

Quality metrics: what to measure

Don’t trust vendor marketing numbers. Measure in production.

• WER (Word Error Rate) — ASR quality. <5% excellent on English; 5–15% acceptable on other top-30 languages; >20% signals a wrong model or bad audio.
• COMET / MQM — MT quality. COMET is the modern neural metric; correlates better with human judgment than BLEU. Aim for COMET >0.80 on high-resource pairs; human sampling for the rest.
• BLEU / chrF — MT quality, older but still reported. DeepL achieves 0.53 BLEU in comparative studies; Google Translate 0.45–0.50; LLMs 0.45–0.55 depending on prompting.
• MOS (Mean Opinion Score) — TTS naturalness. ElevenLabs Flash >4.2 in user studies; Hume EVI 3 >4.1. Target >4.0.
• End-to-end latency — P50 and P95. Caption: <1 s P50, <1.5 s P95. Voice dubbing: <2 s P50, <3 s P95.
• EVS (Ear-Voice Span) — end-to-end speaker-to-listener delay, perceived. The single number your end users care about.
• Terminology hit rate — percentage of domain-specific terms (product names, drug names, proper nouns) that the translation gets right. Glossary-dependent. Aim for >95%.
• User satisfaction — post-event survey, 5-point scale. Target >4.0 on “I could follow the content in my language.”

Accents, dialects, and low-resource languages

AI interpretation degrades gracefully on high-resource languages and falls off a cliff on low-resource ones. Plan around the tier system.

Top 30 (English, Mandarin, Spanish, French, Arabic, Hindi, Japanese, Korean, Portuguese, Russian, German, Italian, Turkish, Vietnamese, Polish, Thai, Indonesian, Tagalog, Dutch, Swedish, Greek, Czech, Hungarian, Romanian, Hebrew, Finnish, Danish, Bulgarian, Norwegian, Slovak): excellent ASR and MT across all vendors. WER <10%, BLEU >0.45, MOS >4.0. You can ship AI interpretation and it will be understood.

30–100 (Swahili, Yoruba, Tamil, Telugu, Kannada, Ukrainian, Urdu, Farsi, Pashto, and ~60 others): acceptable. NLLB-200 shows 70% improvement over prior SoTA on African and Indian languages. BLEU 0.30–0.40, WER 15–25%. Expect errors; disclose AI limitations; keep human fallback for critical sessions.

Below 100: poor. No commercial TTS voice actors. Toponyms, cultural context, and idiom fail often. Use humans.

Accent robustness: heavy non-native accents degrade ASR WER 15–30% versus native speaker baseline. Code-switching (mid-sentence language change) is handled by multilingual models (Deepgram Nova-3 Multilingual, Chirp 3 auto-detect) but still drops accuracy. Regional dialects (Scottish English, Brazilian vs European Portuguese, Arabic dialects) vary widely with training data coverage.

5 pitfalls that kill AI interpretation projects

1. Benchmarking in the cloud, deploying to home Wi-Fi. Your cloud-to-cloud tests will show 800 ms end-to-end. Real users on home broadband with jitter will see 2.5 s P95. Measure in production.

2. Ignoring glossaries. Pharma, legal, financial, and technical domains have hundreds of proper nouns, acronyms, and domain terms that generic MT butchers. Per-event or per-customer glossaries are the single biggest lever for perceived quality. Budget 10–20% of integration time for glossary work.

3. Under-specifying compliance early. GDPR data residency, HIPAA BAA, and EU AI Act Article 50 disclosure are architecture-level decisions. Retrofitting them costs 3–5× more than building them in. Scope compliance in week 1, not week 10.

4. Skipping the human fallback. For high-stakes events — board meetings, investor briefings, regulatory hearings — keep a human interpreter on standby. KUDO and Interprefy hybrid tiers exist for this. The cost of a $2 k/day interpreter is insurance against a $2 M PR incident.

5. Choosing direct S2S too early. SeamlessM4T v2 is impressive. It’s also a moving target, with less operational tooling than the mature cascade. Unless prosody preservation is a core product differentiator, use the cascade. Revisit direct S2S in 12–18 months.

When NOT to use AI interpretation

Be honest about the limits. These are the cases where a human interpreter is still the right answer.

• Court proceedings, depositions, immigration interviews, asylum hearings. AI is not accepted in most US and EU jurisdictions. Certified human interpreters required.
• High-risk medical: informed consent, diagnosis, psychiatric evaluation, medication instruction. Liability and patient safety require certified humans.
• Diplomatic / classified government work. Data residency and security requirements typically exclude commercial cloud vendors.
• Creative content: literature, poetry, film, theater. Metaphor, cultural allusion, wordplay. MT quality is poor here and probably will be for years.
• Low-resource languages below the top 100. Quality is unreliable; no professional TTS voices.
• High-stakes board meetings, investor relations briefings, M&A negotiations. Not a technology limit — a risk-tolerance limit. When one mistranslation can move $100 M, you pay for humans.

A decision framework — pick your stack in five questions

1. How many hours per year? <100 h: stay with humans or use a platform (Zoom AI Companion). 100–1000 h: SaaS interpretation (Wordly, KUDO AI). >1000 h or differentiated product: custom build on WebRTC SFU.

2. How many language pairs? 1–5 top-30 pairs: any vendor works. 10+ pairs or low-resource: need NLLB-200 or enterprise MT with custom terminology.

3. Captions or voice dubbing? Captions-only: simpler pipeline, <1 s latency achievable, cheaper. Voice dubbing: need TTS, longer latency, voice-cloning concerns.

4. What’s your compliance surface? GDPR + EU AI Act only: any major vendor with DPA. HIPAA: Nuance DAX, Abridge, or BAA-signed enterprise vendors. Court / regulatory: stick with humans.

5. Who owns the media stack? Using Zoom/Teams/Meet/Webex: pattern A (SaaS overlay). Running your own WebRTC SFU: pattern B (integrated cascade). Broadcast / streaming: pattern C (caption-first).

Integration playbook: the 10–14-week path

For a SaaS overlay project (Pattern A), compress to 4–6 weeks: skip the SFU integration, keep compliance and pilot. For a broadcast project (Pattern C), add 4–6 weeks for CVAA compliance wiring and CDN caption delivery. If you want a concrete timeline for your specific stack and usage volume, book a 30-minute scoping call — we’ll walk through the phases week by week.

Where AI interpretation is heading in 2026–2027

Direct speech-to-speech becomes mainstream. SeamlessM4T v3 and successors will close the operational tooling gap. We expect 30–40% of new deployments to use direct S2S by end of 2027, led by media and creative use cases.

Speaker voice preservation becomes table-stakes. By 2027, most AI interpretation will output in the speaker’s own voice. ElevenLabs Voice Design, SeamlessExpressive, and PlayHT already enable this.

Sub-second end-to-end latency becomes normal. The cascade gets tighter (co-location, speculative TTS, wait-k tuning) and direct S2S brings theoretical floors down. P50 <1 s for voice dubbing will be the 2027 standard.

Regulatory consolidation. EU AI Act Article 50 is the template; expect US state-level (California CPRA successor) and Asian (Japan, Korea) equivalents by 2027. Disclosure UX becomes a regulated design pattern.

Court and medical adoption inches forward. Don’t expect AI to replace certified humans in high-stakes settings in 2026–2027. Expect more AI-augmented workflows (prep, transcription, glossary) and experimental pilots in lower-stakes court and medical contexts.

FAQ

What to read next

Sum-up

AI simultaneous interpretation in 2026 is a buying decision for most organizations and a build decision for a few. The cascade (ASR → MT → TTS) wins on modularity, terminology control, and operational maturity; direct speech-to-speech wins on prosody preservation and theoretical latency but still lags in production tooling. The economics are decisive: AI replaces $5–13 k/day human interpreter budgets with $300–2 500/day SaaS or $30–400/day DIY spend. Latency and quality are good enough for most enterprise use cases in top-30 languages, as long as glossaries and compliance are done early. High-stakes legal, diplomatic, and medical interpretation remains human territory — and probably will through 2027.