AI Agents on WebRTC 2026: Architecture, Cost & Testing

Updated October 2026: speech-to-speech costs re-priced on gpt-realtime-2.1 (now ~$0.04–0.11/min, not $0.25+), new section on testing and monitoring WebRTC voice agents, EU AI Act Article 50 dates after the Digital Omnibus.

Why Fora Soft wrote this playbook

We have shipped real-time video and voice products for 21 years and 250+ launches. WebRTC is the layer we touch every day — from BrainCert’s virtual classroom (1M+ learners, 500M+ minutes delivered across 10 datacenters) to the software we developed for Blaze Streaming Media’s Volo, which puts live captions and translated audio on attendees’ phones through a QR code. Blaze runs Volo at conferences such as Black Hat USA, GDC and HIMSS.

When we say “AI agents on WebRTC” we mean the production reality: a peer joining the SFU as a participant, listening to a live RTP stream, calling a model in 200ms, streaming TTS back before the user finishes their next breath. That is harder than it looks. This guide is the synthesis we hand our own architects when a client asks “what should I actually build?” — an opinionated map of the four architectures that work, the cost math behind each, and the pitfalls that quietly burn pilots. It is also the blueprint behind our LiveKit AI agent development projects, where the agent joins the room as one more participant.

Read it as a CTO would: skip to the comparison matrix, the cost model, or the decision framework if that is what you need first. Or book a 30-min architecture review and we will walk through your stack live.

What actually changed in 2024–2026

Three things moved AI + WebRTC from demo to deployment. First, native speech-to-speech models — OpenAI’s Realtime models and Google’s Gemini Live ingest audio and emit audio without intermediate transcription, cutting two layers of latency and preserving prosody. Second, ultra-fast TTS: ElevenLabs Flash v2.5 ships at ~75ms model latency and Cartesia quotes sub-90ms for Sonic-3.6 (vendor figures, 2026), which makes the “I heard you” signal land before users get impatient. Third, agent frameworks built on top of SFUs: LiveKit Agents, Pipecat and Daily/Vapi turned a six-week integration into a one-week scaffold. LiveKit Agents — the same open-source WebRTC stack OpenAI runs under ChatGPT’s Advanced Voice — reached 1.0 in April 2025 and now ships an audio-based turn detector, adaptive interruption handling and native MCP tool support.

What moved in 2026 alone. OpenAI released gpt-realtime-2 on 7 May 2026 and gpt-realtime-2.1 (plus a cheaper 2.1 mini) on 6 July 2026; the original gpt-realtime and gpt-realtime-mini shut down on 20 January 2027. GPT-Live 1, a full-duplex voice model billed at $0.05 per session minute plus backend model usage, became generally available on 10 September 2026 (OpenAI API changelog). Google made Gemini 3.8 Live generally available on 15 September 2026 as its default model for low-latency voice agents (Gemini API release notes). Our OpenAI Realtime API pricing breakdown tracks what each of these does to the per-minute bill.

The result: a conversational AI that joins a WebRTC room as a participant, hears the audio track, holds a 30-turn conversation with state, hands off to a human with full context, and costs less per call than a human seat. In 2023 that was a demo. In 2026 it is the new baseline expectation in customer support, sales qualification, telehealth intake and live tutoring.

WebRTC in 90 seconds, for product owners

WebRTC is the open standard browsers and mobile apps use to exchange audio, video and data with sub-200ms latency. The full primer is in our WebRTC architecture guide; the short version is four pieces:

• Media tracks — audio and video carried over UDP/SRTP with built-in DTLS encryption.
• Signaling — the side-channel (your own server) that exchanges SDP offers/answers so peers can find each other.
• ICE / STUN / TURN — NAT traversal so two peers behind firewalls actually connect.
• SFU or P2P topology — for two participants peer-to-peer is enough; from three up you almost always want a Selective Forwarding Unit (LiveKit, mediasoup, Janus) routing media.

For an AI agent, the SFU is the connection point: the agent dials in as a normal participant, subscribes to the audio track of every human in the room, and publishes its own synthesized audio back. No special protocol — the same WebRTC the browser uses. That symmetry is what made the integration tractable.

The four architectures that actually ship

Almost every production AI + WebRTC system collapses into one of four patterns. Pick by your latency budget, your willingness to be vendor-locked, and the call volume you need to serve.

Figure 1. The four production patterns plus fully on-prem, placed on latency (vertical) and cost per minute (horizontal), October 2026 pricing.

1. Speech-to-speech (OpenAI Realtime, Gemini Live)

A single multimodal model takes audio in and emits audio out. The agent connects via WebRTC (browser) or WebSocket (server) directly to OpenAI or Google. Lowest end-to-end latency (about 300ms), best prosody, simplest code — but you cannot swap the LLM, you cannot inspect the “reasoning,” and the per-minute cost is still above a tuned cascade. Billing is per audio token, not per minute: gpt-realtime-2.1 charges $32 per 1M audio input tokens ($0.40 cached) and $64 per 1M audio output tokens, and the 2.1 mini charges $10 / $20 (OpenAI pricing, October 2026). For the production wiring, see our OpenAI Realtime API production guide and the official Realtime API docs.

2. Cascading STT → LLM → TTS pipeline

The classic stack: Deepgram or AssemblyAI streams transcripts; an LLM (a fast “mini” or “flash” tier from OpenAI, Google or Anthropic, or an open 8B–70B model) generates a response; ElevenLabs or Cartesia synthesizes speech back. Orchestrated by LiveKit Agents or Pipecat. Total latency 500–800ms, but every layer is swappable, observable and cheap. Most enterprise deployments live here.

3. SIP / PSTN bridge to WebRTC agent

For inbound or outbound phone calls. A SIP trunk (Twilio, Telnyx) terminates into a WebRTC SFU; the agent runs the same as in pattern 2. Telephony adds 100–300ms of jitter, but the upside is PSTN reach and recognized regulatory paths. Vapi and Retell AI are the managed wrappers; LiveKit, Pipecat and Plivo are the build-it-yourself routes.

4. Multimodal vision agent on a live video track

The agent subscribes to both audio and video tracks, sends sampled frames to a vision-capable model (gpt-realtime-2.1 accepts image input; Gemini Live takes video frames natively), correlates with speech, and replies. Per-frame inference adds 400–800ms, so frame rates are dropped to 1–3 fps for reasoning while audio stays full-rate. Used for KYC verification, in-app guided onboarding, manufacturing QA, and remote field support. See our deep dive on multimodal agents with LiveKit for the production wiring.

Comparison matrix — the four architectures, side by side

The latency budget — where every millisecond goes

Humans start to perceive a pause around 250–300ms of silence. By 500ms it feels “a beat slow,” by 800ms users believe the line dropped, by 1.5s they hang up. To hold conversation feel, the round trip from “user stops talking” to “agent first audio plays” needs to land under 800ms — and ideally under 500ms.

In a cascade, the budget breaks down roughly as:

Figure 2. A typical cascade latency budget — the two orange stages are where tuning pays off most.

Two stages dominate: end-of-turn detection and LLM TTFT. Cut endpointing aggressively and the agent talks over users; pick a slow LLM and the entire pipeline stalls. The art is moving the LLM’s first sentence into TTS as soon as the first 8–15 tokens stream out (“speculative speaking”), so audio starts before the model finishes thinking.

Turn-taking, barge-in and why naive VAD breaks the demo

Latency makes the agent fast. Turn-taking makes it polite. Most failed pilots we have audited had clean STT and reasonable LLMs but felt unnatural because the agent stepped on the user, paused too long after backchannels, or could not be interrupted mid-sentence.

A production turn manager needs four layers:

1. Continuous VAD — a 30–50ms window classifier (Silero VAD, WebRTC VAD) that flags presence of voice. Cheap, runs on CPU, fires on any sound including coughs and keyboard clicks.

2. End-of-turn model — a small model that scores “is this user actually done?” from the audio itself. LiveKit’s turn detector now reads words plus intonation and rhythm in 14 languages (LiveKit, June 2026); Pipecat’s open Smart Turn v3.2 is an 8MB BSD-licensed model covering 23 languages that runs in about 65ms on a cloud CPU; Deepgram Flux returns end-of-turn events from the STT stream itself. Cuts ~300ms off naive 800ms silence timeouts without choppy interruptions.

3. Acoustic Echo Cancellation — mandatory if the agent’s TTS plays through the user’s speaker and the mic re-captures it. Without AEC the agent will “hear itself” and respond to its own voice. Browsers ship usable AEC; native apps and headless agents need to wire it explicitly.

4. Instant TTS pre-emption — the moment a real interruption is detected (not just a “mm-hmm”), the audio buffer is flushed, the LLM generation cancelled, the conversation state rewound to the partial response, and STT picks up the new user turn. Target ~300ms from interruption start to silence on the agent side.

Figure 3. The four layers that turn a fast agent into a polite one, handling barge-in inside ~300 ms.

The tooling layer — pick your framework, then your providers

Below is the shortlist of frameworks our team actually uses in production. None of them are wrong; they just optimize for different things.

For STT we usually pick Deepgram Nova-3 ($0.0077/min list, $0.0048/min promotional streaming rate) or Deepgram Flux when we want end-of-turn detection inside the STT stream ($0.0077/min list, $0.0065 promo) (Deepgram pricing, October 2026); AssemblyAI is our second choice for accented English. For TTS, Cartesia Sonic-3.6 when latency is the bottleneck and ElevenLabs Flash v2.5 or the newer Eleven v4 Turbo (~100ms) when voice quality must convince. For LLMs the picture is fluid — an open 8B-class model on a fast-inference host for cost, a frontier model for tool-heavy workflows, and an in-house fine-tune for any domain you cannot afford to ship to a third party. If LiveKit is your starting point, our LiveKit for AI agents guide maps the full production stack, and the LiveKit Agents docs cover the framework APIs.

A reference architecture you can copy

A clean cascade for a B2B SaaS support voice agent looks like this:

  Browser / Mobile (WebRTC client)
        |  audio + video tracks
        v
  LiveKit / mediasoup SFU  ────────►  Recording & transcript store (S3 + Postgres)
        |  audio track subscribed
        v
  Agent worker (Python / Node)
   ├─ VAD (Silero)
   ├─ End-of-turn model
   ├─ Streaming STT (Deepgram Nova-3)
   ├─ LLM orchestrator (LangGraph / Pipecat)
   │   ├─ tool calls -> CRM / KB / payments
   │   └─ guardrails + PII redaction
   ├─ Streaming TTS (Cartesia Sonic-3.6)
   └─ Barge-in / pre-emption controller
        |  publishes synthesized audio
        v
  SFU -> user
        |
        └─► Observability (OpenTelemetry -> Datadog / Grafana)

Every box on the right is optional in a pilot but mandatory at production: recording for QA, observability for latency SLOs, guardrails for PII redaction, tool calls for the agent to actually do something useful (book a meeting, refund an order, check a KB). The interesting design decision is where state lives — we keep conversation history in Redis with a 24-hour TTL and dump full session transcripts to Postgres for long-term analytics.

A worked cost model — 100,000 call-minutes a month

Numbers cut through architectural arguments faster than diagrams. Take a typical mid-market customer support deployment: 100,000 user-minutes per month, average call length 3.5 minutes, agent talks ~40% of the time. Below is the monthly bill for the three architectures we ship most often.

Inputs, as of October 2026: LiveKit Cloud Scale is $500/month with 50,000 agent-session minutes, then $0.01/min (LiveKit pricing); Deepgram Nova-3 lists at $0.0077/min; Cartesia Sonic 3 through LiveKit Inference is $0.03 per minute of speech, here 40,000 minutes. The Realtime line uses the ~$0.08/min planning midpoint from our pricing breakdown; the 2.1 mini would roughly halve it. That leaves a ~2.7× gap between speech-to-speech and a managed cascade, and ~7–8× against self-hosting — much narrower than the 6× we measured on 2025 prices, because realtime audio got cheaper and caching now works. We typically recommend the managed cascade for the first 6–12 months (it ships fastest and the team builds operational muscle), then migrate the LLM and TTS layers to self-hosted once volumes justify it. Numbers are illustrative; real bills depend on call mix, voice cloning costs, and reserved-instance discounts.

Figure 4. Monthly bill for the same 100,000-minute support workload across three architectures (October 2026 pricing).

Mini case — real-time translation software for Blaze Streaming Media’s Volo

Situation. Blaze Streaming Media wanted live captions and translation for in-person conferences, with nothing for attendees to install.

Plan. We developed a browser app that attendees open from a QR code, a producer console and an admin panel. Speech recognition runs on Speechmatics, translation on Google Translate and voiceover on Google TTS; Vonage carries the speaker’s audio, and captions reach phones over WebSockets.

Outcome. Blaze runs Volo at conferences such as Black Hat USA, GDC and HIMSS. Want a similar walkthrough of your own real-time AI stack? Book a 30-min architecture review.

Security and compliance — what to design for from day one

WebRTC encrypts media in transit by default (DTLS-SRTP). The compliance work is everywhere else: at the SFU (which terminates encryption to route), at the LLM provider (which sees transcripts), at storage (recordings and transcripts), and at the consent layer (you must disclose AI participation in most jurisdictions).

1. HIPAA (US healthcare). You need a signed BAA with every vendor that touches Protected Health Information — SFU, STT, LLM, TTS, storage. LiveKit Cloud, Deepgram and OpenAI all offer BAAs on enterprise tiers; ElevenLabs only on its enterprise plan. PHI in transcripts must be encrypted at rest and access-logged.

2. GDPR (EU users). Data residency is the trap — the SFU and the LLM must run in the EU if any user data touches them. OpenAI offers EU data residency on enterprise; many open-source self-hosted stacks are simpler than the paperwork.

3. SOC 2 Type II. Required by most enterprise procurement. Audit covers security, availability and confidentiality of the whole stack. Pick vendors that already carry it (LiveKit, Deepgram, AssemblyAI, OpenAI, Cartesia) so the chain is intact.

4. PCI-DSS. If the agent ever needs a card number, route the audio segment through a tokenization vendor (Cresta, AudioCodes) so the LLM never sees the raw PAN. Never let the LLM see a raw card number.

5. Consent & AI disclosure. California, Illinois, Colorado and the EU AI Act (Article 50) all require clear disclosure that the user is talking to an AI. The EU duty to tell people they are talking to an AI has applied since 2 August 2026. The AI Digital Omnibus, in force since 27 July 2026, left that date alone but gave generative systems already on the market before 2 August 2026 until 2 December 2026 for machine-readable marking of AI-generated audio; systems launched after 2 August get no grace period. Bake the disclosure into the first 5 seconds of every call and log the user’s acknowledgement.

Five pitfalls that quietly kill production agents

1. Cold-start TTFT. If the LLM container scales to zero, the first call after a quiet period waits 2–5 seconds for model load. Keep at least two warm replicas with a synthetic ping every 30 seconds, or use a managed endpoint that holds memory for you (Groq, OpenAI, Anthropic).

2. Context bloat. A 30-turn conversation can balloon past 10K tokens; LLM cost grows linearly, latency super-linearly. Summarize every 8–10 turns into a rolling state object and drop the raw transcript from the prompt; keep the full transcript in storage for QA.

3. Verbose TTS. Per-character TTS pricing means a chatty model is an expensive one. Cap responses at ~80 words, instruct the LLM to be terse, and prefer extractive answers (“your last invoice was $87”) over generative ones (“great question, let me explain…”).

4. No human handoff path. The agent will fail. Your stack must transfer the call to a human with the full transcript, a one-paragraph summary, and the open intent — not a cold transfer that makes the user repeat themselves. Measure handoff success rate as a top-line KPI.

5. Treating the SFU as “just plumbing.” SFU placement decides geography of latency: a US-East SFU serving an APAC user adds 200–300ms before the agent even hears them. Pick an SFU vendor with edge POPs that match your user map, or self-host on Hetzner / OVH / Equinix in your top regions.

How do you test and monitor a WebRTC voice agent?

Short answer: test a WebRTC voice agent in four layers — the media transport, speech and timing, the conversation itself, and sampled production calls — and block a release when any layer misses its threshold. A clean audio path does not prove the agent answered correctly, and a perfect transcript does not prove the user heard it on time, so QA for a WebRTC agent needs both media metrics and conversation checks.

Transport thresholds are a starting point, not a law. The loss, jitter, RTT and MOS limits above follow the warning and release-block levels Hamming published for WebRTC voice agents in May 2026 (Hamming, 2026); calibrate them by codec, device and region. Browser stats also miss the phone leg: on a SIP-bridged agent, measure the PSTN side separately.

Conversation tests run in CI. LiveKit Agents ships a test framework that asserts on messages, tool calls and handoffs turn by turn in pytest or Vitest, plus Agent Simulations with a simulated caller in text or full audio (LiveKit docs). We keep a golden set of 50–200 redacted, consented real calls and replay it on every prompt, model or provider change; dedicated platforms such as Hamming, Coval and Cekura add load tests and synthetic callers with accents and background noise.

Monitoring closes the loop. In production, trace every turn with OpenTelemetry so a slow answer points to one stage (endpointing, LLM, TTS or the network), sample 1–2% of calls for human review, and alert on the KPIs in the next section rather than on averages.

KPIs — what to measure once you are live

Quality KPIs. P50 and P95 round-trip latency (target P50 < 600ms, P95 < 1s), word error rate on STT (< 8% for support workloads), interruption false-positive rate (< 5%), AEC self-trigger rate (~0).

Business KPIs. Containment rate / call deflection (% of calls fully handled by AI, target 50–75% in support, 80%+ in self-service queries), CSAT on AI-handled calls (within 0.3 of human-baseline), cost per resolved interaction (target < 25% of human cost), upsell or qualified-lead rate on outbound.

Reliability KPIs. Agent uptime (≥ 99.9%), call drop rate (< 0.5%), handoff success rate to human (> 98% with full context), guardrail catch rate on PII / unsafe content (track precision and recall separately).

When not to ship an AI agent on WebRTC

Three scenarios where the answer is no, or not yet. Highly emotional or safety-of-life calls — suicide hotlines, post-trauma intake, end-of-life decisions. AI is not the right first responder; route to humans, use the AI for backend support and post-call summarization. Domains with hallucination cost > latency benefit — legal advice, controlled-substance prescribing, regulated financial advice. The cost of one wrong sentence outweighs three months of saved minutes. Volumes below ~5,000 minutes a month — the integration, monitoring and compliance overhead does not amortize. Use a chatbot or a human until you cross the threshold.

A decision framework — pick your architecture in five questions

Q1. What is the strict latency ceiling? Under 400ms means speech-to-speech (OpenAI gpt-realtime-2.1, Gemini 3.8 Live). 500–800ms means a managed cascade. Above that, anything works.

Q2. What is the unit-cost ceiling per call? Below ~$0.04/min forces a cascade with cost-tuned providers (Deepgram + a small or open LLM + Cartesia). A budget of $0.05–0.11/min opens up speech-to-speech and the simpler vendor stacks.

Q3. Where do calls originate? Inside your app → pure WebRTC. From phone numbers → SIP bridge through Vapi, Retell or LiveKit Telephony. Both → build the cascade once, plug both transports.

Q4. What compliance regime applies? HIPAA + EU GDPR + PCI together push you toward LiveKit Cloud + AssemblyAI + Anthropic Claude on AWS Bedrock with EU data residency, or fully self-hosted open-source.

Q5. Will the agent need vision? If yes (KYC, telehealth, field service), build on LiveKit Agents or Pipecat with a vision-capable realtime model or a separate vision model running at 1–3 fps. Otherwise, stick to audio-only — vision triples cost and latency.

Use cases that already pay back in 2026

Tier-0 customer support. Voice agent fields password resets, order status, billing FAQs and routes everything else with full context. Industry deployments report 50–80% containment on these queries; cost per resolved interaction drops to roughly a quarter of a human seat.

Sales discovery and qualification. The agent runs the first call, asks ICP questions, scores the lead, books a meeting on the AE’s calendar. Useful for high-volume inbound where SDRs cannot keep up. See our deeper write-up on AI call assistants for vendor specifics.

Telehealth intake and follow-up. AI gathers symptoms, medication list and consent before the clinician joins, then handles routine post-visit follow-ups. Pairs naturally with our AI telehealth video platform development — HIPAA-grade WebRTC plus AI agent with PHI guardrails.

Live tutoring and onboarding. The agent joins a class, follows the lesson, answers learner questions, summarizes for the teacher. Built well, it lifts engagement without replacing the instructor — the same pattern BrainCert uses across 1M+ learners.

Real-time meeting copilots. Live transcription, action items, summaries delivered to email in minutes. The infrastructure is the same as a voice agent — WebRTC SFU + STT + LLM — minus the synthesized voice. Our meeting translation comparison covers the adjacent space.

A 12-week roadmap from zero to live agent

Weeks 1–2 — discovery and call analysis. Pull 200 representative recorded calls. Tag intents, escalation triggers, PII patterns, and the failure modes the agent must handle. Pick the architecture using the framework above.

Weeks 3–6 — pilot build. Wire SFU + STT + LLM + TTS for one intent (e.g. order status). Ship to internal users first; instrument latency and CSAT.

Weeks 7–9 — closed beta. 5–10% of real traffic, A/B against human baseline. Wire human handoff with context. Tune endpointing, AEC, and prompts.

Weeks 10–12 — production rollout. Expand to 100% of the chosen intent, add observability dashboards, set SLOs, plan the second intent. By end of week 12 you should have a defensible cost-per-resolution number and a clean migration plan to add the next workflow.

What is coming next — trends to budget for in 2026–2027

Native multimodal models replace cascades. Single models that ingest audio + video + text and emit speech will collapse three-vendor stacks into one API call; OpenAI’s full-duplex GPT-Live 1 (GA September 2026) and Gemini 3.8 Live with asynchronous function calling are the first production-grade steps. Expect lower latency and tighter cross-modal reasoning, at the cost of even more vendor concentration.

Edge inference normalizes. Open-source 7–13B LLMs running on NVIDIA Jetson or local AMD GPUs bring sub-200ms latency and zero data egress. Regulated industries (defense, healthcare, public sector) are the early adopters; expect retail and field-service to follow.

Agent-to-agent calls. Two AI agents negotiating a refund, scheduling between calendars, or completing supplier onboarding. Still experimental; loop prevention and the question of which agent has authority to commit are not solved yet.

Voice biometrics inside WebRTC. Identity confirmed by the user’s voice during the call rather than a separate step. Reduces friction in banking and healthcare; brings new privacy regulations.

FAQ

Is WebRTC secure enough for AI agents handling sensitive conversations?

WebRTC media is encrypted with DTLS-SRTP by default, which covers the wire. The exposure points are the SFU (it terminates encryption to route packets), the LLM provider (it sees transcripts), and your storage. For HIPAA, GDPR or SOC 2 you need BAAs / DPAs with every vendor on that path and access controls plus audit logs on storage.

Do I have to rebuild my WebRTC product to add an AI agent?

Almost never. The agent joins your existing SFU as a normal participant: it subscribes to the audio tracks of the humans in the room and publishes its own synthesized audio. If your SFU is LiveKit, mediasoup or Janus the integration is days, not weeks. The work that takes time is the prompt design, guardrails, observability and human-handoff path — not the WebRTC layer itself.

How long does a production deployment really take?

A focused pilot for one intent ships in 4–6 weeks. A production rollout with monitoring, compliance, human handoff and at least two intents typically takes 12 weeks. Multimodal vision agents add another 2–4 weeks for vision pipeline tuning.

What is the cheapest sane stack for a production voice agent?

A self-hosted Pipecat orchestrator on a Hetzner box, mediasoup as the SFU, an open Whisper-class model for STT, an open 8B-class LLM on a fast-inference host or local GPU, and an open-weight TTS. At 100,000 minutes a month the infrastructure lands around $0.01–0.02/min. The trade-off is you own the operations, observability and security work.

Will the agent replace our human support team?

Almost no successful deployment we have seen replaces humans wholesale. The pattern that works is AI handles the bulk of routine queries (50–80% containment) so humans focus on complex, emotional and revenue-critical conversations. Headcount usually stays flat while volume per agent doubles.

How do you measure AI agent success in production?

Three buckets: quality (P95 latency, WER, interruption accuracy), business (containment rate, CSAT vs. human baseline, cost per resolution), reliability (uptime, drop rate, handoff success with full context). Pick a primary KPI per call type — usually containment or conversion — and track everything else as guardrails.

How do you test a WebRTC voice agent before launch?

Run four checks: WebRTC media quality under injected packet loss and jitter (block above 3% loss, 50ms jitter or MOS under 3.5), per-stage latency with a P95 round trip under 1 second, scripted multi-turn scenarios that assert on tool calls and human handoff, and a replay of real redacted calls on every prompt or model change.

How do you use WebRTC for bi-directional voice and video streaming in an AI agent?

Either let the browser open a WebRTC session straight to a speech-to-speech model such as OpenAI Realtime or Gemini Live, or run the agent as a participant in your SFU: it subscribes to the user’s audio and video tracks, sends audio to STT or a realtime model and sampled frames to a vision model, and publishes its synthesized voice back as an audio track. The SFU route is the one to pick when you need recording, multiple humans or a provider you can swap.

What are the alternatives to a provider-specific SDK for real-time audio agents?

Use an open orchestration framework — LiveKit Agents or Pipecat — on standard WebRTC. Both treat STT, LLM and TTS providers as plugins, so you can move from one vendor’s model to another without rewriting the client, and both can self-host the media path so transcripts and recordings stay in your cloud.

Speech-to-speech (OpenAI Realtime) or cascade — which should I start with?

If you need to ship in days and call duration is short, start with OpenAI Realtime; you will validate the product before you optimize the stack. If unit economics matter from day one or compliance forces you to inspect every layer, start with a cascade on LiveKit Agents or Pipecat. Many of our clients prototype on Realtime, then migrate the chatty intents to a cascade once volume justifies it.

Can the agent handle non-English languages and accents?

Yes, but choose providers carefully. Deepgram Nova-3 Multilingual and Flux Multilingual, AssemblyAI and OpenAI’s gpt-live-transcribe all handle mixed-language audio; turn detection is multilingual too (LiveKit’s detector covers 14 languages, Smart Turn v3.2 covers 23). Cartesia and ElevenLabs ship multilingual voices; for languages outside the top 30, expect to fine-tune. Latency and accuracy both degrade somewhat outside English — budget extra QA time.

What to read next

Ready to put an AI agent on your WebRTC stack?

AI agents on WebRTC moved from showpiece to production baseline in two years. Speech-to-speech models give you about 300ms latency for premium voice experiences, now at $0.04–0.11 a minute. Cascading pipelines built on LiveKit or Pipecat give you 2–3× lower unit costs managed, up to ~8× self-hosted, and full observability. Either way, the SFU layer your team already runs is the connection point — the integration is days of code on top of months of tuning, compliance and QA.

If you have a real-time product and you are not at least piloting an AI agent, your competitors are. That is what our AI integration and voice agent development team builds day to day, on top of the WebRTC layer. The opportunity is to ship the right architecture for your unit economics, not the trendiest one — and to engineer the unglamorous parts (turn-taking, AEC, handoff, compliance) that decide whether the agent feels human or not.