Voice AI Agent on LiveKit: Latency, Stack & Cost 2026

A voice AI agent is software that holds a spoken conversation in real time: it listens with speech-to-text, reasons with a large language model, and replies with text-to-speech, fast enough to feel like a phone call. Building one that actually sounds human is a latency problem first and a model problem second. Get the full round trip, microphone to speaker, under 800 ms on a pipeline stack, or trim the model latency to ~200–300 ms with a speech-to-speech model, and the conversation feels natural. Miss that budget and every caller notices: they talk over the agent, repeat themselves, or press zero for a human. LiveKit Agents is the open-source framework most serious teams reach for to hit those numbers without locking into one vendor.

This playbook is the condensed version of what our engineers learn on day one of a voice-AI project. Fora Soft, a software development company that has shipped 250+ projects since 2005, builds LiveKit voice AI agents for support desks, sales dialers, clinical intake, and in-app assistants, and we have watched every latency trap, cost blowout, and compliance surprise at least twice. If you are a founder, CTO, or senior engineer who needs a real answer to “should we build this, and on what?” by Friday, start here.

Updated October 2026: Claude Haiku 5.5 replaces Haiku 4.5 as our default LLM, GPT-Live-1 and Gemini 3.8 Live in the speech-to-speech picks, current LiveKit Agents 1.8 code and October 2026 cost per minute, plus a new section on prompting and call state.

Why teams pick LiveKit for voice AI agents in 2026

Every voice AI agent team hits the same fork. Either you bolt your agent onto someone else’s managed platform (Vapi, Retell, Bland, Synthflow) and accept their latency, their prompt templates, and their margin, or you build on a realtime transport you control. LiveKit is the second path: an Apache-2.0 realtime media stack with a production-grade Agents framework on top. Shortlisting Python SDKs? The three most teams compare in 2026 are LiveKit Agents, Pipecat and the OpenAI Agents SDK, and we put Pipecat, LiveKit Agents, and the OpenAI Agents SDK side by side. LiveKit is not a side project that disappears next year: it raised a $100M Series C at a $1B valuation in January 2026, and the framework ships as livekit-agents 1.8 in October 2026, with 2.0 in beta since 12 October.

What you get out of the box: WebRTC between your users and the agent, plug-in STT/LLM/TTS providers, a trained turn-detection model, licensed Krisp noise cancellation, per-turn latency metrics, call recording, and PSTN bridging through Telnyx or Twilio. Open source, no lock-in, community plug-ins for every major provider. It is the stack our LiveKit AI agent development team ships voice and multimodal agents on, with sub-500 ms turn-taking as the target.

What you trade: you write Python or Node, you own the ops if you self-host, and you budget one to three engineering weeks for a production agent instead of the three hours Retell advertises. Past a proof of concept, that trade is the right one — the cost breakdown below shows why. For the platform reference and SDK internals, see our LiveKit AI Agents engineer’s guide.

What “human-sounding” actually means

Across ten languages, the typical gap between conversational turns is about 200 ms (Stivers et al., PNAS, 2009). Cross 500 ms and callers consciously notice the delay. Cross a second and they start interrupting, hanging up, or reverting to touch-tone. That is the real performance envelope for a voice AI agent. Not model quality, not voice realism, not tool coverage.

“Human-sounding” is four things stacked together, and most teams obsess over the wrong one:

1. Fast first response. 300–800 ms from end-of-user-speech to start-of-agent-audio, depending on architecture. This is the one that breaks demos.

2. Graceful interruption. The agent stops speaking within 120–200 ms when the caller talks over it, then resumes the right context.

3. Prosody that matches intent. Emphasis on the right words, breath groups that track meaning, expressive voices for empathy-heavy moments.

4. Factual grounding. The agent knows when to say “let me check” and actually checks — through function calling, not a confident guess.

Miss any of the four and the illusion breaks. Most teams pour money into item three (“we need a better voice”) when their real gap is item one. ElevenLabs is not your bottleneck at 75 ms first-byte; your LLM at 1.2 s is.

The 800 ms latency budget and where it leaks

The classic pipeline runs five stages. Here is a realistic best-case budget with 2026 providers, measured end to end on a US-region deployment. The chart makes the point faster than the numbers: the LLM is where the budget goes.

Figure 1. Where the 800 ms goes. The LLM first-token wait is the biggest slice of a pipeline turn; the speech-to-speech bar is model latency, before network.

The two fastest leaks in every production deploy we audit are LLM first-token latency and cross-region hops. Both are fixable. Pick a low-latency model, pin STT, LLM, and TTS to the same cloud region, and stream every stage so audio starts playing before the model finishes thinking. Still missing 800 ms after that? The problem is prompt length, not infrastructure.

Pipeline vs speech-to-speech: which fits your product

Speech-to-speech models skip transcription entirely: audio goes in, audio comes out. In October 2026 the ones to know are OpenAI’s GPT-Live-1 (10 September 2026, $0.05 per minute for the voice layer, full-duplex: it listens and speaks at the same time and hands reasoning and tool calls to a backend model), its turn-based predecessor gpt-realtime-2.1 (6 July 2026), and Google’s Gemini 3.8 Live (stable since September 2026, with asynchronous function calling). OpenAI reports almost 80% fewer interruptions for GPT-Live-1 than for turn-based systems; model latency can hit 200–300 ms, though real voice-to-voice lands closer to the ~800 ms above once network and PSTN are counted. The older gpt-realtime and gpt-realtime-mini snapshots shut down on 20 January 2027, so new builds should start on the current models. The catch is unchanged: you buy reasoning, voice, interruption, and tool calling as one bundle from one vendor.

The pipeline is the opposite trade. You stitch three or four providers, eat 500–700 ms of overhead, and in exchange you can swap the LLM the week Anthropic ships a faster Haiku, point STT at a vertical-specific provider, use a cloned brand voice, and log every stage for audit. Our rule of thumb is below; the decision tree makes it a 30-second call.

Figure 2. The first Yes wins. LiveKit runs both, so you can prototype speech-to-speech and keep the pipeline as your fallback.

Our practical rule: go speech-to-speech for consumer-facing assistants where personality beats precision. Go pipeline for anything that talks to a database, quotes a price, or could land you in court if it hallucinates. LiveKit supports both, and the pipeline path is the one it was designed for. We keep a deeper teardown in our OpenAI Realtime API production guide.

Reference architecture on LiveKit Agents

A production LiveKit voice AI agent has five moving parts and one traffic pattern. The user joins a LiveKit room. An agent worker picks up the job, subscribes to the user’s audio, streams it through STT, passes partial transcripts to the LLM, streams LLM tokens into TTS, and publishes the audio back, all while a turn-detection model decides when the user has actually finished talking.

Figure 3. The whole shape. Everything past the four plug-ins is your business logic: tools, prompts, state, and human handoff.

from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, TurnHandlingOptions, inference
from livekit.plugins import anthropic

load_dotenv(".env.local")

class Scheduler(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a scheduling assistant. Reply in one or two short sentences. "
                         "Always call check_availability before offering a time.",
            tools=[check_availability, book_meeting],
        )

server = AgentServer()

@server.rtc_session(agent_name="scheduler")
async def scheduler(ctx: agents.JobContext):
    session = AgentSession(
        stt=inference.STT(model="deepgram/nova-3", language="en"),
        llm=anthropic.LLM(model="claude-haiku-5-5"),
        tts=inference.TTS(model="cartesia/sonic-3", voice="<voice-id>"),
        turn_handling=TurnHandlingOptions(
            turn_detection=inference.TurnDetector(),  # audio turn model, not fixed VAD
        ),
    )
    await session.start(room=ctx.room, agent=Scheduler())
    await session.generate_reply(instructions="Greet the caller and offer to book a time.")

if __name__ == "__main__":
    agents.cli.run_app(server)

That is the full worker in the AgentServer style the LiveKit quickstart uses in October 2026 (livekit-agents 1.8). STT, TTS and the turn detector run through LiveKit Inference, so one LiveKit key covers them; the LLM comes from the Anthropic plug-in. Run it locally with lk agent dev and deploy to LiveKit Cloud with lk agent create. Everything else, from tools and prompts to state and human handoff, is code you already know how to write.

STT: Deepgram vs AssemblyAI vs Gladia vs Soniox

Speech-to-text is the one stage where latency is cheap and accuracy is everything. A 150 ms versus 300 ms STT barely changes perceived conversation; a 5% versus 12% word-error rate is the difference between “it worked” and “it dialed the wrong customer.” One honesty note: vendor WER figures are best-case. Deepgram’s ~5.3% Nova-3 number is its own benchmark; independent tests on noisy, accented audio land well above it. Test on your own audio, not the datasheet.

Our default is Deepgram Nova-3 for English-first US traffic, or Deepgram Flux when we want end-of-turn detection inside the STT itself. AssemblyAI Universal-3 Pro takes over when the agent reads back account numbers or emails, and Gladia handles call centers with heavy accents or code-switching. Batch transcription models belong in transcript exports, not live calls. Prices above are Deepgram and OpenAI list prices checked in October 2026.

LLM: Claude Haiku 5.5, GPT-6 Luna, or Gemini 3.8 Flash

The LLM is where most voice budgets break. Time-to-first-token matters far more than total throughput — the agent must start speaking quickly, not finish quickly. A 600 ms first token that then streams at 80 tokens/sec feels faster than a 300 ms first token that stalls after one sentence.

Haiku 5.5 is our default for voice since its 7 October 2026 release: Anthropic prices it about 75% below Haiku 4.5, and early customers such as Box report roughly half the latency. GPT-6 Luna matches the price for OpenAI-only stacks. Groq stays on the shortlist for anything latency-obsessed — sales dialers, game NPCs — now on open-weight GPT OSS 120B, because Llama 3.3 70B moved to enterprise-only pricing. Reach for Sonnet 5.5 or GPT-6 Sol only when the agent does genuine reasoning: multi-step tool chains, constrained policy, high-value outcomes. Token prices are vendor list prices and per-minute figures are Retell’s published LLM rates, both checked in October 2026; measure time-to-first-token on your own prompt, because public leaderboards move weekly.

TTS: ElevenLabs, Cartesia, Aura, or Rime

TTS is the part users actually judge, and the stage where prices vary 10x for quality differences most callers never hear. The fast tier got better in autumn 2026: Cartesia shipped Sonic-3.6 on 27 August (listeners preferred it to Sonic-3.5 up to 93% of the time in blind tests), and ElevenLabs released Eleven v4 Turbo on 28 September at about 150 ms from request to first audio over WebSocket.

We reach for Cartesia Sonic when latency and cost matter and brand voice does not, Deepgram Aura-2 for the best naturalness-per-dollar, and ElevenLabs when the client licenses a cloned or talent voice — Eleven v4 Turbo when the agent must switch languages mid-call, Flash v2.5 when every millisecond counts. Spend the extra only where it changes the business case — nobody books a meeting because the scheduling bot has better vibrato.

Turn detection, VAD, and graceful interruption

Voice activity detection answers “is someone speaking right now?” Turn detection answers the harder question: “did they finish, or just pause?” That gap is the difference between a conversation that breathes and one that steps on every other word.

Since June 2026 LiveKit’s turn detector is an audio model: it reasons over the caller’s speech directly instead of waiting for a transcript, which avoids cutting people off at mid-sentence pauses. It covers 14 languages, runs as v1 on LiveKit Cloud and v1-mini on self-hosted agents, and the older text-based detector is deprecated for Agents 2.0. Adaptive interruption handling uses acoustic cues to tell a real barge-in from a backchannel like “uh-huh”; it is the default on LiveKit Cloud, while self-hosted agents use VAD-based interruption.

Interruption is the other half. When the caller cuts in, the agent has to stop within ~150 ms, flush its TTS buffer, and hand the new utterance back to STT. LiveKit does this automatically. The one common bug is the agent interrupting itself, usually because a noisy TTS signal bleeds back through the user’s mic — fix it with echo cancellation on the client or background-voice cancellation on the agent.

Noise cancellation and Krisp: now a line item

LiveKit Cloud ships licensed Krisp models for two jobs: removing ambient noise (fans, traffic, keyboards) and cancelling background voices (a spouse on the phone, an open-plan office). Both cut STT errors and false interruptions on noisy channels.

Budget it as a line item. On LiveKit Cloud in October 2026, voice isolation is included up to 1,000 minutes a month on Ship and 10,000 on Scale, then $0.0012 per minute. The current LiveKit quickstart template ships ai-coustics audio enhancement rather than Krisp, so check which model your project actually loads. Turn it off for clean-room audio — in-app desktop assistants, studio mics — and leave it on for PSTN and mobile traffic, where it pays for itself in accuracy.

Function calling that actually ships

A voice agent that cannot do anything is a chatbot that happens to talk. Function calling is the line between “thanks for calling” and “you’re booked for Tuesday at 2.” Three rules keep it honest in production:

1. Echo back anything the agent captured. “I heard four-one-five, two-two-five… is that right?” STT mishears numbers; AssemblyAI Universal-3 Pro is meaningfully better on entities, but nothing is perfect.

2. Call synchronously, narrate asynchronously. Say “let me pull that up” while the tool runs. A two-second silence reads as a dropped line, and callers hang up.

3. Cap retries and budget. One hallucinating model with one broken tool can redial the same API 40 times in 10 seconds. Hard-cap tool calls per turn and dollars per session.

Current Claude and GPT-6 models both support parallel tool calls, so a well-designed schema lets the agent fire “check calendar,” “look up customer,” and “pull policy” at once and answer in a single turn. That is the biggest hidden latency win we see teams miss.

Prompting and state: a LiveKit voice agent prompt guide

A good LiveKit voice agent prompt is short, written for the ear, and backed by tools and typed session state instead of long instructions. In our builds the system prompt stays at a few hundred words: who the agent is, how it sounds, the one job it does, the tool rules, and when to hand off to a person. Everything that changes during the call — the caller’s name, the booking slot, the step of the flow — lives in code, not in the prompt.

Write for the ear. No markdown, bullet points, emojis or URLs in replies, because TTS reads them literally. Tell the model to answer in one or two short sentences, ask one question at a time, and spell out numbers, dates and email addresses the way a person would say them.

Put rules where the model looks for them. Tool rules belong in the tool description plus one line of the prompt: “Always call check_availability before offering a time.” Guardrails get their own short block: what the agent never promises, what it never reads back, and when it says “let me get a colleague.”

Keep the prompt short, because prompt length is latency. Every token is processed before the first word comes back. Prompt caching makes a long static prefix cheap — Claude Haiku 5.5 charges $0.01 per million cached input tokens against $0.10 uncached (Anthropic, October 2026) — but cheap is not the same as fast.

One job per agent. In LiveKit Agents a tool can return a new Agent, which takes over the session with its own instructions and tools. A greeter that routes to a booking agent and a billing agent beats one 3,000-word prompt that tries to do everything, and each piece is easier to test.

How to manage voice agent state. Keep call state in one typed object on the session, so every tool reads and writes the same fields and every handoff carries them along. Trim or summarize the chat history on long calls instead of letting it grow, and write the state to your backend when the session closes, not after every turn.

from dataclasses import dataclass
from livekit.agents import AgentSession, RunContext, function_tool

@dataclass
class CallState:
    caller_name: str | None = None
    slot: str | None = None
    step: str = "greeting"

session = AgentSession[CallState](userdata=CallState(), stt=..., llm=..., tts=...)

@function_tool
async def book_meeting(ctx: RunContext[CallState], slot: str) -> str:
    """Book the meeting slot the caller confirmed."""
    ctx.userdata.slot = slot
    ctx.userdata.step = "confirmed"
    return f"Booked {slot}"

Reduce perceived latency with fillers. When a tool takes more than about a second, say something: session.say("Let me pull that up") before the call, or play a soft keyboard “thinking” sound through LiveKit’s background audio player. Since Agents 1.6 (June 2026) tools can also run asynchronously, so the agent keeps talking while a slow lookup finishes. Callers forgive a two-second wait they can hear; they hang up on two seconds of silence.

True cost per minute, all in, no surprises

Every managed platform advertises a headline rate — Vapi at $0.05 a minute, Retell at $0.055 for infrastructure, Bland at $0.12–$0.14 on its plans — and none of them is the full bill. Once you add STT, LLM, TTS, and telephony, a Vapi agent lands between $0.11 and $0.33 a minute (September 2026 roundups). A LiveKit stack you assemble yourself runs roughly $0.04 to $0.13 depending on the tier. Here is what three representative LiveKit stacks cost at October 2026 list prices.

Figure 4. All-in cost by stack. The LLM and TTS choices, not the platform fee, decide which tier you land in.

How the totals add up: STT $0.005–$0.008 a minute; the LLM about $0.003 a call minute on Haiku 5.5 and about $0.064 on Sonnet 5.5 (Retell’s published LLM rates); TTS $0.018 on Aura-2, $0.030 on Sonic-3.6, or about $0.034 on Eleven v4 Turbo at $0.04 per 1,000 characters; $0.01 per agent minute on LiveKit Cloud and $0.01 per inbound US minute on LiveKit telephony (LiveKit and vendor pricing pages, October 2026). Multiply by your volume: a sales dialer running 50,000 minutes a month on the Production stack costs about $3,000 in infrastructure, less than one human SDR’s fully loaded cost, and the agent works 24/7.

LiveKit vs Vapi, Retell, Bland, Synthflow

At 10,000 minutes a month — a busy mid-market use case — the platforms sort cleanly by total cost of ownership. The managed tools win on time-to-first-call; LiveKit wins on everything past the first month.

Figure 5. Monthly cost at 10,000 minutes. LiveKit is the cheapest to run and the slowest to stand up — the classic build trade.

All four managed platforms are well-reviewed and genuinely good at what they do; independent roundups rank Retell and Vapi at the top of the developer tier. This table is about total cost and control, not quality. The numbers say the obvious thing: Retell or Vapi for validation and sub-5K-minute prototypes, LiveKit the moment the use case is real. The break-even sits between 10,000 and 20,000 minutes a month — past that, LiveKit’s margin repays the engineering within six months. Already live on Vapi or Retell? We migrate voice agents off Vapi and Retell onto LiveKit and SIP without rewriting the prompts and tools. If you are also weighing WebRTC vendors, our Agora alternatives teardown covers the transport layer under all of this.

LiveKit Cloud vs self-hosted vs Telnyx

Three ways to run LiveKit Agents in production, each a different trade:

LiveKit Cloud. Build is free with 1,000 agent minutes a month, Ship is $50 with 5,000, Scale is $500 with 50,000, then $0.01 per agent minute (October 2026). Managed dispatch, observability, regional edges, SOC 2, and a BAA on Scale and above. Zero ops. The default.

Self-hosted. No platform fee; you pay STT/LLM/TTS directly. Break-even above ~50,000 min/month if your DevOps is already in place. Pick it for regulated workloads where the platform must never touch the audio.

LiveKit on Telnyx (6 April 2026). Telnyx hosts LiveKit next to its own telephony and runs its own STT and TTS on its GPUs; it claims 50% savings against LiveKit Cloud pricing and 200 ms round trips, without publishing the test setup. Worth pricing if you need PSTN at scale. If the agent lives on a phone line, our AI call agent development team wires the SIP trunks, warm transfer, and CRM hooks around it.

One gotcha since February 2026: LiveKit observability data is processed in the US regardless of your media region. If GDPR residency forbids US processing of any call metadata, disable project-level observability and log to your own stack. The media itself stays in the region you pick.

Compliance: HIPAA, SOC 2, TCPA, GDPR

Voice touches three compliance surfaces at once — PHI if users discuss health, PII on every customer record, and recording consent in every state, many with teeth. Platforms help; they do not save you.

HIPAA. LiveKit signs BAAs on the Scale plan and above; OpenAI’s Realtime endpoints are HIPAA-eligible under a BAA with Modified Retention (2026); Deepgram and ElevenLabs offer BAAs at the enterprise tier. Verify every vendor in your stack and keep a countersigned PDF per review cycle.

SOC 2 Type II. LiveKit, OpenAI, Deepgram, AssemblyAI, ElevenLabs, and Cartesia are all certified. Pull their reports for your own audit.

TCPA. Since the FCC declaratory ruling of 8 February 2024, AI-generated voices count as “artificial” under the TCPA: marketing calls need prior express written consent, non-marketing calls need prior express consent, plus caller identification and opt-out. Store consent alongside the phone number and surface it on every dial.

Two-party recording consent. California, Florida, Pennsylvania, Illinois and several other states require all-party consent. Play a disclosure at the start of every recorded call — and record the disclosure itself.

GDPR. Recording consent under Article 6 must be explicit, not “legitimate interest.” Sign data-processing agreements with every provider and pin media to an EU region. On top of GDPR, Article 50 of the EU AI Act applies from 2 August 2026: tell callers they are talking to an AI unless it is obvious, and mark synthetic audio. The 2026 omnibus delayed the high-risk rules, not this one.

Our production defaults: real-time PII redaction on all logs (scrub emails, phone numbers, card numbers before they hit observability), per-call consent metadata, audit trails with 90-day minimum retention, and an annual red-team pass on the agent’s prompt boundaries. Cutting corners here is what ends careers; our teardown of AI agents on WebRTC covers the same compliance surface at the transport layer.

Mini-case: a voice-first coaching agent on LiveKit

One of our EdTech clients, Career Point, wanted a voice-first coaching agent that runs in the browser: a learner speaks, the agent listens, reasons over lesson state, and answers out loud. The hard constraint was the same one this whole playbook is about — if the reply lags, the coaching illusion collapses and learners drift off.

The plan was the reference stack, tuned. LiveKit client SDK in the browser for transport, an agent worker on a Deepgram Nova-3 plus Claude Haiku plus Cartesia pipeline, custom tools for lesson state, region-pinned providers, and the trained turn-detection model instead of a fixed VAD. A vanilla AgentSession starts around 1.2–1.4 s p95; the work is closing that gap without giving up accuracy.

We shipped it at 550–700 ms P50 turn latency on browser WebRTC clients, comfortably under the 800 ms mark, in weeks rather than months, with a recording loop that lets the pedagogy team review real interactions. The full story is in our Career Point coaching case study. Want a similar assessment of your own voice stack? Book a 30-minute scoping call and we will map the latency budget with you.

Use cases that pay for themselves

Four patterns return investment in under six months. Everything else is experimentation worth running small.

Tier-1 support deflection

Password resets, order status, balance checks, policy FAQs. These are the calls a voice agent closes on its own most often, at a few cents per minute against several dollars for a human-handled call. On a large support line, even a modest deflection rate pays for the build within months.

Outbound sales and lead qualification

Inbound-lead callbacks drop from hours to under a minute, because the agent calls back the moment the form is submitted. Speed to lead is where outbound voice agents earn their keep; the agent qualifies, a human closes.

Clinical intake and appointment confirmation

Clinic networks use voice agents for pre-visit history and 24-hour confirmation calls, so front-desk staff handle walk-ins instead of the phone. Most confirmation calls are short and scripted, which is exactly where an agent is reliable. HIPAA stack required — see above.

In-app voice tutors and assistants

This one sits closest to our portfolio: voice-first learning platforms, real-time coaching, and in-browser WebRTC agents. The playbook is the same — LiveKit client SDK in the browser, a Haiku plus Cartesia worker, custom tools for app state, and a feedback loop that records interactions for review.

Seven failure modes we see in production

In rough order of how often they ruin a demo:

1. Cold-start latency. The first call after deploy hits 3–5 seconds. Fix with idle warm-up calls and connection-pool preload on boot.

2. Interruption misfires. The agent talks over the user or cuts them off mid-word. Tune the turn model, enable background-voice cancellation, and test with 20 real users before launch.

3. Hallucinated facts. The agent invents a confirmation number. Force function-call grounding for anything quotable; ban free-text answers to factual queries.

4. Entity-capture errors. “My email is john.doe” becomes “johndough.” Switch to AssemblyAI Universal-3 Pro for entity-heavy flows and echo back.

5. Cascading region latency. STT in us-east, LLM in us-west, TTS in Frankfurt. Pin every provider to one region; pay the egress if you must.

6. Cost runaways. A retry loop fires 40 LLM calls in ten seconds after a failed tool. Hard-cap tool calls per turn and dollars per session.

7. PII in logs. Full transcripts with card numbers land in CloudWatch. Scrub at the edge before anything touches observability.

KPIs to measure from day one

A voice agent without instrumentation is a demo. Track these from the first production call:

Turn latency P50 / P90 / P99. Averages lie; the P99 tail is what makes users hang up.

Task completion rate. Did the agent finish what the caller wanted without escalation? 50–70% is realistic for a well-tuned support bot.

Handoff rate. Escalations to humans per call. If it climbs, a tool is broken or the prompt is drifting.

Cost per successful outcome. Not cost per minute — cost per meeting booked, per reset, per appointment confirmed.

Interruption and WER drift. Rising interruptions mean the voice is too long-winded; sample 100 calls a week and hand-score transcription as the canary for everything else.

A decision framework in five questions

Before you write a line of code, answer these five. They decide build-vs-buy faster than any feature grid.

If two rows describe you, the lower-latency and higher-volume one wins — it is the constraint you cannot engineer around later. Still torn between two rows? That is exactly the 30-minute call we are happy to take.

When NOT to build on LiveKit

Four cases where a managed platform beats LiveKit honestly:

1. Deterministic IVR. “Press 1 for sales, 2 for support.” Use Twilio Studio and move on.

2. A demo due tomorrow. Retell ships in three hours, Vapi in a day. If you need something to show, buy it.

3. Sub-5K minutes a month, forever. The engineering payback stretches past a year; the managed fee is cheaper.

4. No DevOps on the team. Synthflow’s no-code builder beats a broken Python agent nobody can maintain.

The honest question is “will this matter enough to justify owning it?” If the voice AI agent is core to the product, or the savings are real past ten thousand minutes a month, the answer is yes — and LiveKit is the right bet. Otherwise start managed and migrate later.

FAQ

What is a voice AI agent?

A voice AI agent is software that carries a spoken conversation in real time. It transcribes the caller with speech-to-text (STT), decides what to say with a large language model (LLM), and answers with text-to-speech (TTS), either as a chained pipeline you assemble or as a single speech-to-speech model. On LiveKit, those pieces run as swappable plug-ins inside an agent worker.

How do you build a voice AI agent on LiveKit?

Create an AgentServer with an rtc_session handler and wire an AgentSession: streaming STT (Deepgram Nova-3 or Flux), a low-latency LLM (Claude Haiku 5.5), streaming TTS (Cartesia Sonic-3.6), and LiveKit’s audio turn detector. The session subscribes to the caller’s audio, streams partial transcripts to the LLM, and streams tokens into TTS. A scoped agent is roughly 100 lines of Python plus your tools; run it with lk agent dev.

How much does a voice AI agent cost per minute?

All-in at October 2026 list prices: about $0.04 a minute on a budget LiveKit stack (Nova-3, Haiku 5.5 or GPT-6 Luna, Aura-2), about $0.06 on a production stack with PSTN (Flux, Haiku 5.5, Sonic-3.6), and about $0.13 on a premium stack (Sonnet 5.5, Eleven v4 Turbo). Managed headline rates ($0.05–$0.14) are platform fees only; a Vapi agent lands at $0.11–$0.33 once STT, LLM, TTS, and telephony are added.

What is the realistic latency for a sub-700 ms agent?

With Deepgram Nova-3 streaming STT, a Claude Haiku-class LLM, and Cartesia streaming TTS pinned to one region and short prompts, we routinely measure 550–700 ms P50 end to end on WebRTC clients. PSTN hops add 80–150 ms. A vanilla, untuned AgentSession starts around 1.2–1.4 s p95.

Should you use the OpenAI Realtime API instead of a pipeline?

If latency and natural turn-taking matter most and you accept one vendor, yes: GPT-Live-1 (September 2026, $0.05 a minute for the voice layer) listens and speaks at once and hands reasoning to a backend model, and gpt-realtime-2.1 (July 2026) remains the turn-based option. The older gpt-realtime snapshots shut down on 20 January 2027. For compliance, tool-calling reliability, or a cloned brand voice, the pipeline still wins, and LiveKit runs both.

Can LiveKit handle PSTN phone calls or only WebRTC?

Both. LiveKit has its own telephony (US numbers from $1 a month, inbound at $0.01 a minute on Ship, October 2026) and bridges to Twilio or Telnyx over SIP. Since April 2026, LiveKit on Telnyx runs the agent next to Telnyx telephony and claims 50% savings against LiveKit Cloud pricing.

Is a voice AI agent legal for outbound calls?

Yes, with consent. Since the FCC ruling of 8 February 2024, AI-generated voices are “artificial” under the TCPA: marketing calls need prior express written consent and non-marketing calls need prior express consent, plus caller ID and opt-out. Store consent with the phone number and disclose recording where two-party consent applies.

Is LiveKit a good fit for multilingual voice agents?

Yes. Deepgram Flux Multilingual covers 10 languages with end-of-turn detection, Gladia Solaria-1 handles code-switching in a single stream, Soniox leads on 30+ languages, Eleven v4 Turbo speaks 90+ languages, and LiveKit’s audio turn detector supports 14. The pipeline stays the same; only the STT/TTS plug-ins change.

Can the agent hand off to a human mid-call?

Yes. A transfer-to-agent tool is the standard pattern: the LLM (or the caller) requests handoff, the LiveKit room adds a human participant, and the AI steps out. Transcript and context follow, so the human picks up the thread instead of starting over.

What is the best SDK for building a real-time voice AI agent in Python?

In 2026 most teams shortlist three: LiveKit Agents (transport, telephony, and turn detection in one open-source stack), Pipecat (a flexible pipeline framework), and the OpenAI Agents SDK (fastest path if you stay on OpenAI models). We default to LiveKit Agents for production voice because it owns the media path and runs both pipeline and speech-to-speech models.

How do you write a prompt for a LiveKit voice agent?

Keep it short and written for the ear: identity, tone, one job, tool rules, and handoff rules in a few hundred words. Ban markdown, lists, and URLs in replies, ask for one or two short sentences per turn, and spell out numbers and emails. Split big flows into several agents that hand off to each other instead of one long prompt.

How do you manage state in a voice AI agent?

Keep call state in a typed object on the session (AgentSession[CallState](userdata=...) in LiveKit Agents) that every tool reads and writes, pass it along on agent handoffs, trim the chat history on long calls, and persist the state to your backend when the session ends.

How do you reduce perceived latency in a LiveKit voice agent?

Stream every stage and pin providers to one region first. Then cover unavoidable waits: say “let me check” before slow tools, play a soft thinking sound through LiveKit’s background audio player, and run slow tools asynchronously (Agents 1.6+), so the caller never hears dead air.

What to read next

Ready to ship a voice AI agent that sounds human?

The order of operations does not change: win the latency budget first, pick the pipeline unless raw speed and vendor lock-in are both acceptable, and instrument from the first call. Deepgram, Claude Haiku, and Cartesia get most teams to 550–700 ms; region pinning and streaming close the rest.

Build on LiveKit when the agent is core to the product or the volume clears ten thousand minutes a month. Below that, start on a managed platform and migrate when the math flips. Either way, the compliance work — TCPA consent, HIPAA BAAs, PII redaction — is yours to own from day one, and it is what separates a demo from something you can put on a real phone line. Our AI integration team can take it from architecture to a compliant production line.