Build AI Agents on LiveKit: 2026 Multimodal Guide

Building AI agents on LiveKit puts voice, vision, and tool use into one live session. In 2026 that stopped being a demo and became the default way real companies answer phones, triage patients, and run field support. One open-source framework sits under a surprising share of those deployments: LiveKit. It powers OpenAI’s ChatGPT voice mode, it just raised $100M at a $1B valuation (Bloomberg, January 2026), and its Agents SDK is what most teams reach for when a voice bot has to work on a real 1-800 number, not a laptop.

We’re Fora Soft, a software development company: 625+ delivered projects, real-time video and audio since 2005, a 100% Upwork Success Score. This is how we build production agents on LiveKit, and it isn’t a getting-started tutorial. It’s a build guide for the parts that actually decide whether a project ships: the architecture that holds sub-500ms latency, the model choice that controls your unit economics, the SIP path to the phone network, the compliance chain nobody mentions at a demo, and the cost math that separates a $0.40 call from a $6 one.

If you’re weighing whether to build on LiveKit, what to budget, and how to dodge the three or four mistakes that quietly kill voice-AI projects in their first 90 days, start here.

Why LiveKit runs the 2026 voice-agent stack

Short answer: LiveKit solved the boring, hard part (moving low-latency media reliably) and then gave that plumbing an agent framework on top. Three years ago a voice agent meant hand-stitching a Twilio media stream, an STT vendor, an LLM, a TTS vendor, a jitter buffer, and a brittle WebSocket layer. Every handoff added delay and every vendor added a bill.

LiveKit Agents 1.x ships those pieces together on open-source WebRTC: a room abstraction for audio, video, and data tracks; SIP for real phone calls; Turn Detector v1 for knowing when a human is done talking; and noise cancellation tuned for 8kHz telephony audio. You point the same agent at OpenAI’s gpt-realtime, Google’s Gemini Live, or a self-hosted pipeline with a config change, and you run it on LiveKit Cloud or in your own VPC. The media layer stays open source, so you’re never locked in.

The traction is real, not marketing. LiveKit is the infrastructure behind ChatGPT’s Advanced Voice Mode, reports 100,000+ developers on the platform, and carries billions of calls a year. We’ve shipped real-time media since 2005 — our video-evidence platform V.A.L.T runs across 770+ organizations with 50,000+ users — and the building blocks that platform needs (multi-track sync, recording, security) are the same ones a 2026 agent leans on. That overlap is why we bet on this stack.

What counts as a multimodal agent in 2026

“Multimodal” used to mean a chatbot that could caption a photo. In 2026, multimodal AI agents reason over several live streams at once — usually voice, video, and structured tool outputs — and answer in whichever modality fits. Four patterns cover almost everything we build:

• Voice-first with tools. A support or sales agent that talks, listens, and calls internal APIs (CRM, scheduling, payments). Roughly 80% of production work.
• Voice plus live video. Telehealth triage, field-service support, insurance intake. The agent watches a camera feed, grounds its reply in what it sees, and can annotate a shared screen.
• Voice plus screen share. Onboarding, tech support, training. The agent reads the user’s screen as another video track and walks them step by step.
• Voice plus telemetry. Robotics and remote operations. The agent fuses sensor streams with speech and issues commands.

The hard part isn’t wiring modalities together; LiveKit hands you a unified room with all three track types. The hard part is the context budget: what the model sees, how often, at what resolution, and for how long. Get that wrong and your token bill doubles every month. We come back to that math in the cost section.

The reference architecture on LiveKit Agents 1.x

Every production LiveKit voice agent we ship has the same five-layer shape. Naming the layers matters because most scaling pain comes from mixing concerns between them.

Figure 1. The five-layer LiveKit agent stack. A native realtime model collapses perception, reasoning, and expression into one endpoint, which is why latency drops.

With a native realtime model (gpt-realtime, Gemini Live) layers 3-5 collapse into one endpoint. That’s why latency falls to sub-300ms and why it’s the default for most 2026 builds. The trade-off: you give up swapping STT and TTS vendors independently, and audio-token pricing can surprise you on long calls. If you’re choosing that layer, we compare the best text-to-speech APIs in a separate guide.

In code, the 1.x entry point is an AgentSession that binds a model to a room. Install the SDK with the plugins you need:

pip install "livekit-agents[openai,silero,turn-detector,noise-cancellation]"

To build a voice agent with LiveKit, one file is a complete, runnable worker. This is the whole realtime-path agent, ready to answer a room:

from livekit import agents
from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import openai, noise_cancellation

class Assistant(Agent):
    def __init__(self):
        super().__init__(instructions="You are a concise support agent.")

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()
    session = AgentSession(
        llm=openai.realtime.RealtimeModel(voice="marin"),
    )
    await session.start(
        agent=Assistant(),
        room=ctx.room,
        room_input_options=RoomInputOptions(
            noise_cancellation=noise_cancellation.BVC(),
        ),
    )

if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

Run python agent.py dev to test locally and python agent.py start in production. Prefer the classic pipeline (swappable STT, LLM, and TTS, and easier to keep inside a HIPAA BAA)? Change the session, nothing else:

session = AgentSession(
    stt=deepgram.STT(model="nova-3"),
    llm=openai.LLM(model="gpt-4.1-mini"),
    tts=cartesia.TTS(),
    turn_detection=MultilingualModel(),
)

Tools are plain functions with a decorator; the model calls them by name, and a tight schema keeps token cost and hallucinations down:

from livekit.agents import function_tool

class Assistant(Agent):
    @function_tool
    async def lookup_order(self, order_id: str) -> dict:
        "Look up a customer order by its ID."
        return await crm.get_order(order_id)

That one model seam is why teams standardize on LiveKit instead of a bespoke pipeline. Full working examples, including SIP inbound and outbound, live in the open-source livekit/agents repo and the LiveKit Agents docs. The transport underneath is ordinary WebRTC; for the ground truth there, see our WebRTC production architecture guide.

Model choice: gpt-realtime vs Gemini Live vs STT-LLM-TTS

This is the single biggest design decision, and there’s no “best model.” It comes down to four things: your latency target, your vision needs, language coverage, and per-minute budget. Here’s how we decide in 2026.

Audio bills by time: gpt-realtime charges $32 per million input audio tokens and $64 per million output, where a user minute is roughly 800-1,200 input tokens and an agent minute 1,500-2,000 output tokens (OpenAI pricing, 2026). A three-minute call with a 60/40 human-to-agent split and no retries lands near $0.28-$0.42 in model cost alone. We walk the full stack in the cost section.

For regulated work we often pair a realtime model for conversation with a classic pipeline fallback (Deepgram plus an LLM plus Cartesia) that can sit inside a HIPAA BAA or run on-premises. LiveKit’s plugin system makes that a one-line switch, so you can keep both paths in the same codebase. Our OpenAI Realtime API production guide has the model-side detail.

The latency budget: hitting sub-500ms

“Feels natural” has a number: end of the user’s turn to first audible token under 500ms for web and mobile, under 300ms for telephony. That budget is easier to break than to keep. On a well-tuned gpt-realtime deployment it splits like this.

Figure 2. Where the 500ms goes. A native realtime model removes the separate TTS step; co-locating regions is the biggest single win.

• Network in (WebRTC + jitter buffer): 30-80ms.
• VAD and turn detection: 40-120ms. Turn Detector v1 adds roughly 25ms of inference and noticeably cuts false interruptions versus silence-only VAD.
• Model time-to-first-token: 180-280ms — the biggest slice.
• TTS or audio first chunk: 20-60ms. Native audio models skip this entirely.
• Network out: 30-80ms.

The most common latency bug we find on audits is region mismatch: LiveKit egress in us-east-1 but the model endpoint in eu-west adds 80ms per trip. Co-locate them. The second is oversized context on every turn. Fix it with a rolling summary and cached tool results, which also cuts your token bill.

Turn detection, interruptions, and natural flow

Turn detection is where demos look magical and production falls apart. Plain silence-based VAD handles maybe 80% of calls, then fires on every “um,” mid-thought pause, or bit of background noise. LiveKit’s Turn Detector v1 listens to the audio directly, fusing what was said with how it was said, and posts the strongest end-of-turn results across English and 13 other languages. It runs free on LiveKit Cloud and is the default in 1.x.

We layer three controls on top of it in production:

• Don’t-yield lists. Specific utterances (reading a policy, confirming an order) are set so a cough can’t cut them off.
• Noise cancellation on ingress. Trained on telephony audio, it fixes turn accuracy in noisy rooms more than any model upgrade.
• Per-use-case barge-in. A sales agent yields the instant the caller speaks; an emergency-triage agent finishes its current sentence first.

Adding vision: live video and VLM grounding

Gemini Live’s video track and gpt-realtime’s image input both attach to LiveKit as extra tracks. The pattern that works: the agent samples one frame every 500-1500ms — not 30fps, which would blow up your token bill — and requests a single high-res look on demand when it needs detail. Subscribe, sample, downsample, send.

We used a version of this on V.A.L.T, syncing nine HD streams across 22+ camera models with SSL/RTMPS security. The multimodal building blocks (track subscription, frame sampling, recording) overlap heavily with what a 2026 agent needs, which is why a real-time media background shortens the build. More on the computer-vision side in our video anomaly-detection guide.

Telephony and SIP: agents on the PSTN

If your agent answers a 1-800 number, you need SIP. LiveKit’s SIP bridge connects inbound and outbound calls as normal room participants, handles DTMF, warm and cold transfers, and conference bridging. Bring a trunk from Twilio, Telnyx, Vonage, Plivo, or your own Asterisk/Kamailio — LiveKit is provider-agnostic.

Three things to get right from day one: turn on noise cancellation (telephony audio is 8kHz and hostile to ASR), set SIP dispatch rules for country-specific routing (carrier pricing swings 10x), and wire recording with redaction before your first production call. A card number captured into a training set is uninsurable. Our AI call-assistant buyer’s guide covers the telephony vendor options in more depth.

Multi-agent orchestration and tool use

The 2026 pattern is rarely one giant prompt. It’s a front-line agent that handles intake and small talk, plus specialist agents it hands off to: scheduling, billing, escalation. Agents 1.x makes the handoff swap the room’s agent implementation without tearing down the call, so the caller hears one continuous conversation.

For tools, wire them at the agent level instead of passing the kitchen sink every turn. A tight schema — 5 to 12 tools, strict JSON, well-typed arguments — costs 20-40% fewer tokens and gives the model far fewer chances to hallucinate a call. For anything touching money or PHI, we add a human-in-the-loop step: the agent proposes, a person approves. If you want the general patterns beyond LiveKit, our guide to AI agents on WebRTC maps them out.

The real 2026 cost math

Here’s an honest stack for a gpt-realtime agent answering a 1-800 number: three-minute average call, 60/40 talk split, 1,000 calls a day.

Figure 3. Per-call economics. The agent’s all-in ~$0.38 versus $7-$12 for a human is why voice AI moved from pilot to production so fast.

That $11.4K a month handles 30,000 calls. The same volume staffed by humans at $9 a call is $270K a month. The unit economics are the whole story. Where projects derail is always the same three token-bloat traps, and each has a one-line fix:

• Re-sending the whole transcript every turn. Fix: a rolling summary plus the last few turns, not the full history.
• Unbounded model output. Fix: a max-response-tokens cap and a strict system prompt, so the agent doesn’t monologue.
• Un-cached reference data. Fix: cached prompt input at $0.40 per million — a 98% discount — for policies and catalogs the model re-reads all day.

Those three changes cut token spend 40-60% on a typical deployment. If your per-call cost is north of $1, one of them is the culprit.

LiveKit Cloud vs self-hosted: when each wins

For about 80% of the projects we see, LiveKit Cloud is the right call. It removes four months of infrastructure work, gives you global media relays, ships observability, and meets SOC 2 and HIPAA out of the box. Self-hosting earns its keep in three cases:

• Sovereign data residency — EU public sector, some DACH enterprise, Middle East government.
• An existing SFU/MCU — you already run Jitsi, Janus, or mediasoup and want LiveKit Agents alongside it.
• Very high volume — past roughly 10M agent-minutes a month the dedicated-cluster math can favor self-hosting, but only with platform-engineering capacity on staff.

The hosting decision itself is the easy part; agent evaluation is the real bottleneck. We compared the underlying providers in our AWS vs DigitalOcean vs Hetzner breakdown, and the discipline that keeps a build honest is in our QA-at-every-stage playbook.

Observability, evals, guardrails, and the KPIs that matter

The two questions engineering leaders ask in 2026 are “how do we know the agent is working?” and “how do we stop it going off the rails?” We ship every agent with four layers of production hygiene:

• Session recording with PII redaction — LiveKit Egress to object storage, automated redaction before any analyst hears audio.
• Turn-level evals — a small LLM judge scores each turn on relevance, safety, and tone; scores drive dashboards and regression alerts.
• Golden-set regressions — 100-500 recorded calls that must pass every deploy, auto-replayed against the new agent.
• Guardrails — out-of-domain detection, PII and PCI filters on inputs and outputs, response-length caps, per-caller rate limits.

Measure three buckets, each with a hard threshold, or you’re flying blind:

• Quality KPIs. Containment (share of calls resolved without a human) above 40% for tier-1; task-success above 85%; barge-in error rate under 5%.
• Business KPIs. Cost per resolved call, deflection rate versus your old IVR, and CSAT within two points of a human baseline.
• Reliability KPIs. p95 first-token latency under 500ms, session error rate under 1%, and transcription word-error-rate under 15% on your real audio, not a clean benchmark.

Compliance: HIPAA, EU AI Act, PCI, and recording law

Compliance is the number-one reason we see voice-AI projects stall past pilot. The 2026 reality, dated precisely:

• EU AI Act. Under Article 50, from 2 August 2026 a voice agent must tell people they’re talking to AI; the machine-readable marking rule for generated media (Article 50(2)) now applies from 2 December 2026. The May 2026 Digital Omnibus pushed the heavier high-risk duties back: standalone Annex III systems (credit, hiring, insurance, essential services) must comply by 2 December 2027, and product-embedded Annex I systems by 2 August 2028. The dates moved once already, so build the disclosure and audit trail in from day one.
• HIPAA. You need a Business Associate Agreement with every vendor that touches PHI — LiveKit, the model vendor, STT, TTS, and recording storage. Plan the four-to-six-vendor BAA chain before you write code, not after.
• PCI-DSS. If the agent might hear a card number, redact audio on ingress, route to a vault such as Very Good Security or Skyflow, and keep card data out of the prompt entirely.
• Call-recording law. Two-party consent varies by state — California, Florida, Illinois, Washington, and Pennsylvania are strict — and disclosure at the start is mandatory across most of the EU. We inject the disclosure as the agent’s first sentence in regulated deployments.

What we learned shipping real-time multimodal

A concrete one. A legal-tech client ran V.A.L.T, a video-evidence platform recording interviews across courtrooms and clinics. The brief was scale and reliability: multi-camera capture, tamper-evident recording, and playback that investigators trust, across hundreds of sites.

The plan we ran was ordinary and unglamorous: lock the security model (SSL/RTMPS, access control) before touching features, standardize track handling so 22+ camera models behaved identically, and build a recording pipeline with redaction baked in rather than bolted on. We treated evals as first-class — a golden set of recorded sessions that had to replay cleanly on every deploy.

The outcome held up where it counts: the platform now runs across 770+ organizations with 50,000+ active users and nine synchronized HD streams per session. The same spine — synchronized tracks, redacted recording, replayable evals — is exactly what a compliant 2026 voice or vision agent needs, which is why we reuse it. Want a similar assessment of your stack? Book a 30-minute call and we’ll sketch the path.

High-value 2026 use cases we build

Not every voice agent earns its keep. The five where we see clear ROI right now:

• Telehealth intake and triage. Voice plus optional video in a WebRTC room; cuts medical-assistant time meaningfully and lifts patient satisfaction when it’s well designed.
• Tier-1 support deflection. Replaces IVR trees, resolves 40-70% of calls end-to-end, and warm-transfers the rest with full context.
• Outbound reminders and pre-arrival prep. Voice-only, scales to six-figure daily call volume without new hiring.
• Remote expert plus field technician. Voice plus video from a phone; the vision model grounds the conversation in what the worker sees.
• Real-time interpretation. A bidirectional interpreter over a video call, for telemedicine and legal settings where a human interpreter runs $3-5 a minute.

Pick your stack in five questions

When a client asks “build, buy, or something in between,” we answer with five questions, not a sales pitch.

• 1. What’s your latency target? Under 400ms rules out most STT-LLM-TTS pipelines and points at a native realtime model.
• 2. Does the agent need to see? Live video pushes you toward Gemini Live or gpt-realtime with image input, and up a cost tier.
• 3. What’s your regulatory surface? PHI, card data, or EU users mean the compliance chain drives the architecture, not the other way around.
• 4. What volume, honestly? Under a few million minutes a month, Cloud wins. Past that, self-hosting math starts to matter.
• 5. Who maintains it after launch? No eval-and-guardrails owner means buy, not build — an unmaintained agent regresses silently.

Figure 4. Answer four questions top to bottom and you land on realtime model, pipeline, or self-hosted. Compliance overrides everything below it.

Those five questions usually collapse to one of four paths. Here’s the grid we actually use — pick the row that matches your team and regulatory surface, not the one that sounds most ambitious.

Five pitfalls that kill voice-AI projects

1. Treating it like a chatbot. Text-agent habits — long responses, no barge-in, blocking tool calls — feel broken in voice. Design for interruption and sub-second turns from day one.

2. Skipping evals until launch. Without a golden set and turn-level scoring, you can’t tell whether a prompt tweak helped or quietly broke three flows. Build the eval suite in week one.

3. Ignoring the token budget. Full-context turns and un-capped output turn a $0.40 call into a $4 one. Rolling summaries, caps, and cached inputs are not optional at volume.

4. Leaving compliance for later. The BAA chain and EU AI Act disclosures take longer than the code. Start them first, or watch the launch slip a quarter.

5. One region for a global audience. A single deployment region adds 80-200ms for far-away callers and tanks perceived quality. Put media and model in the caller’s region.

When NOT to build a multimodal agent

Honesty sells better than hype, so here’s when we tell clients to wait. If your call volume is under a few hundred a day, a well-configured off-the-shelf tool beats a custom build on cost and time. If the task is genuinely simple — an appointment confirmation with two branches — a scripted IVR is cheaper and more predictable than an LLM.

Skip vision unless the agent truly needs to see; it triples token cost and adds privacy risk for little gain in most support flows. And if nobody on your side will own evals and guardrails after launch, don’t build — an agent without a maintainer drifts, and a drifting voice agent damages the brand faster than no agent at all. We’d rather scope you a $0 recommendation than a project that regresses in month three.

Realistic build timeline and budget

Every LiveKit agent proposal we write in 2026 lands in one of three tiers. These are numbers we actually quote, not a marketing ladder. They cover the build; run-rate (models, telephony, infra) is separate and lives in the cost section above.

We estimate with Agent Engineering, so these run tighter than typical agency quotes for the same scope. If a number in your head is materially higher, it’s usually paying for coordination overhead we don’t carry. For how we keep estimates honest, see our software-estimating guide; to hire the team directly, our LiveKit agent development page and broader AI integration services lay out engagement models.

FAQ

Is LiveKit free?

LiveKit Server and the Agents SDK are open source under Apache 2.0, free to self-host. LiveKit Cloud is a managed service priced per connection-minute and agent-minute, with a free tier that covers prototypes. Most teams start on Cloud and migrate only if volume or data-residency rules demand it.

What latency should I aim for?

Under 500ms from end-of-turn to first audible token for web and mobile, under 300ms for telephony. With gpt-realtime or Gemini Live on LiveKit Cloud in the caller’s region, 250-400ms is routine. Past 800ms, callers notice the lag and satisfaction drops.

gpt-realtime or Gemini Live — which should I pick?

gpt-realtime for voice-first support, sales, and strong tool use. Gemini Live for vision-heavy jobs (live-video grounding, screen understanding) and wide multilingual coverage. LiveKit’s plugin system lets you prototype both in a day and compare on your own calls.

Can a LiveKit agent answer a real phone number?

Yes. LiveKit SIP bridges your trunk (Twilio, Telnyx, Vonage, Plivo, and others) to an agent as a normal room participant, with DTMF, warm and cold transfers, conferencing, and recording. Always enable noise cancellation for 8kHz telephony audio.

How do I keep costs under control at scale?

Four levers: a rolling summary instead of full context each turn; cached prompt input for reference data (about 98% cheaper per token); strict max-response length and tool schemas; and a smaller model for simple flows. Together they cut token spend 40-60% on typical deployments.

Is this HIPAA-compliant?

It can be. You need BAAs with LiveKit (Cloud supports them), your model vendor, your STT and TTS, and your recording storage — a chain of four to six vendors. Self-hosted LiveKit plus an open model on HIPAA-eligible infrastructure shortens the chain at the cost of more engineering.

How do I test and debug a voice agent?

Record a golden set of real calls, score each turn with an automated judge, and replay the set on every deploy so regressions surface before users do. For live debugging, trace every turn end to end — audio in, transcript, model, audio out, and latency at each hop.

Do I need a full-time MLOps team?

No. For most deployments, LiveKit Cloud plus a managed model API covers what an MLOps team would otherwise build. What you do need is voice-design discipline and a small team owning evals, golden-set regressions, and guardrails. Two people part-time is enough for a mid-sized deployment.

What to read next

Ready to ship a multimodal agent in 2026?

Multimodal agents on LiveKit went from frontier tech to production default in under two years. The teams getting real ROI stopped treating voice AI as a chatbot add-on and started building it as a real-time system, with the latency budget, evals, telephony path, and compliance posture that deserves.

We’ve done real-time video and audio since 2005 and shipped AI-first products across 625+ projects. If you want a partner that ships a scoped pilot in 6-10 weeks and can grow it into a regulated, multi-region platform, let’s talk.