OpenAI Realtime API: WebRTC, SIP & WebSocket in 2026

Why Fora Soft wrote this playbook

Adding a realtime voice layer looks like a weekend project until the first production call drops mid-sentence, the ephemeral key leaks into a browser bundle, or the bill for a chatty 20-minute session lands at ten times your estimate. The gap between a demo that works and a voice agent that survives real users is where most teams lose a quarter.

We’ve built video and real-time communication software since 2005, and we’ve worked with WebRTC since the standard was young. That’s 250+ shipped products: telehealth intake on CirrusMED (HIPAA, real patients), ultra-low-latency audio on Tyxit (sub-30 ms, live from the Montreux Jazz Festival), and conferencing platforms like ProVideoMeeting. Voice AI is the same plumbing with a new brain bolted on.

This is the document we walk engineering leaders through when they ask, “can we add a realtime AI voice feature this quarter?” You’ll get the 2026 integration code for all three transports, an honest latency budget, cost math with the arithmetic shown, the compliance notes, and the pitfalls we’ve watched other teams walk into. For the groundwork we also keep a WebRTC development page and a sub-second latency playbook.

Short on time? Jump to the transport comparison, the cost math, or the five-question decision framework. The rest is here when you need it.

What the OpenAI Realtime API actually is in 2026

In one sentence: it’s a single, bidirectional audio-and-text session over a persistent connection, backed by a speech-to-speech model that takes streamed audio in and emits streamed audio out, with function calls riding a side channel. No separate speech-to-text. No separate text-to-speech.

The model is gpt-realtime (the stable alias; the current snapshot in 2026 is gpt-realtime-2.1), which reached general availability on 28 August 2025. It replaced the old speech-to-text → LLM → text-to-speech chain, and that’s the whole point: collapsing three services into one removes roughly 200–800 ms of handoff latency and hands turn-taking to a single model that controls its own timing.

Figure 1. The old pipeline chained three services and paid a latency tax at each handoff. gpt-realtime does speech-to-speech in one model, over one socket.

Once the session is open, you stream 24 kHz audio in as it’s captured, receive audio frames back for immediate playback, and exchange JSON events for function calls, session updates, and control (commit, cancel, truncate). The GA release also added the pieces that make it a product rather than a demo: image input, reusable prompts, and remote MCP server support, so the model can call your tools without you wiring each integration by hand.

On OpenAI’s own benchmarks the GA model is a real step up from the December 2024 preview: 82.8% vs 65.6% on Big Bench Audio reasoning, and 66.5% vs 49.7% on the ComplexFuncBench function-calling eval (OpenAI, 2026). Those numbers matter because a voice agent that mis-hears a phone number or calls the wrong tool isn’t charming, it’s a support ticket. New to building AI into real-time media? Our AI for video engineering hub covers the fundamentals.

Six use cases that pay back the build

Not every feature needs sub-second voice. These are the six patterns we see shipping in 2026 with a return that justifies the integration cost.

1. Inbound AI receptionist / call deflection

The highest-value case. The agent handles routine calls — order status, password resets, booking, FAQs — and hands the 20–40% of edge cases to a human. SIP transport, phone number from a trunk provider. Break-even against one human agent is roughly 60 billed hours of call volume a month.

2. Voice copilot inside a SaaS product

A mic button in your web app. The copilot reads the current page context and takes actions through function calls. WebRTC straight from the browser. Common in dashboards, CRMs, and field-service tools.

3. AI-augmented video conferencing

A voice bot that joins the meeting, answers questions, and captures action items. We go deeper on these in AI video conferencing features, and it pairs naturally with our video conferencing development work.

4. Live translation and interpretation

Voice in one language, voice out in another, code-switching mid-sentence. OpenAI even ships a dedicated gpt-realtime-translate model billed at a flat $0.034/min. See our hybrid AI-human translation playbook for when to keep a human in the loop (spoiler: regulated and high-stakes settings).

5. Outbound calling, with guardrails

Appointment confirmations, survey collection, lead qualification. The technology is easy; the regulatory posture is not. Respect two-party consent laws and disclosure rules, and always make it obvious the caller is talking to AI — OpenAI’s usage policy requires it.

6. Voice-enabled telehealth intake

HIPAA territory: symptom intake, scheduling, medication reconciliation by voice. It needs an OpenAI BAA and Zero Data Retention, which we cover below and in our telemedicine development work.

WebRTC vs WebSocket vs SIP

The transport decides where your audio starts, how authentication works, and who carries the bandwidth. Pick first, wire second; retrofitting the wrong one costs more than any code you’ll write.

Figure 2. The one-glance version. The full trade-off table with codecs and echo handling is below.

WebRTC integration, the 2026 way

The browser pattern is a three-step dance: your server mints a short-lived key, the browser opens a peer connection with it, and audio flows both ways while JSON events ride a data channel. What changed in the GA release is the endpoints, and getting them wrong copies a 2024 tutorial straight into a 401.

Figure 3. The ephemeral-token flow. Your server never ships the real key to the browser; the browser exchanges SDP directly with OpenAI using the short-lived key.

Server: mint a short-lived client secret

// server route - your real API key never leaves this box
app.get('/token', async (req, res) => {
  const r = await fetch('https://api.openai.com/v1/realtime/client_secrets', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json',
      'OpenAI-Safety-Identifier': hashUserId(req.user.id)  // ties abuse to a user
    },
    body: JSON.stringify({
      session: { type: 'realtime', model: 'gpt-realtime',
                 audio: { output: { voice: 'marin' } } }
    })
  });
  const data = await r.json();
  res.json({ value: data.value });   // the ephemeral key the browser will use
});

Browser: peer connection + SDP exchange

const { value: KEY } = await (await fetch('/token')).json();
const pc = new RTCPeerConnection();
pc.ontrack = e => { audioEl.srcObject = e.streams[0]; };        // play model audio

const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(mic.getTracks()[0]);
const dc = pc.createDataChannel('oai-events');                 // JSON events ride here
dc.onmessage = e => handle(JSON.parse(e.data));

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// NOTE: SDP now goes to /v1/realtime/calls (not /v1/realtime?model=...)
const sdp = await fetch('https://api.openai.com/v1/realtime/calls', {
  method: 'POST', body: offer.sdp,
  headers: { Authorization: `Bearer ${KEY}`, 'Content-Type': 'application/sdp' }
});
await pc.setRemoteDescription({ type: 'answer', sdp: await sdp.text() });

That’s the entire baseline. From here you send JSON events over the data channel to register tools, update instructions, or cancel an in-flight response; the media just works through the standard WebRTC stack. If you’d rather keep your server in the loop for every session, the unified interface lets the server relay SDP to /v1/realtime/calls itself — simpler, at the cost of one extra hop.

WebSocket integration for server-side agents

WebSocket is the right call when the audio source is a server: bridging an IVR, running a purely server-side agent, or fanning one model session out to your own clients. You hold the real key, base64-encode PCM16 frames at 24 kHz, and listen for response events. OpenAI recommends WebRTC over WebSocket for anything client-side, because you’d otherwise reimplement jitter buffering and echo cancellation by hand.

import WebSocket from 'ws';

const ws = new WebSocket('wss://api.openai.com/v1/realtime?model=gpt-realtime', {
  headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }
});

ws.on('open', () => ws.send(JSON.stringify({ type: 'response.create' })));

ws.on('message', raw => {
  const ev = JSON.parse(raw);
  if (ev.type === 'response.output_audio.delta') {
    playPcm16(Buffer.from(ev.delta, 'base64'));   // 24 kHz mono
  }
});

Native SIP integration (the bridge is gone)

Here’s the biggest 2026 change, and the one most tutorials still get wrong. Before GA, connecting a phone number meant standing up a media bridge (Twilio Media Streams into FreeSWITCH into a WebSocket) and babysitting codec transcoding. Since August 2025 the Realtime API speaks SIP natively. You point a SIP trunk at OpenAI and it terminates the call leg for you.

You still need a SIP trunk provider (Twilio, Telnyx, or similar) for the phone number and PSTN termination. What you no longer build is the media plumbing in between. The flow: buy a number, point the trunk at sip:$PROJECT_ID@sip.api.openai.com;transport=tls, and register a realtime.call.incoming webhook. (Migrating trunks? Our Twilio-to-Telnyx guide covers the gotchas.)

// webhook fires on every inbound call
app.post('/webhook', express.json(), async (req, res) => {
  const event = openai.webhooks.unwrap(req.body, req.headers);   // verify signature
  if (event.type === 'realtime.call.incoming') {
    await fetch(
      `https://api.openai.com/v1/realtime/calls/${event.data.call_id}/accept`, {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
                 'Content-Type': 'application/json' },
      body: JSON.stringify({
        type: 'realtime', model: 'gpt-realtime',
        instructions: 'You are Alex, a friendly concierge for Acme.'
      })
    });
  }
  res.sendStatus(200);
});

After accepting, you monitor and steer the call over a WebSocket at wss://api.openai.com/v1/realtime?call_id=..., and you get first-class verbs for the messy parts of telephony: reject (busy country codes), refer (warm transfer to a human), and hangup. Full reference: OpenAI SIP docs.

The latency budget, millisecond by millisecond

“Feels human” and “is the line dead?” are separated by about 400 ms. Glass-to-glass, a first turn typically lands around 500–1,200 ms and steady turns around 300–600 ms. Knowing where the time goes tells you what’s worth optimizing and what isn’t.

Figure 4. Where the milliseconds go. The model’s think-and-first-audio time dominates the first turn; network and buffering dominate steady turns.

The model’s time-to-first-audio dominates the first turn, and you can’t code your way around it. But you can hide it with a fast acknowledgement (“let me check that”) while the real response generates. Everything else is yours to tune: keep the peer connection warm, put your token endpoint near your users, and don’t over-buffer playback. Chasing 50 ms in your own stack while ignoring a 700 ms cold model start is effort in the wrong place.

Pricing in 2026 and the real cost per minute

The sticker price is simple; the bill is not. Audio bills per million tokens, and the number that surprises teams is context replay: every turn resends the prior conversation as input tokens, so a long call costs more than a naive per-minute estimate suggests. Prompt caching is what saves you.

Source: OpenAI pricing, 2026. GA cut gpt-realtime 20% below the old preview. Now the arithmetic, shown, because a per-minute number you can’t reconstruct is a number you can’t trust:

Figure 5. One conversational minute is roughly 600 tokens heard and 1,200 tokens spoken. Multiply by the rate; add cached context replay; you land near $0.10/min on gpt-realtime.

A minute of audio input is about 600 tokens; a minute of generated speech about 1,200. So on gpt-realtime a minute heard is 600 × $32/1M ≈ $0.019, and a minute spoken is 1,200 × $64/1M ≈ $0.077. A real two-way minute mixes both, plus a small tax for replayed context (billed at the $0.40 cached rate, not $32), landing around $0.05–0.15/min. On gpt-realtime-mini the same minute is roughly $0.02–0.05.

For a call center, that’s the whole business case: a five-minute call near $0.50 on gpt-realtime, or $0.15 on mini, against a loaded human-agent cost of several dollars. Cap long-session cost with the GA token-limit and turn-truncation controls, and route the easy calls to mini. We break the numbers down further in our dedicated OpenAI Realtime API pricing guide.

Function calling and MCP: where voice gets useful

A voice agent that can only talk is a novelty. One that can check an order, book a slot, or open a ticket is a product. The model emits a function-call event over the data channel, your code runs the function, and you send the result back — the model weaves it into speech without dropping the conversation.

Two 2026 upgrades matter here. Async function calling is native: a slow lookup no longer freezes the conversation, because the model keeps talking while it waits. And remote MCP support means you can point a session at an MCP server (your own, or a hosted one like Stripe’s) and its tools become available without you hand-wiring each call. That turns “wire up six integrations” into “pass one URL.”

Mini case: a voice copilot we shipped

A mid-market SaaS team came to us wanting a voice copilot inside their analytics dashboard — press a button, ask “why did churn spike in March,” get a spoken answer that reads from the live account. The demo they’d built worked; it also leaked the API key into the browser bundle and locked up whenever a user interrupted the assistant mid-answer.

Over about eight weeks we rebuilt it on the ephemeral-token WebRTC flow (key stays server-side), added proper barge-in so interruptions cancel the in-flight response cleanly, and wired the dashboard’s existing REST endpoints as function calls. First-token latency came down from a sluggish ~1.4 s to ~600 ms once we kept the peer connection warm and moved the token endpoint next to the users. Per-call cost landed near $0.09/min on gpt-realtime, and the easy “what’s my MRR” questions were routed to mini at a third of that. Want a similar assessment of your stack? Book a 30-minute call and bring your architecture.

The lesson wasn’t the model — it was the plumbing around it. The model was the easy part; ephemeral tokens, interruption handling, and cost routing were where the real engineering lived.

A decision framework in five questions

Skip the transport debate and answer these in order. The last one is usually the one people forget until it’s expensive.

1. Where does the audio start? Browser or mobile app → WebRTC. A phone number → SIP. A server or existing IVR → WebSocket. This single answer picks your transport before anything else.

2. How chatty is a session? Long, multi-turn calls make context replay the dominant cost. If sessions run long, budget for caching and turn truncation, and lean toward gpt-realtime-mini for the routine turns.

3. What’s your latency ceiling? If users tolerate ~1 second, you have room. If it’s a fast back-and-forth, keep connections warm and co-locate your token endpoint — and plan a filler acknowledgement to mask first-turn model time.

4. Do you touch regulated data? Health, finance, or minors change the build: BAA, Zero Data Retention, consent capture, and audit logging aren’t features you bolt on later without pain.

5. Who owns failure at 2 a.m.? Voice touches carriers, browsers, and a model API — three moving parts. Decide up front whether your team runs that, or whether a partner who’s done it before carries the pager.

Five pitfalls that sink realtime voice projects

1. Shipping the real API key to the browser. The demo does it because it’s one fetch. In production it’s a credential leak that bills to you. Mint an ephemeral key server-side via /v1/realtime/client_secrets and hand the browser only that.

2. Ignoring barge-in. Users interrupt. If you don’t cancel the in-flight response when they start talking, the agent talks over them and the turn state deadlocks. Handle interruption from day one, not after the first user test.

3. Estimating cost without context replay. The naive “$0.077 a minute” forgets that each turn resends history as input tokens. Model it with caching on, or a chatty 15-minute call will blow your estimate.

4. Copying a 2024 tutorial. The endpoints moved at GA. Anything POSTing to /v1/realtime/sessions or exchanging SDP at /v1/realtime?model= is pre-GA and won’t connect. Use /client_secrets and /calls.

5. Treating SIP like WebRTC. Phone audio arrives as G.711 through a carrier with its own echo and packet-loss behavior. Test with real PSTN calls across a couple of carriers before you trust the happy path — the demo over wifi lies to you.

KPIs to track from day one

Quality KPIs. Time-to-first-audio (target under ~800 ms steady), interruption-recovery success rate, and word-error rate on the things that must be exact — names, numbers, addresses. Instrument these before launch, not after complaints.

Business KPIs. Containment rate (share of calls resolved without a human), cost per resolved call, and escalation reasons. If containment is high but satisfaction drops, you’re deflecting calls you should be routing.

Reliability KPIs. Session setup success, mid-call drop rate, and function-call error rate. A voice agent that connects 98% of the time feels broken; phone users expect four nines, and they notice the gap immediately.

Security and compliance in plain English

Keep the real key server-side, always — ephemeral tokens exist so the browser never sees it. Set an OpenAI-Safety-Identifier (a hashed user ID) so abuse traces to an account without exposing personal data. For regulated work, request a BAA and Zero Data Retention from OpenAI before you touch protected health information, and note that the Realtime API supports EU data residency if that’s your requirement.

Two-party consent and AI-disclosure rules apply the moment you record or dial out. Tell callers they’re speaking with AI — it’s both OpenAI policy and, in many places, law. For healthcare specifics we keep a healthcare software guide; the short version is that compliance is a design constraint, not a launch-day checkbox.

What changed since the beta API

If you built against the October 2024 beta, budget half a day to migrate. The concepts are identical; the surface moved. Here’s the diff that actually breaks code.

Nothing here is conceptually hard, but every one of these will fail silently or loudly if you copy an old snippet. When you read a tutorial, check the endpoints against the table above before you trust it — most of the WebRTC guides still ranking in search were written for the beta.

When NOT to use OpenAI Realtime

Honesty sells better than hype, so here’s where we’d talk you out of it. If your feature is async — transcribing recordings, summarizing calls after the fact, generating a voiceover — you don’t need a realtime session. A speech-to-text plus chat-model plus text-to-speech pipeline is cheaper, simpler to debug, and easier to cache.

If you need a specific cloned or custom voice, note that Realtime uses a preset voice list — deliberately, to prevent impersonation. If you’re fully on-prem with no outbound calls allowed, a hosted model API is a non-starter and you’re looking at self-hosted speech models instead. And if latency genuinely doesn’t matter to your users, you’re paying a premium for a property you won’t use.

FAQ

Does the OpenAI Realtime API support WebRTC?

Yes. WebRTC is the recommended transport for browser and mobile clients. Your server mints an ephemeral key from /v1/realtime/client_secrets, and the browser exchanges SDP with https://api.openai.com/v1/realtime/calls using that key.

What does the OpenAI Realtime API cost per minute?

Roughly $0.05–0.15 per minute on gpt-realtime and $0.02–0.05 on gpt-realtime-mini for a two-way conversation, once you account for context replay and prompt caching. The raw rate is $32 per 1M audio input tokens and $64 per 1M output tokens for gpt-realtime (2026).

Can OpenAI Realtime connect to a phone number over SIP?

Yes, natively since the August 2025 GA. You buy a number from a SIP trunk provider (Twilio, Telnyx) and point the trunk at sip:$PROJECT_ID@sip.api.openai.com;transport=tls. OpenAI terminates the call leg — no self-built media bridge required.

Which model should you choose, gpt-realtime or gpt-realtime-mini?

Use gpt-realtime for complex reasoning, precise tool calls, and high-stakes calls. Route routine, high-volume turns to gpt-realtime-mini, which costs about a third as much. Many production systems run both and switch per turn.

Is the Realtime API generally available or still beta?

Generally available since 28 August 2025. The stable model alias is gpt-realtime; the current 2026 snapshot is gpt-realtime-2.1. The 2024 public beta used different endpoints and the gpt-4o-realtime-preview model.

How do you keep your OpenAI API key out of the browser?

Never send it. Mint a short-lived ephemeral key on your server via /v1/realtime/client_secrets and return only that to the client. The browser authenticates its session with the ephemeral key, which expires quickly.

Can the voice agent call my own functions or APIs?

Yes. The model emits function-call events over the data channel; you run the function and return the result. In 2026 you can also attach a remote MCP server so its tools are available to the session without wiring each one by hand.

Is OpenAI Realtime HIPAA-compliant for telehealth?

It can be used in HIPAA workflows if you sign a BAA with OpenAI and enable Zero Data Retention before processing protected health information. Compliance also depends on your consent capture, audit logging, and access controls — treat it as a design constraint from day one.

What to Read Next

Ready to ship your first realtime voice feature?

Pick the transport by where the audio starts, use the 2026 endpoints (not the beta ones), model cost with context replay in mind, and handle barge-in and compliance from the first commit. Do those four things and the model itself becomes the easy part.

The teams that win with voice AI aren’t the ones with the cleverest prompt — they’re the ones who got the plumbing right: ephemeral tokens, warm connections, honest cost routing, and a real answer for what happens when a carrier hiccups. That’s the boring work, and it’s exactly the work we’ve been doing since 2005.