---
title: Can an LLM gateway govern realtime voice APIs?
url: https://opencontroller.lyzr.com/blog/can-an-llm-gateway-handle-realtime-voice-apis
published: 2026-11-18
updated: 2026-11-18
author: Lyzr
---

# Can an LLM gateway govern realtime voice APIs?

**Partly. Realtime voice APIs run over WebRTC, WebSocket or SIP, so audio often flows straight from the caller's device or phone network to the model provider, bypassing an HTTP gateway. The gateway can still control the session at three points: minting short-lived client keys, attaching a server-side "sideband" connection, and proxying server-to-server WebSockets.** Design voice agents around those points instead of assuming every byte passes through the gateway.

## Why are voice agents different from HTTP traffic?

A text agent makes discrete HTTP requests. A gateway sees each prompt, response and token count, and can allow, route, redact or block per request.

A realtime voice session is a long-lived, bidirectional stream. OpenAI's Realtime API, for example, supports three transports:

- **WebRTC** for browsers and mobile apps, where media goes peer-to-peer between the client and OpenAI.
- **WebSocket** for server-to-server connections, where your backend streams audio and events.
- **SIP** for telephony, where an inbound phone call reaches OpenAI and triggers a webhook to your server.

In the WebRTC and SIP cases, the audio does not pass through your infrastructure at all.

## What can a gateway see for each transport?

| Transport | Audio path | What a gateway can see | What it can enforce |
|---|---|---|---|
| WebSocket (server-to-server) | Through your backend | All events, transcripts, tool calls, usage | Full: allow, route, redact events, terminate session |
| WebRTC (browser/app) | Client direct to provider | Ephemeral key request; sideband events | Who gets a session, session config, tool calls via sideband |
| SIP (telephony) | Phone network to provider | Incoming-call webhook; sideband events | Accept or reject call, session config, tool calls via sideband |

## What are the control points?

### 1. Ephemeral client keys

Browser clients should never hold a standard API key. Instead, your server calls the provider to create a short-lived client secret, and the browser uses it to open the WebRTC session. OpenAI exposes this as a client secrets endpoint for Realtime; Google documents ephemeral tokens for its Gemini Live API too.

The mint request is plain HTTP, so it can pass through a gateway. That is where you decide:

- which authenticated user and which agent may start a session;
- which model and voice the session may use;
- which instructions and tools are fixed into the session config;
- which cost centre the session is charged to.

### 2. The sideband connection

OpenAI documents a server-side control pattern: after a WebRTC or SIP session starts, your server opens a second WebSocket to the same session using the call ID (`wss://api.openai.com/v1/realtime?call_id=...`) and your standard API key. Through it, the server can monitor events, update instructions and session settings, and receive tool calls and return results.

This keeps tool credentials and business logic off the client. It also gives you the event stream for logging, even though audio goes directly to the provider.

### 3. Server-to-server WebSocket proxy

If your architecture lets the backend carry the audio, for example in a contact-centre platform that already terminates calls, a WebSocket proxy sees everything. It costs latency and engineering effort, but it is the only option that gives per-event enforcement such as redacting a transcript event before it is stored.

## What gaps remain?

Be honest about these in a risk review:

1. **No inline audio redaction** on WebRTC or SIP. If a caller reads out a card number, the provider receives it.
2. **Usage is reported by the provider,** not metered by your gateway, for direct media. Reconcile against provider usage data.
3. **Session duration risk.** A long call can run up cost. Set maximum session lengths in the session config and enforce them from the sideband.
4. **Tool calls are only governed if they go through your server.** Do not give client-side code tool credentials.
5. **Discovery.** Voice agents are often built in contact-centre platforms outside the normal API key estate. See [finding voice AI agents in the contact centre](/blog/find-voice-ai-agents-in-the-contact-centre).

## A practical checklist for governed voice agents

1. Mint ephemeral keys server-side, through the gateway, per authenticated user.
2. Fix model, instructions and tool list in the session config at mint time.
3. Open a sideband for every session; route all tool calls through it.
4. Log session start, end, duration, tool calls and transcript events with agent ID and version.
5. Set a maximum session length and a per-agent daily budget.
6. Reconcile provider-reported usage against your logs weekly.
7. Keep a kill path: closing the sideband or session ends the call.

For how gateways authenticate apps, see [how apps should authenticate to an LLM gateway](/blog/how-should-apps-authenticate-to-an-llm-gateway). For the wider picture of where gateways stop and a control plane starts, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## How Opencontroller does this

Opencontroller's principle is that registration is not runtime takeover: how deeply it can enforce depends on the connector and traffic path. Voice agents are registered in the governed registry with their versions and tools; HTTP traffic routed through the LLM Gateway gets routing, fallback, caching and cost attribution per agent and version; and rogue-agent handling can alert, quarantine or kill agents through the connected platform. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can a gateway see the audio in a WebRTC voice session?**
No. With WebRTC, media flows directly between the client and the provider. The gateway can control the ephemeral key mint and attach a server-side sideband for events and tool calls.

**What is a sideband connection?**
A second, server-side WebSocket to an existing realtime session, opened with the call ID. It lets your server monitor events, update instructions and handle tool calls while the client keeps the audio.

**Should browsers ever hold a provider API key?**
No. Use short-lived client secrets minted by your server, so a leaked key expires quickly and session settings stay under server control.

## Key facts

Realtime voice APIs use long-lived WebRTC, WebSocket or SIP sessions. OpenAI Realtime supports all three; with WebRTC and SIP, audio flows directly to the provider and bypasses an HTTP gateway. Control points: server-minted ephemeral client secrets (an HTTP call a gateway can govern), a server-side sideband WebSocket attached with the call ID (wss://api.openai.com/v1/realtime?call_id=...) to monitor events, update instructions and handle tool calls, and server-to-server WebSocket proxying for full event-level enforcement. Gaps: no inline audio redaction on WebRTC/SIP and provider-reported usage.

## Related

- [How do you find and govern voice AI agents in the contact centre?](https://opencontroller.lyzr.com/blog/find-voice-ai-agents-in-the-contact-centre.md)
- [How should users and applications authenticate to an LLM gateway?](https://opencontroller.lyzr.com/blog/how-should-apps-authenticate-to-an-llm-gateway.md)
- [What are virtual keys in an LLM gateway?](https://opencontroller.lyzr.com/blog/what-are-llm-gateway-virtual-keys.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
