---
title: How much latency does an LLM gateway add?
url: https://opencontroller.lyzr.com/blog/how-much-latency-does-an-llm-gateway-add
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# How much latency does an LLM gateway add?

**A well-built LLM gateway usually adds single-digit milliseconds or less, which is small next to model response times measured in hundreds of milliseconds to seconds. The overhead grows with the features you switch on inline, such as PII redaction, model-based guardrails or synchronous logging. Measure added p95 latency with the same prompts sent direct and through the gateway, not end-to-end response time.**

The question comes up in almost every gateway evaluation, usually from the team that owns a customer-facing agent. The honest answer is "it depends on what the gateway does in the path", so it pays to know where the milliseconds go.

## Where does gateway latency come from?

| Source | Typical impact | How to reduce it |
|---|---|---|
| Extra network hop | Depends on distance between app, gateway and provider | Deploy the gateway in the same region as the apps |
| TLS termination and re-encryption | Small per request; larger on new connections | Keep-alive and connection pooling to providers |
| Authentication | Small if tokens are cached; larger if each call hits the IdP | Cache validated tokens until expiry |
| Routing decision | Small for rule-based routing | Keep routing rules in memory |
| Cache lookup | Small; a hit removes the model call entirely | Tune cache scope per agent |
| Policy checks (rules, regex) | Small | Compile rules once, not per request |
| PII redaction | Grows with prompt size | Apply only to agents that handle personal data |
| Model-based guardrails | A full extra model call | Use for high-risk agents or steps only |
| Logging | Near zero if asynchronous; noticeable if synchronous | Ship logs and traces asynchronously |

The pattern: plumbing is cheap, inspection costs more. Any check that calls another model roughly doubles the model portion of the path for that request.

## How should you measure gateway overhead?

A fair test isolates what the gateway adds:

1. **Use the same prompts and model** for the direct path and the gateway path.
2. **Run both paths at the same time**, so provider slowdowns affect both equally.
3. **Measure added latency per request**, ideally from gateway-side timestamps: time from request received to request sent upstream, plus time from response received to response returned.
4. **Report p50, p95 and p99.** Averages hide the tail that users feel.
5. **Test at your real concurrency**, not one request at a time. Overhead often rises only under load.
6. **Turn features on one at a time.** Baseline routing first, then caching, then redaction, then guardrails.
7. **For streaming, measure time to first token** separately from total time.

OpenTelemetry's semantic conventions for generative AI give you consistent span and metric names, so gateway and application traces line up in the same tool. See [monitoring LLM provider latency and errors](/blog/monitor-llm-provider-latency-and-errors) for the provider side.

## What do published benchmark numbers look like?

Lyzr published an [LLM gateway benchmark](/llm-gateway-benchmark) in October 2026. Under its stated test conditions, Lyzr Gateway added under 0.7 ms at p95 per in-gateway routing method, handled 5,134 requests per second on one 4-vCPU gateway, and reduced model cost by 23.5% with caching on mixed synthetic traffic. Those figures apply to the tested setup only. Your numbers will depend on region, payload size, features enabled and provider behaviour, which is why the measurement plan above matters more than any vendor figure, including ours.

## Can a gateway make agents faster?

Yes, in three ways:

- **Caching.** A cache hit returns without a model call. Caching suits repeated questions and fixed system prompts; see [caching savings and risks](/blog/llm-caching-savings-and-risks).
- **Fallback and retry.** When a provider slows down or errors, routing to a fallback model cuts the tail. See [fallback and retry policies](/blog/llm-fallback-and-retry-policies-for-agents).
- **Routing simple tasks to smaller models**, which respond faster.

For many agents these gains outweigh the gateway's own overhead. Agents that make several model calls per task feel the difference most.

## How Opencontroller does this

Lyzr Gateway, part of Opencontroller, provides routing, fallback and retry, caching and cost attribution per agent and per version, behind OpenAI-compatible endpoints. Opencontroller keeps control in the path, so it can allow, route, redact or block, and you choose which checks run inline for each agent. It deploys on Lyzr SaaS or customer-controlled infrastructure. Read the [gateway vs control plane pillar](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), compare at [/compare/llmgateway](/compare/llmgateway), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is gateway latency noticeable to users?**
Rarely, for routing and policy rules alone. It becomes noticeable when model-based checks run inline on every request, so scope those to high-risk agents.

**Should the gateway run in the same region as the model provider?**
Place it close to the calling apps and on a short path to the provider. A gateway in a distant region adds a network round trip on every call.

**Does streaming hide gateway latency?**
Partly. Users see the first token sooner, but the gateway still adds overhead before the first token. Measure time to first token separately.

## Key facts

LLM gateway overhead is usually small next to model response times; it grows with inline features (PII redaction, model-based guardrails, synchronous logging). Measure added p50/p95/p99 latency with same prompts on direct and gateway paths at real concurrency. Lyzr benchmark (October 2026, scoped to stated test conditions): under 0.7 ms added p95 per in-gateway routing method; 5,134 requests per second on one 4-vCPU gateway; 23.5% lower model cost with caching on mixed synthetic traffic. Lyzr Gateway in Opencontroller provides routing, fallback/retry, caching and cost attribution per agent and version. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [How do you monitor LLM provider latency and errors across agents?](https://opencontroller.lyzr.com/blog/monitor-llm-provider-latency-and-errors.md)
- [How much can LLM caching save, and when is it unsafe?](https://opencontroller.lyzr.com/blog/llm-caching-savings-and-risks.md)
- [How should an LLM gateway handle fallback and retries for AI agents?](https://opencontroller.lyzr.com/blog/llm-fallback-and-retry-policies-for-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
