---
title: Can an LLM gateway sit behind your existing API gateway?
url: https://opencontroller.lyzr.com/blog/llm-gateway-behind-existing-api-gateway
published: 2026-11-11
updated: 2026-11-11
author: Lyzr
---

# Can an LLM gateway sit behind your existing API gateway?

**Yes. The common pattern is to keep your enterprise API gateway at the edge for authentication, network policy and coarse rate limits, and put the LLM gateway behind it for model-specific work: routing between providers, fallback, caching, token accounting, cost attribution and content policies such as redaction. The two layers do different jobs, so layering them is usually cleaner than forcing one to do both.**

Security teams often plan to route model traffic through the API gateway they already run. That is sensible: it keeps one front door and reuses existing controls. The questions are which layer does what, how to keep streaming responses working, and how much latency the extra hop costs.

## What does each layer do?

Read the stack from the agent outwards:

1. **Agent or application** calls an OpenAI-compatible endpoint with its own identity.
2. **Enterprise API gateway** terminates TLS, authenticates the caller (OAuth, mTLS or API key), applies network and IP policy, enforces coarse per-client rate limits, and logs the request at the HTTP level.
3. **LLM gateway** identifies the agent and version, applies content policy (allow, redact, block), chooses a model and provider, retries or fails over, serves cache hits, counts input and output tokens, and attributes cost.
4. **Model providers** — hosted APIs, cloud model services or self-hosted models.

| Concern | Enterprise API gateway | LLM gateway |
|---|---|---|
| TLS termination, WAF, IP allowlists | ✔ | — |
| Caller authentication | ✔ | Validates identity passed through |
| Request-count rate limits | ✔ | — |
| Token-based budgets and limits | — | ✔ |
| Model routing and provider fallback | — | ✔ |
| Semantic or exact-match caching | — | ✔ |
| Prompt and response policies (PII, secrets) | — | ✔ |
| Cost attribution per agent and version | — | ✔ |
| HTTP access logs | ✔ | — |
| Model-call traces (model, tokens, latency) | — | ✔ |

The key point: an API gateway counts requests, but LLM cost and risk scale with tokens and content. That is why model-aware controls belong in the second layer. For a wider comparison of the layers, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## How do you pass identity through both layers?

Do not let the API gateway swap the agent's identity for its own. If the LLM gateway only ever sees "API gateway", it cannot attribute cost or apply per-agent policy. Options:

- **Forward a validated token.** The API gateway validates an OIDC/OAuth token and forwards it, or forwards verified claims in signed headers.
- **Inject trusted headers** (agent ID, client ID) and make the LLM gateway accept them only from the API gateway's network address or mTLS identity.
- **Pass user context separately** when an agent acts on behalf of a user, so both the agent and the user are recorded. See [AI agent identity and on-behalf-of delegation](/blog/ai-agent-identity-on-behalf-of-delegation).

## What breaks when you put model traffic through an API gateway?

Four things, in order of how often they bite:

1. **Streaming.** Chat completions often stream tokens as server-sent events ([OpenAI streaming reference](https://platform.openai.com/docs/api-reference/streaming)). The API gateway must not buffer the full response, or users see nothing until generation ends. Turn off response buffering and compression for these routes.
2. **Timeouts.** Long generations and agent tool loops can exceed default integration timeouts. For example, [Amazon API Gateway quotas](https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html) set a default integration timeout of 29 seconds for REST APIs. Check yours and set it per route.
3. **Body size limits.** Prompts with retrieved documents or images can exceed default request size limits.
4. **Double retries.** If both layers retry, one failed call can become many. Let the LLM gateway own retries and fallback; set the API gateway to not retry model routes.

## What is the latency budget?

Each hop adds network time plus processing. A practical budget:

- **API gateway:** typically a few milliseconds for auth and policy, more if it calls an external authoriser on every request. Cache authorisation decisions where you can.
- **LLM gateway:** keep routing and policy overhead to single-digit milliseconds. Opencontroller's October 2026 benchmark measured under 0.7 ms added p95 per in-gateway routing method, and 5,134 requests per second on one 4-vCPU gateway, under the stated test conditions ([benchmark details](/llm-gateway-benchmark)).
- **Model:** hundreds of milliseconds to many seconds. Gateway overhead is small next to this, but measure time-to-first-token, not just total latency, because buffering problems show up there first.

Co-locate the two gateways in the same region and network to avoid adding a cross-region round trip.

## Checklist for layering the two gateways

1. Expose OpenAI-compatible routes on the API gateway and forward them unchanged.
2. Forward or assert agent identity; reject identity headers from any other source.
3. Disable buffering on streaming routes; set timeouts per route.
4. Set request-count limits at the API gateway and token budgets at the LLM gateway.
5. Disable retries on the API gateway for model routes.
6. Block direct egress from agents to model providers, so the gateway path cannot be bypassed.
7. Correlate logs with a shared request or trace ID across both layers.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) provides routing, fallback and retry, caching and cost attribution per agent and per version, behind OpenAI-compatible endpoints, so it can sit behind an existing API gateway without changing agent code. It follows Opencontroller's principle of control without migration, and uses open standards such as OIDC and OpenTelemetry for identity and traces. See [how to attribute LLM costs to AI agents](/blog/how-to-attribute-llm-costs-to-ai-agents), compare options at [/compare/llmgateway](/compare/llmgateway), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can an LLM gateway sit behind an API gateway?**
Yes. Keep the API gateway for TLS, authentication and coarse rate limits, and use the LLM gateway behind it for model routing, token budgets, caching, content policy and cost attribution.

**Why not do everything in the API gateway?**
API gateways count requests. LLM cost and risk depend on tokens, models and content, which need model-aware routing, caching and policy.

**How do you keep streaming working through both gateways?**
Disable response buffering and compression on model routes, raise integration timeouts, and measure time-to-first-token.

## Key facts

An LLM gateway can sit behind an enterprise API gateway: the API gateway handles TLS, auth and request-count limits; the LLM gateway handles model routing, fallback, caching, token budgets, content policy and cost attribution. Opencontroller's LLM Gateway (Lyzr Gateway) exposes OpenAI-compatible endpoints. Oct 2026 benchmark (stated test conditions): under 0.7 ms added p95 per in-gateway routing method; 5,134 req/s on one 4-vCPU gateway. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [LLM gateway vs agent gateway vs AI control plane: what's the difference?](https://opencontroller.lyzr.com/blog/llm-gateway-vs-agent-gateway-vs-control-plane.md)
- [How do you attribute LLM costs to individual AI agents?](https://opencontroller.lyzr.com/blog/how-to-attribute-llm-costs-to-ai-agents.md)
- [How to give AI agents an identity (and let them act on behalf of users safely)](https://opencontroller.lyzr.com/blog/ai-agent-identity-on-behalf-of-delegation.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
