---
title: How do you make an LLM gateway highly available?
url: https://opencontroller.lyzr.com/blog/llm-gateway-high-availability
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# How do you make an LLM gateway highly available?

**Make the gateway stateless, run at least three replicas across zones behind a load balancer, and keep shared state (keys, policies, cache) in replicated stores. Then add two layers of failover: across gateway regions, and across model providers or deployments when an upstream model degrades. Size capacity from your own load tests, with headroom to lose a zone.**

Once every agent's model traffic flows through one gateway, that gateway is on the critical path for every agent. Treat it like any tier-one service.

## Why does gateway availability matter so much?

A gateway concentrates risk by design. Routing, cost attribution, redaction and policy only work if traffic goes through it. That creates two failure modes:

- **Gateway down:** every agent that depends on it stops.
- **Gateway bypassed:** teams hard-code provider endpoints "just in case", and your controls disappear.

A reliable gateway prevents both. If it is not dependable, people route around it.

## What does a highly available gateway architecture look like?

| Layer | Pattern | Why |
|---|---|---|
| Gateway nodes | Stateless replicas, minimum three, spread over availability zones | Lose one node or zone without an outage |
| Load balancing | L7 load balancer with active health checks | Remove unhealthy nodes in seconds |
| Configuration | Policies and routes in a replicated store or GitOps repo, cached locally | Nodes keep serving if the config store blips |
| Secrets | Provider keys in a managed secret store, cached with short TTL | No single secret service on the hot path |
| Cache | Shared cache with replication; treat as optional | A cache outage should slow you down, not stop you |
| Logs and metering | Asynchronous export with local buffering | Telemetry problems never block requests |
| Regions | Active-active or active-passive across two regions, with DNS or global load balancing | Survive a regional outage |

The key principle: anything not strictly needed to forward a request should degrade gracefully rather than fail the request.

## How do you fail over across model providers?

Gateway uptime is half the story. Upstream models fail too: rate limits, regional capacity issues, latency spikes. Configure these in the gateway:

1. **Retries with backoff** for transient errors such as 429s and 5xxs, with a cap so retries do not multiply load.
2. **Fallback chains** per route, for example the same model in another region, then an approved alternative model.
3. **Circuit breakers** that stop sending traffic to an upstream that keeps failing, then probe for recovery.
4. **Timeouts per route**, tuned to the use case. An interactive agent needs a shorter timeout than a batch job.
5. **Approved-model lists for fallback.** Only fail over to models that passed the same evaluation and data-residency checks. A fallback that sends data to an unapproved region is a policy violation, not resilience.

For the details, see [LLM fallback and retry policies for agents](/blog/llm-fallback-and-retry-policies-for-agents) and [monitoring LLM provider latency and errors](/blog/monitor-llm-provider-latency-and-errors).

## How do you size gateway capacity?

Use a worked method, not a guess:

1. Measure peak requests per second across all agents, including batch jobs.
2. Load test one gateway node with your real mix of routes, payload sizes, streaming and policies. Published benchmarks give a starting point only.
3. Divide peak by tested per-node throughput, then add enough nodes to survive losing your largest zone.
4. Add headroom for growth, typically reviewed quarterly.
5. Autoscale on CPU and concurrent connections, but keep a minimum floor so a sudden spike does not wait for scale-out.

As a reference point, Lyzr's October 2026 benchmark of its gateway measured 5,134 requests per second on one 4-vCPU gateway and under 0.7 ms added p95 latency per in-gateway routing method, under the stated test conditions. Your numbers will differ with your traffic, so test. See the [LLM gateway benchmark](/llm-gateway-benchmark).

## How do you test that failover works?

High availability you have not tested is a hope. Run game days:

- Kill a gateway node under load and confirm no client errors.
- Block one model provider at the network and confirm fallback routes engage.
- Take the config store offline and confirm nodes keep serving with cached policy.
- Fail the cache and confirm requests still succeed, only slower and costlier.
- Fail over a whole region and time recovery.

Record results as evidence. Operational resilience rules for financial services, such as DORA, expect tested continuity for critical services.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) provides routing, fallback and retry, caching and cost attribution per agent and per version. It can run on Lyzr SaaS or on customer-controlled infrastructure, so you can place replicas in the zones and regions your resilience plan needs. For where the LLM gateway fits against an agent gateway and control plane, read the [gateways pillar](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**How many LLM gateway replicas do you need?**
At least three, spread across availability zones, so you can lose one zone and still serve peak traffic. Add more based on load tests of your real traffic mix.

**Should the LLM gateway cache be required for requests to succeed?**
No. Treat the cache as an optimisation. If it fails, requests should go straight to the model, slower and costlier but still working.

**What is the difference between gateway failover and model fallback?**
Gateway failover moves traffic to healthy gateway nodes or regions. Model fallback reroutes a request to another approved model or deployment when the upstream provider fails.

## Key facts

Highly available LLM gateways use stateless replicas across zones, health-checked load balancing, multi-region deployment and model fallback chains. Lyzr Oct 2026 benchmark (scoped to stated test conditions): 5,134 req/s on one 4-vCPU gateway; under 0.7 ms added p95 per in-gateway routing method. Opencontroller LLM Gateway offers routing, fallback/retry, caching and cost attribution; deployable on Lyzr SaaS or customer infrastructure. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [How should an LLM gateway handle fallback and retries for AI agents?](https://opencontroller.lyzr.com/blog/llm-fallback-and-retry-policies-for-agents.md)
- [How do you monitor LLM provider latency and errors across agents?](https://opencontroller.lyzr.com/blog/monitor-llm-provider-latency-and-errors.md)
- [What network setup does a self-hosted LLM gateway need?](https://opencontroller.lyzr.com/blog/self-hosted-llm-gateway-network-checklist.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
