---
title: How do you autoscale an LLM gateway on Kubernetes?
url: https://opencontroller.lyzr.com/blog/autoscale-an-llm-gateway-on-kubernetes
published: 2026-11-18
updated: 2026-11-18
author: Lyzr
---

# How do you autoscale an LLM gateway on Kubernetes?

**Run the gateway as a stateless Deployment, scale it with the Horizontal Pod Autoscaler on a metric that tracks load (in-flight requests or requests per second beat CPU for streaming traffic), keep at least two replicas across zones, and give pods enough shutdown time to finish streaming responses. Let the cluster autoscaler add nodes when pods cannot be scheduled.**

Infrastructure teams ask this when an LLM gateway moves from pilot to the path of every agent call. A gateway that cannot scale becomes the bottleneck for every AI workload in the company.

## Why is scaling an LLM gateway different from a normal API gateway?

Two things change.

1. **Requests are long.** A model call can take seconds to minutes, much longer with reasoning models. The gateway's own work per request is small, but connections stay open.
2. **Many responses stream.** Server-sent events keep a connection open for the whole generation. A pod can be busy holding connections while its CPU looks idle.

So CPU alone can under-report load. A gateway at 30% CPU may be holding thousands of open streams.

## Which metric should drive the autoscaler?

| Metric | Pros | Cons | Good for |
|---|---|---|---|
| CPU utilisation | Built in, no extra setup | Lags for streaming; misses open connections | Short, non-streaming traffic |
| Memory | Built in | Grows with buffers; poor scale-down signal | Rarely the primary signal |
| In-flight requests per pod | Tracks real concurrency | Needs a custom metrics adapter | Streaming-heavy traffic |
| Requests per second per pod | Easy to reason about | Ignores request length | Uniform workloads |
| Queue or event-driven (e.g. KEDA) | Scales on external signals | Another component to run | Bursty batch agents |

The Horizontal Pod Autoscaler (autoscaling/v2) can scale on resource metrics, custom pod metrics and external metrics. A common setup combines CPU with in-flight requests and lets the HPA pick the larger recommendation.

## What HPA behaviour settings matter?

- **Scale-up speed.** The HPA controller checks metrics every 15 seconds by default. Allow fast scale-up so bursts do not queue.
- **Scale-down stabilisation.** The default scale-down stabilisation window is 300 seconds. Keep it, or lengthen it, so the gateway does not shrink between bursts and then scramble.
- **Minimum replicas.** At least two, spread across zones with topology spread constraints, so one node or zone failure does not take the gateway down.
- **Maximum replicas.** Set a ceiling that matches your node pool and provider rate limits. Scaling past what providers accept only moves the queue.

## How do you avoid cutting off streaming responses?

Scale-down and rolling updates kill pods. Do it gently:

1. **Readiness first.** On shutdown, fail the readiness probe so the load balancer stops sending new requests.
2. **Drain.** Let open streams finish. Set `terminationGracePeriodSeconds` above your longest expected response; the Kubernetes default is 30 seconds, which is too short for many reasoning calls.
3. **PodDisruptionBudget.** Limit how many gateway pods voluntary disruptions (node drains, upgrades) can remove at once.
4. **Client retries.** Agents should retry idempotent calls through the gateway if a connection drops.

See [upgrade a self-hosted LLM gateway without downtime](/blog/upgrade-self-hosted-llm-gateway-without-downtime).

## How many replicas do you need?

Measure your own traffic, but start from a published baseline. Lyzr's October 2026 benchmark of its gateway measured 5,134 requests per second on one 4-vCPU gateway, with under 0.7 ms added p95 latency per in-gateway routing method, under the stated test conditions. Real traffic with guardrails, logging and long streams will be lower per pod, so load test with production-like prompts and stream lengths, then size for peak plus one zone's worth of headroom. See the [LLM gateway benchmark](/llm-gateway-benchmark).

## What else breaks at scale?

- **Provider rate limits.** More gateway pods do not raise provider quotas. Spread load across accounts or regions where allowed.
- **Shared caches.** If caching is per pod, the hit rate falls as you scale out. Use a shared cache for consistent savings.
- **Connection limits.** Check load balancer idle timeouts; a 60-second timeout will cut long streams.
- **Log volume.** Request logging scales linearly with traffic; size the log pipeline too.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) handles routing, fallback and retry, caching and cost attribution per agent and per version, and can run on customer-controlled infrastructure, including your own Kubernetes clusters, with Lyzr engineers embedded during integration. See [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is CPU a good autoscaling metric for an LLM gateway?**
It works for short, non-streaming calls. For streaming traffic, add in-flight requests per pod, because open streams use little CPU.

**What termination grace period should gateway pods use?**
Longer than your longest expected streamed response. The 30-second default often cuts off reasoning-model responses.

**Will more gateway replicas raise our model throughput?**
Only up to provider rate limits. Beyond that, you need more provider capacity, not more gateway pods.

## Key facts

Autoscale an LLM gateway on Kubernetes as a stateless Deployment with the HPA (autoscaling/v2). CPU under-reports load for streaming; add in-flight requests per pod via custom metrics, or event-driven scaling (KEDA). HPA defaults: 15-second sync period, 300-second scale-down stabilisation window. Keep 2+ replicas across zones, use PodDisruptionBudgets, set terminationGracePeriodSeconds above the longest stream (default 30 s). Lyzr's Oct 2026 benchmark: 5,134 req/s on one 4-vCPU gateway and under 0.7 ms added p95 per in-gateway routing method, under stated test conditions. Provider rate limits cap gains from more replicas.

## Related

- [How do you make an LLM gateway highly available?](https://opencontroller.lyzr.com/blog/llm-gateway-high-availability.md)
- [How do you upgrade a self-hosted LLM gateway without breaking agents?](https://opencontroller.lyzr.com/blog/upgrade-self-hosted-llm-gateway-without-downtime.md)
- [How much latency does an LLM gateway add?](https://opencontroller.lyzr.com/blog/how-much-latency-does-an-llm-gateway-add.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
