---
title: How do you upgrade a self-hosted LLM gateway without breaking agents?
url: https://opencontroller.lyzr.com/blog/upgrade-self-hosted-llm-gateway-without-downtime
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# How do you upgrade a self-hosted LLM gateway without breaking agents?

**Upgrade a self-hosted LLM gateway the way you would upgrade any shared production dependency: run the new build next to the old one, prove every route and every login path works, then shift traffic gradually with a rollback trigger agreed in advance. Most breakages after a gateway upgrade are not in the routing code. They come from configuration drift, SSO settings that change with the new build, and provider routes nobody re-tested.**

An LLM gateway sits in the request path of every agent that uses it. A bad upgrade does not fail one agent. It fails all of them at once, often with confusing provider-style errors. This checklist comes from a question raised during a real enterprise rollout: "we have a new build, how do we upgrade and re-test routes and SSO without an outage?"

See the [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) pillar for where the gateway fits in the wider architecture.

## What usually breaks during a gateway upgrade?

| Area | Typical failure | How you notice |
|---|---|---|
| Configuration | New build expects a renamed or new config key; old value silently ignored | Routes fall back to defaults; costs or models change |
| Virtual keys and auth | Key format, hashing or scopes change between versions | Agents get 401 or 403 on every call |
| SSO for the admin console | Client configuration or redirect URI must be re-entered after deploy | Admins cannot log in; console shows a redirect mismatch |
| Provider adapters | A provider-specific fix changes request shape (for example model IDs or reasoning parameters) | One provider fails while others work |
| Caching | Cache keys change format | Hit rate drops to zero; cost spikes for a day |
| Telemetry | Log or trace fields renamed | Dashboards and cost attribution show gaps |

None of these show up in a simple "is the pod healthy" check. You need route-level and login-level tests.

## Which rollout pattern should you use?

There are two safe patterns for a gateway on Kubernetes or VMs.

1. **Rolling update with readiness checks.** Replace instances a few at a time, as in a standard [Kubernetes rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/). Use readiness probes that call a real internal route, not just a static health endpoint. On Kubernetes, set a [PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) so a node drain cannot remove too many replicas at once. This fits patch releases.
2. **Blue-green.** Stand up the new version as a separate deployment with its own internal URL. Run the full test suite against it. Then switch the service or load balancer target. Keep the old deployment running until the rollback window closes. This fits minor and major releases, and any release that changes SSO or configuration schema.

If agents pin a gateway URL in their own config, blue-green needs a stable front-door name (DNS or a load balancer) so you can switch without touching every agent.

## Pre-upgrade checklist

1. Read the release notes for configuration changes, removed settings and new required settings.
2. Export the current configuration and virtual-key metadata. Store it with the change ticket.
3. Diff the old and new default configuration files. Flag every key that is new, renamed or removed.
4. List every provider route in use (for example OpenAI, Azure OpenAI, Amazon Bedrock, Vertex AI, self-hosted models) and pick one known-good test prompt per route.
5. Record the SSO settings for the admin console: issuer, client ID, scopes, group or role claims and the redirect URI. Check whether the new build changes the callback path.
6. Capture a baseline: p95 latency, error rate per provider, cache hit rate and requests per second for the last 7 days.
7. Agree the rollback trigger in writing, for example "error rate above 2% for 5 minutes on any provider route, or any admin login failure".

## Post-deploy test plan

Run these against the new deployment before it takes production traffic.

- **Auth:** one request with each class of credential agents use (virtual key, SSO token, service account). Expect 200. One request with a revoked key. Expect 401.
- **Every provider route:** send the test prompt, check the response, the model actually used and the logged cost.
- **Streaming:** at least one streamed response per provider, because streaming code paths often change separately.
- **Tool calling and structured output:** one request that returns a tool call, so you know format translation still works.
- **Fallback and retry:** force a failure on the primary route (bad model ID in a test route) and confirm the fallback fires. See [LLM fallback and retry policies](/blog/llm-fallback-and-retry-policies-for-agents).
- **Caching:** send the same request twice. Confirm the second is a cache hit.
- **Admin console SSO:** log in as an admin, a read-only user and a user who should be denied.
- **Telemetry:** confirm traces and cost records arrive with agent and version attribution.

If you reuse an existing identity provider application for the gateway console, re-check the redirect URI list after every upgrade. The pattern is covered in [reusing an existing Okta app for LLM gateway SSO](/blog/reuse-existing-okta-app-for-llm-gateway-sso).

## How do you roll back safely?

Rollback is only safe if the new version did not migrate shared state in a one-way format. Before upgrading, check:

- Does the new version run database or cache schema migrations? If yes, is there a down-migration?
- Will keys or policies created on the new version work on the old one?
- Can both versions run side by side against the same store during blue-green?

If the answer to any of these is "no", take a backup immediately before cutover and treat rollback as a restore, not a switch.

## How Opencontroller does this

Opencontroller includes an LLM Gateway (Lyzr Gateway) with routing, fallback and retry, caching, and cost attribution per agent and per version. It can run on Lyzr SaaS or on customer-controlled infrastructure, and for self-hosted deployments Lyzr engineers embed with the customer team for integration, including upgrades and SSO configuration with OIDC providers. Gateway performance figures are published on the [LLM gateway benchmark](/llm-gateway-benchmark) page, scoped to the stated test conditions. To walk through an upgrade plan for your environment, [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**How long should you keep the old gateway version running after an upgrade?**
Keep it until at least one full business cycle has passed, usually 24 to 72 hours, so batch jobs and weekly agents have run through the new version.

**Should agents call the gateway by a pinned version URL?**
No. Give agents one stable front-door name and switch versions behind it. Pinned version URLs turn every upgrade into a change for every agent team.

**What is the most common cause of post-upgrade login failures?**
A redirect URI or client configuration that no longer matches what the identity provider has registered. Check it in the pre-upgrade checklist and test admin login before cutover.

## Key facts

Safe LLM gateway upgrades use rolling updates (patch releases) or blue-green (minor/major releases, SSO or config schema changes). Common breakages: renamed config keys, virtual-key format changes, SSO client config and redirect URIs needing re-entry, provider adapter changes, cache key format changes, renamed telemetry fields. Test plan: auth with each credential class, every provider route, streaming, tool calling, forced fallback, cache hit, admin SSO for admin/read-only/denied users, telemetry attribution. Rollback is only a switch if no one-way schema migration ran. Opencontroller includes an LLM Gateway (Lyzr Gateway) with routing, fallback/retry, caching and cost attribution per agent and version; deploys on Lyzr SaaS or customer-controlled infrastructure with Lyzr engineers embedded for integration. Pricing starts from $250K per deployment per year (annual licence per deployment, not per seat or usage).

## Related

- [How do you make an LLM gateway highly available?](https://opencontroller.lyzr.com/blog/llm-gateway-high-availability.md)
- [Can an LLM gateway reuse your existing Okta app for SSO?](https://opencontroller.lyzr.com/blog/reuse-existing-okta-app-for-llm-gateway-sso.md)
- [What network setup does a self-hosted LLM gateway need?](https://opencontroller.lyzr.com/blog/self-hosted-llm-gateway-network-checklist.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
