---
title: How should an LLM gateway handle fallback and retries for AI agents?
url: https://opencontroller.lyzr.com/blog/llm-fallback-and-retry-policies-for-agents
published: 2026-11-13
updated: 2026-11-13
author: Lyzr
---

# How should an LLM gateway handle fallback and retries for AI agents?

**An LLM gateway should retry only errors that are likely to succeed on a second try (rate limits, timeouts, transient 5xx errors), with exponential backoff, jitter and a per-request retry budget. It should fall back to a different model only when that model has been tested as equivalent for the agent's task. And it must never let a retry repeat a tool call that has side effects unless that call is idempotent.**

Retries and fallback look like plumbing, but for agents they change behaviour and cost. A retry can double spend. A fallback model can answer differently. A retried step can send the same email twice. This guide sets out rules for each.

## Which LLM errors should a gateway retry?

| Error | Retry? | How |
|---|---|---|
| 429 rate limit | Yes | Honour the provider's Retry-After header if present; otherwise exponential backoff with jitter |
| Request timeout | Yes, once or twice | Retry with a fresh timeout; count it against the budget |
| 500, 502, 503, 504 | Yes | Backoff with jitter; switch region or provider after repeated failures |
| 400 bad request | No | The request is malformed; retrying wastes tokens |
| 401 or 403 | No | Credential or permission problem; alert instead |
| Context length exceeded | No | Fix the prompt or route to a longer-context model by policy |
| Content filter refusal | No | Retrying to get past a safety filter is a policy problem, not a reliability fix |

The Retry-After header is defined in HTTP semantics (RFC 9110). Many model providers send it with rate-limit responses, and respecting it reduces retry storms.

## What is a retry budget, and why do agents need one?

A retry budget caps how much extra work retries can create. Without one, a provider outage turns every agent step into several calls, and an agent loop can multiply that again.

Set budgets at three levels:

1. **Per request.** For example, no more than two retries for one model call.
2. **Per agent run.** A cap on total retries across all steps in one task, so a long chain does not amplify failures.
3. **Per agent per time window.** If more than a set share of calls in a window are retries, stop retrying and fail fast. This protects both your spend and the provider.

Use exponential backoff with jitter (random spread) so many agents do not retry at the same instant. This is standard practice in distributed systems and is described in detail in the AWS Builders' Library.

## When is it safe to fall back to a different model?

Fallback means sending the request to another model or provider when the primary fails. It keeps the agent available, but the answer may differ. Treat a fallback model as a change to the agent, not just a routing detail.

Use these equivalence rules before adding a model to an agent's fallback list:

1. **Same capabilities.** It supports the features the agent uses: tool calling, structured output, context length, vision.
2. **Passes the agent's evals.** Run the agent's test suite against the fallback model and require a pass rate you set in advance. See [testing a cheaper model before switching](/blog/test-a-cheaper-model-before-switching-an-agent).
3. **Same data boundary.** The fallback must meet the same data residency and provider approval rules as the primary. A fallback that crosses a region boundary breaks policy silently.
4. **Approved model list.** It must be on the organisation's allowed models list.
5. **Cost ceiling.** A fallback to a more expensive model during an outage can be acceptable, but set a limit.
6. **Logged as fallback.** Every response served by a fallback should be tagged so you can review quality and cost separately.

Self-hosted models can be good fallbacks for some tasks; see [routing agents to internal models](/blog/route-agents-to-internal-models-through-llm-gateway).

## How do you stop retries from repeating tool calls?

This is the agent-specific risk. A model call is usually safe to retry. A tool call that creates a ticket, sends a payment or emails a customer is not.

- **Separate model retries from tool retries.** The gateway retries the model call. Tool calls should have their own policy, set per tool.
- **Use idempotency keys for side-effecting tools.** The agent attaches a unique key per intended action; the tool ignores duplicates. An Idempotency-Key HTTP header is being standardised in an IETF draft, and several payment APIs already use the pattern.
- **Mark tools as read-only or side-effecting** in your tool registry, and allow automatic retry only for read-only tools.
- **Check state before repeating.** For tools without idempotency, the agent should check whether the first attempt succeeded before trying again.

## What should you monitor?

Track retry rate, fallback rate, added latency from retries and cost of retries per agent and per provider. A rising fallback rate is an early sign of provider trouble or a quota set too low. See [monitoring LLM provider latency and errors](/blog/monitor-llm-provider-latency-and-errors).

## How Opencontroller does this

The Lyzr Gateway, Opencontroller's LLM Gateway, provides routing, fallback and retry, caching and cost attribution per agent and per version, behind OpenAI-compatible endpoints. In the October 2026 benchmark, scoped to its stated test conditions, each in-gateway routing method added under 0.7 ms at p95. Opencontroller's registry records an agent's tools and MCP servers, and the Agent Gateway sits in the path of agent traffic. For how gateway types differ, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane). [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**How many times should an LLM gateway retry?**
Usually one or two retries per call is enough, with backoff and jitter, plus caps per agent run and per time window.

**Should fallback models be from a different provider?**
Often yes, because provider outages are a common cause. The fallback must still pass the agent's evals and data residency rules.

**Can retries increase AI costs?**
Yes. Every retried call that reaches a model can be billed. Retry budgets and cost tracking per agent keep this visible.

## Key facts

LLM gateway retry policy for agents: retry 429 (honour Retry-After, defined in RFC 9110), timeouts and transient 5xx with exponential backoff and jitter; do not retry 400, 401/403, context-length errors or content-filter refusals. Use retry budgets per request, per agent run and per agent per time window. Fallback model equivalence rules: same capabilities, passes the agent's evals, same data boundary, on the approved model list, cost ceiling, tagged in logs. Never auto-retry side-effecting tool calls without idempotency keys (an Idempotency-Key HTTP header is an IETF draft). Opencontroller's Lyzr Gateway provides routing, fallback/retry, caching and per-agent, per-version cost attribution; Oct 2026 benchmark (scoped): under 0.7 ms added p95 per in-gateway routing method.

## Related

- [How do you monitor LLM provider latency and errors across agents?](https://opencontroller.lyzr.com/blog/monitor-llm-provider-latency-and-errors.md)
- [How do you test whether a cheaper model can replace the one an AI agent uses?](https://opencontroller.lyzr.com/blog/test-a-cheaper-model-before-switching-an-agent.md)
- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
