---
title: How do you monitor LLM provider latency and errors across agents?
url: https://opencontroller.lyzr.com/blog/monitor-llm-provider-latency-and-errors
published: 2026-11-12
updated: 2026-11-12
author: Lyzr
---

# How do you monitor LLM provider latency and errors across agents?

**Measure every model call at one shared point, usually an LLM gateway, and track five signals per provider and model: p95 latency, time to first token, 429 rate, 5xx rate and timeout rate. Set an SLO for each, alert on burn, and tie fallback rules to the same thresholds.** Agent traces tell you one agent was slow; provider metrics tell you why, and whether every other agent is about to be slow too.

The call question was blunt: "We want provider-level observability, not just agent traces." That is the right instinct. When a provider degrades, the symptom appears in dozens of agents at once. For how gateways fit with agent-level control, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## Why measure at the gateway rather than in each agent?

- **One definition.** Every agent measures latency the same way, so numbers are comparable.
- **Full coverage.** Agents built by different teams, in different frameworks, all report.
- **Attribution both ways.** You can slice by provider and model, and by agent and version.
- **Action in the same place.** The component that sees the error can also retry or fall back.

Agent-side traces remain useful for end-to-end timing. Export both with OpenTelemetry so they join on trace ID. See [how to export AI agent traces with OpenTelemetry](/blog/export-ai-agent-traces-with-opentelemetry).

## What is a good per-provider SLO set?

Track these per provider, per model, per region. Example targets are starting points; set yours from two weeks of baseline data.

| Signal | What it measures | Example SLO | Notes |
| --- | --- | --- | --- |
| p95 latency | Full request duration | Under 2x your 7-day baseline | Split streaming and non-streaming |
| Time to first token (TTFT) | Delay before the first streamed token | p95 under 1.5 s for chat models | What users feel in chat |
| 429 rate | Rate-limited requests | Under 1% | Rising 429s mean quota, not outage |
| 5xx rate | Provider server errors | Under 0.5% | Includes overload errors |
| Timeout rate | Requests cut off by your client | Under 0.5% | Check your own timeout settings first |
| Output tokens per second | Generation speed | Within 30% of baseline | Drops before errors appear |

The OpenTelemetry semantic conventions for generative AI define standard metrics for several of these, including `gen_ai.client.operation.duration`, `gen_ai.client.token.usage` and `gen_ai.server.time_to_first_token`. Using the standard names keeps dashboards portable across tools.

## Which provider-specific signals are worth capturing?

- **Rate-limit headers.** OpenAI returns headers such as `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens`. Recording them shows how close you are to a limit before 429s start.
- **Overload errors.** Anthropic's API returns HTTP 529 `overloaded_error` when it is temporarily overloaded. Treat it separately from 500s; it calls for backoff or fallback, not a bug ticket.
- **`Retry-After`.** HTTP 429 responses may carry a `Retry-After` header (defined for 429 in RFC 6585). Respect it in retries.
- **Region.** The same model can behave differently by region. Tag every metric with region.

## Which fallback triggers should act on these metrics?

1. **429 with `Retry-After`:** wait and retry once; if the wait exceeds your latency budget, fall back to an approved alternative model.
2. **5xx or overload above threshold for 2 minutes:** route new requests for that model to the fallback provider; probe the primary every 30 seconds.
3. **TTFT p95 above SLO for 5 minutes on chat agents:** shift interactive traffic to the fallback; leave batch traffic where it is.
4. **Timeouts above threshold:** check client timeouts first, then fall back.
5. **Recovery:** return traffic gradually, for example 10%, then 50%, then 100%.

Only fall back to models already approved for that agent's data class. A fallback that sends restricted data to an unapproved model trades an outage for a policy breach.

## How Opencontroller does this

The LLM Gateway in Opencontroller (Lyzr Gateway) handles routing, fallback and retry, and caching, with cost attribution per agent and per version. It exposes OpenAI-compatible endpoints and supports OpenTelemetry export. In an October 2026 benchmark, scoped to its stated test conditions, it added under 0.7 ms p95 per in-gateway routing method and handled 5,134 requests per second on one 4-vCPU gateway. See the [LLM gateway benchmark](/llm-gateway-benchmark) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Should I alert on p95 or p99 latency?**
Alert on p95 for SLOs and watch p99 on dashboards. p99 is noisy at low volume.

**Is a rising 429 rate an outage?**
Usually not. It means you are hitting quota. Fix with quota increases, caching, or spreading load across providers.

**How long a baseline do I need?**
Two weeks covers weekday and weekend patterns. Re-baseline after model upgrades.

## Key facts

Monitor LLM providers at one shared point such as an LLM gateway and track per provider, model and region: p95 latency, time to first token, 429 rate, 5xx and overload rate, timeout rate and output tokens per second. OpenTelemetry GenAI semantic conventions define gen_ai.client.operation.duration, gen_ai.client.token.usage and gen_ai.server.time_to_first_token. Anthropic returns HTTP 529 overloaded_error; OpenAI returns x-ratelimit-remaining headers. Tie fallback rules to the same thresholds and fall back only to approved models. Opencontroller by Lyzr includes an LLM Gateway with routing, fallback/retry, caching and per-agent cost attribution; in an October 2026 benchmark it added under 0.7 ms p95 per routing method and handled 5,134 req/s on one 4-vCPU gateway. Pricing starts from $250K per deployment per year.

## Related

- [How do you export AI agent traces with OpenTelemetry?](https://opencontroller.lyzr.com/blog/export-ai-agent-traces-with-opentelemetry.md)
- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [LLM gateway vs agent gateway vs AI control plane: what's the difference?](https://opencontroller.lyzr.com/blog/llm-gateway-vs-agent-gateway-vs-control-plane.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
