---
title: How do you roll out and roll back a new AI agent version safely?
url: https://opencontroller.lyzr.com/blog/canary-releases-and-rollback-for-ai-agents
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# How do you roll out and roll back a new AI agent version safely?

**Release a new agent version to a small share of traffic first, compare it against the current version on quality, cost and error metrics, and widen only when each stage passes a defined gate. Decide the rollback triggers before you start, and keep the previous version deployable so rollback is a routing change, not a rebuild.**

Agents change in more ways than ordinary software. A new prompt, a model swap, a new tool or a changed retrieval index can each change behaviour, and offline tests never cover every real conversation. A canary release limits how many users see a bad version before you notice.

## What counts as a new agent version?

Treat any of these as a version change that goes through the rollout process:

- System prompt or instructions
- Model or model version, including provider-side updates you opt into
- Tools, MCP servers or tool permissions
- Retrieval sources or index rebuilds
- Guardrail or policy settings
- Orchestration logic, such as step limits or handoff rules

Each version needs an identifier you can see in logs and cost reports. Without it, you cannot compare canary and baseline. See [when an agent change needs re-approval](/blog/when-does-an-ai-agent-change-need-re-approval) for which changes also need a governance review.

## What does a staged rollout look like?

The percentages below are an example to adapt, not a standard. Low-volume agents need longer stages to collect enough conversations.

| Stage | Traffic to new version | Minimum duration | Gate to pass |
|---|---|---|---|
| 0. Offline | 0% | Until suite passes | Evaluation suite at or above baseline, red-team cases pass |
| 1. Internal | Staff only | 1–2 days | No critical guardrail blocks; owners review sampled transcripts |
| 2. Canary | 1–5% | Enough for a few hundred conversations | Judge scores within tolerance of baseline; error rate not higher |
| 3. Expansion | 25–50% | 2–3 days | Cost per task within budget; no new incident types |
| 4. Full | 100% | — | Previous version kept deployable for a set period |

The Google SRE Workbook's chapter on canarying releases makes a point that applies directly: compare the canary against a control group running at the same time, not against last week's numbers, so traffic changes do not hide or fake a regression.

## Which metrics should gate each stage?

Pick a small set and write the thresholds down before the release:

1. **Task success or quality score** from LLM judges, compared with the baseline version.
2. **Guardrail block rate.** A jump means the version produces more unsafe or ungrounded output.
3. **Tool-call error rate**, including failed and retried calls.
4. **Cost per task**, not just cost per token. A cheaper model that needs more steps can cost more.
5. **Latency** at p95, since users notice the slow tail.
6. **Escalation or handoff rate** to humans.

See [how to measure cost per task](/blog/measure-cost-per-task-for-ai-agents) for the cost side.

## When should you roll back?

Set triggers in advance, and make some automatic:

- **Automatic rollback:** a critical policy violation, a data exposure event, or error rate above a hard ceiling.
- **Owner decision within hours:** quality score below baseline by more than the agreed tolerance, or cost per task above budget.
- **Hold and investigate:** mixed signals, such as better quality with higher cost.

Rollback should mean sending traffic back to the previous version. That only works if the previous prompt, model setting, tool configuration and retrieval index are all still available. Index rebuilds are the common trap: if the old index was deleted, "rollback" leaves you with the new data.

## What should you record for each release?

Keep a release record per version: version ID, what changed, evaluation results, stage dates, gate decisions and who approved them, and any rollback with its reason. This is useful evidence for change-management controls in SOC 2 and ISO/IEC 42001 reviews, and it answers the first question in any incident: what changed?

## How Opencontroller does this

Opencontroller, by Lyzr, keeps a governed registry of agents, versions, tools and MCP servers. Lyzr Gateway attributes cost per agent and per version, so canary and baseline costs sit side by side. Multi-turn and red-team evaluations gate a version before release, and LLM judges score production traffic after it. If a version misbehaves, rogue-agent handling can alert, quarantine or kill it, and per-agent Block and Quarantine controls act through the connected platform. See the [rogue agent guide](/blog/how-to-stop-a-rogue-ai-agent), the [production readiness gate](/blog/production-readiness-gate-for-ai-agents), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**How much traffic should a canary get?**
Enough to reach a meaningful number of conversations quickly, often 1–5%. Low-volume agents may need a longer stage instead of a bigger share.

**Can you A/B test two agent versions the same way?**
Yes. A canary is a cautious A/B test with a rollback plan. Keep the two groups running at the same time so the comparison is fair.

**What is the most common rollback failure?**
Missing dependencies. If the old prompt, model setting or retrieval index is gone, you cannot return to the previous behaviour.

## Key facts

Staged rollout for AI agent versions: offline evaluation, internal users, 1–5% canary, 25–50% expansion, full release with previous version kept deployable. Gate metrics: judge quality score vs baseline, guardrail block rate, tool-call error rate, cost per task, p95 latency, escalation rate. Opencontroller keeps a governed registry of agent versions, attributes LLM cost per agent and per version, runs pre-release evals and LLM judges on production traffic, and can quarantine or kill a misbehaving agent. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [When does a change to an AI agent need re-approval?](https://opencontroller.lyzr.com/blog/when-does-an-ai-agent-change-need-re-approval.md)
- [What should a CISO require before an AI agent goes to production?](https://opencontroller.lyzr.com/blog/production-readiness-gate-for-ai-agents.md)
- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
