---
title: How do you run a new AI agent in shadow mode before it acts?
url: https://opencontroller.lyzr.com/blog/run-a-new-ai-agent-in-shadow-mode
published: 2026-11-16
updated: 2026-11-16
author: Lyzr
---

# How do you run a new AI agent in shadow mode before it acts?

**Feed the agent a copy of real work, let it reason and call read-only tools, but replace every write or send tool with a stub that records what it would have done. Compare those proposed actions with what humans actually did, and only switch it on when it meets agreed accuracy and safety thresholds over a set number of cases.**

Shadow mode sits between offline testing and a [canary release](/blog/canary-releases-and-rollback-for-ai-agents). It answers "would this agent have done the right thing on last week's real work?" without risk to customers. It is part of our [runtime control](/blog/how-to-stop-a-rogue-ai-agent) series.

## Why use shadow mode instead of only offline evals?

Offline evaluation sets are curated. Real traffic is messy: odd phrasing, missing fields, edge cases nobody wrote down. Shadow mode tests the agent on the real distribution of work.

It also tests things evals miss:

- Real tool latency and errors.
- Real permission boundaries (a read tool returning "access denied").
- Real cost per task at production volume.
- How often the agent would escalate or give up.

## How do you design a shadow mode?

1. **Mirror the inputs.** Copy incoming tickets, emails or requests to the shadow agent. The human process continues unchanged.
2. **Classify every tool.** Mark each tool as read or write. Reads (search, look up order) can run for real against production with read-only credentials. Writes (refund, send email, update record) are replaced.
3. **Stub write tools.** The stub validates arguments, records the proposed call and returns a realistic success response so the agent continues.
4. **Isolate identity.** Give the shadow agent its own identity with no write permissions at all, so a misclassified tool fails safely.
5. **Suppress outbound messages.** Nothing the shadow agent writes reaches a customer or employee.
6. **Capture the human outcome.** Record what the human actually did on the same case, for comparison.
7. **Trace everything.** Store prompts, tool calls, proposed actions and cost per case.

## How do you compare the agent with humans?

Compare decisions, not wording. Build a comparison table per case type.

| Measure | How to score | Typical threshold to exit |
|---|---|---|
| Action agreement | Proposed action matches human action | Agreed by the business, e.g. 90%+ on target case types |
| Harmful action rate | Proposed action that a reviewer marks unsafe | Zero on high-risk actions |
| Escalation rate | Agent says "needs a human" | Within the planned automation rate |
| Groundedness | Claims supported by retrieved sources | Agreed minimum score |
| Policy violations | Guardrail or policy hits | Zero blocks on high-severity rules |
| Cost per case | Model and tool cost | Within budget |
| Latency | Time to proposed action | Within service target |

Use [LLM judges](/blog/use-llm-judges-on-production-agent-traffic) to score volume, and have humans review a sample, especially every disagreement on high-risk actions. Disagreement is not always the agent's fault; sometimes the human was wrong, which is worth knowing too.

## What exit criteria should you set?

Write them down before the shadow run starts, so nobody moves the goalposts.

- **Minimum volume.** For example, at least a few hundred cases per case type, or two full business cycles.
- **Thresholds per measure.** From the table above, agreed with the agent owner and risk team.
- **Zero tolerance items.** Any proposed action that would breach a policy, move money incorrectly or contact the wrong person resets the clock.
- **Stability.** Thresholds met for consecutive weeks, not one good day.
- **Version lock.** Any prompt, model or tool change restarts the shadow period for affected case types.

## What comes after shadow mode?

Move to a staged rollout:

1. Human-approved actions: the agent proposes, a person clicks approve.
2. A canary: a small share of cases acted on automatically.
3. Wider rollout with [monitoring and SLOs](/blog/slos-for-production-ai-agents).

Keep the shadow comparison running on a sample even after launch. It is a cheap ongoing check for drift.

## How Opencontroller does this

Opencontroller supports multi-turn and red-team evaluations before release, and LLM judges that score production traffic after it. Its registry tracks agent versions, tools and MCP servers, and policy as code can restrict which tools an agent version may call. Traces export over OpenTelemetry. See the [capability reference](/capability-reference) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is shadow mode safe if read tools hit production?**
It is if the shadow agent uses a separate read-only identity, write tools are stubbed, and outbound messaging is suppressed. Test the stubs before mirroring real traffic.

**How long should shadow mode last?**
Long enough to reach the minimum case volume and cover normal business cycles, often two to four weeks. High-risk agents warrant longer.

**Does shadow mode double our model costs?**
It adds the cost of running the agent on mirrored traffic. Sample a share of traffic if volume is high; you need representative cases, not all of them.

## Key facts

Shadow mode runs a new AI agent on mirrored real inputs with read tools live (read-only identity) and write or send tools stubbed to record proposed actions. Compare proposed actions with human outcomes on action agreement, harmful action rate, escalation rate, groundedness, policy violations, cost and latency. Set exit criteria before starting: minimum volume, thresholds, zero-tolerance items and stability over consecutive weeks.

## Related

- [How do you roll out and roll back a new AI agent version safely?](https://opencontroller.lyzr.com/blog/canary-releases-and-rollback-for-ai-agents.md)
- [How do you test AI agents with simulated users before production?](https://opencontroller.lyzr.com/blog/simulation-testing-ai-agents-before-production.md)
- [How do you use LLM judges to score production AI agent traffic?](https://opencontroller.lyzr.com/blog/use-llm-judges-on-production-agent-traffic.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
