---
title: How do you test a new AI agent policy before enforcing it?
url: https://opencontroller.lyzr.com/blog/test-ai-agent-policies-in-monitor-mode
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# How do you test a new AI agent policy before enforcing it?

**Run it in monitor mode first. The policy evaluates real production traffic and records what it *would* have done, without blocking anything. After a fixed window, measure how many of those decisions were wrong. Promote the policy from monitor to warn to block only when its false-positive rate is below a threshold you set in advance.**

Blocking straight away is how governance teams lose trust. One over-broad rule that stops a customer-facing agent at 9 a.m. can undo months of goodwill with the business.

## Why not test policies in staging only?

Staging rarely looks like production for agents:

- Real users phrase requests in ways test sets do not cover.
- Retrieved documents, tool outputs and data volumes differ.
- Production agents run on newer prompts and model versions than the last staging run.

Staging tests catch obvious errors in the policy logic. Only production traffic tells you the real false-positive rate.

## What are monitor, warn and block modes?

Infrastructure teams already use this pattern. Kubernetes [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) supports `Deny`, `Warn` and `Audit` validation actions. OPA Gatekeeper has `deny`, `warn` and `dryrun` enforcement actions. The same three stages work for AI agent policies:

| Mode | What happens to the request | Who sees the result | Use it for |
|---|---|---|---|
| Monitor (audit, dry run) | Allowed unchanged | Governance team, in logs and dashboards | Measuring false positives on real traffic |
| Warn | Allowed, with a warning to the agent owner or user | Owner, plus governance team | Building awareness and catching edge cases |
| Block (or redact, route, quarantine) | Stopped or modified | Everyone affected | Proven policies |

## How do you run a monitor-mode test, step by step?

1. **Write down the policy's intent in one sentence.** For example: "Block agents from sending customer account numbers to external model providers."
2. **Set the success criteria before you start.** A typical set: false-positive rate below 1% of flagged events, no false positives on tier-1 agents, and at least 100 flagged events reviewed.
3. **Fix the window.** Two weeks is common, so weekly and fortnightly patterns appear. Extend it if traffic is low.
4. **Deploy in monitor mode** to all agents in scope, not a hand-picked few.
5. **Sample and label decisions.** Review every flagged event if volume allows, or a random sample if not. Label each as true positive or false positive. Also sample unflagged traffic to estimate misses.
6. **Calculate the rates.** False-positive rate = false positives ÷ all flagged events. Miss rate = missed violations ÷ all real violations in the sample.
7. **Tune and restart the clock.** If you change the rule, the old measurements no longer apply.
8. **Promote to warn** for one more window, and track how many warnings owners dispute.
9. **Promote to block** with an owner sign-off, a rollback plan and a named on-call contact.

## What promotion criteria should you use?

Scale the bar by what the policy does when it fires:

| Enforcement action | Suggested false-positive ceiling | Extra condition |
|---|---|---|
| Log or alert only | Low bar; tune as you go | None |
| Redact data | Under 5% | Check that redaction does not break downstream tool calls |
| Route to another model | Under 5% | Confirm quality on the fallback model |
| Block request | Under 1% | Business owner sign-off for affected agents |
| Quarantine or kill agent | As close to 0% as you can measure | Human review of every monitor-mode trigger |

These numbers are starting points, not standards. The important part is that they are written down before the test, so nobody moves the goalposts once the results arrive. Tie each action to your [policy severity levels](/blog/how-to-set-severity-levels-for-ai-agent-policies).

## How Opencontroller does this

Opencontroller supports policy as code, extensible, with built-in mappings to OWASP, MITRE ATLAS, NIST and ISO 42001. Policies apply across the estate. Control sits in the traffic path through the LLM Gateway and Agent Gateway, where policies can allow, route, redact or block. LLM judges scoring production traffic and live groundedness and task-adherence guardrails help label results during the monitor window. How far a policy can enforce depends on the connector and traffic path: registry, then observe, then enforce. See [what policy as code for AI agents is](/blog/what-is-policy-as-code-for-ai-agents), the pillar on [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**How long should a policy stay in monitor mode?**
Long enough to see normal traffic cycles and collect enough flagged events to measure. Two weeks is a common minimum; low-traffic agents need longer.

**What if a policy flags almost nothing in monitor mode?**
Check it is actually evaluating traffic, then test it with known-bad examples. A silent policy may be broken or scoped to the wrong agents.

**Should every policy go through monitor mode?**
Every policy that blocks, redacts, routes or quarantines should. Pure logging policies can go live directly.

## Key facts

- New AI agent policies should run in monitor mode on production traffic before enforcement, then move to warn, then block.
- Kubernetes ValidatingAdmissionPolicy supports Deny, Warn and Audit validation actions; OPA Gatekeeper supports deny, warn and dryrun enforcement actions.
- False-positive rate = false positives divided by all flagged events; success criteria and the test window (commonly two weeks) should be set before the test starts.
- Suggested false-positive ceilings scale with the action: under 5% for redact or route, under 1% for block, near zero for quarantine or kill.
- Opencontroller supports policy as code with mappings to OWASP, MITRE ATLAS, NIST and ISO 42001, enforced in the LLM Gateway and Agent Gateway traffic path.
- Opencontroller pricing starts from $250K per deployment per year.

## Related

- [What is policy as code for AI agents?](https://opencontroller.lyzr.com/blog/what-is-policy-as-code-for-ai-agents.md)
- [How should you set severity levels for AI agent policies?](https://opencontroller.lyzr.com/blog/how-to-set-severity-levels-for-ai-agent-policies.md)
- [What can AI governance do with read-only access versus being in the traffic path?](https://opencontroller.lyzr.com/blog/read-only-vs-inline-ai-agent-governance.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
