---
title: How do you regression-test an AI agent after changing its prompt?
url: https://opencontroller.lyzr.com/blog/regression-test-ai-agent-after-prompt-change
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# How do you regression-test an AI agent after changing its prompt?

**Keep a versioned golden dataset of real tasks, run the old and new prompt against it, and score both with a mix of deterministic checks and calibrated LLM judges. Release only if the new version clears fixed pass thresholds and does not get worse on any critical slice. Wire this into CI so no prompt change reaches production untested.**

Small prompt edits change behaviour in places nobody expected. Adding "be concise" can drop a required disclaimer. Reordering instructions can change which tool the agent calls first. Without a regression suite, you find out from users.

## What goes into a golden dataset?

A golden dataset is a fixed set of inputs with known-good outcomes. Build it from:

1. **Real production tasks**, sampled across intents and redacted of personal data.
2. **Past failures and incidents**, each turned into a test so it cannot recur silently.
3. **Edge cases**: empty inputs, ambiguous requests, very long contexts, other languages.
4. **Policy cases**: requests the agent must refuse or escalate.
5. **Multi-turn scripts** for agents that hold conversations, because many regressions only show on turn three or later.

Start with 50 to 200 cases for a single-purpose agent and grow it with every incident. Version the dataset alongside the prompt so results are comparable over time.

## How should you score the results?

Use the cheapest reliable check for each property.

| Property | Best check | Example |
|---|---|---|
| Format and schema | Deterministic | Output parses as valid JSON with required fields |
| Tool choice | Deterministic | Called `lookup_order` before `issue_refund` |
| Exact facts | Deterministic or reference match | Refund amount equals the order total |
| Groundedness | LLM judge | Every claim is supported by retrieved context |
| Task completion | LLM judge with rubric | Customer's question fully answered |
| Tone and policy | LLM judge with rubric | No promises outside policy |
| Cost and latency | Measured | Tokens and seconds per task |

LLM judges need calibration. Have people label 50 or more cases, compare the judge's scores with theirs, and adjust the rubric until agreement is acceptable to you. Re-check calibration when you change the judge model.

## What pass thresholds should block a release?

Set them before you look at results, or they drift to whatever the new version scored. A workable rule set:

- **Hard gates:** 100% on schema validity and on policy refusal cases. Any failure blocks.
- **Overall quality:** the new version's mean judge score is no lower than the current version's, within a tolerance you set (for example one point on a 100-point scale).
- **No critical slice regresses:** check each intent or customer segment separately. An overall gain can hide a drop in one segment.
- **Cost and latency budgets:** tokens per task and p95 latency stay within agreed limits.

Run each case more than once if the agent is non-deterministic, and compare distributions rather than single runs.

## How do you gate releases in CI?

1. Store prompts, tool definitions and model settings in version control.
2. On every pull request that changes them, run the suite against the current and proposed versions.
3. Post a diff report: scores by slice, failed cases, cost change.
4. Block merge on any hard-gate failure. Require an approver for threshold misses.
5. After merge, release as a canary and keep scoring live traffic. See [canary releases and rollback for AI agents](/blog/canary-releases-and-rollback-for-ai-agents).

The NIST AI Risk Management Framework's MEASURE function calls for AI systems to be evaluated regularly with documented methods. A versioned suite with recorded results gives you evidence for that.

## How Opencontroller does this

Opencontroller runs multi-turn and red-team evaluations against registered agents and versions, and LLM judges score production traffic so regressions that slip past pre-release tests still surface. Live groundedness and task-adherence guardrails act at runtime. Cost attribution in the LLM Gateway is tracked per agent and per version, so a prompt change that raises cost is visible. More on the runtime side in [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent), [use LLM judges on production agent traffic](/blog/use-llm-judges-on-production-agent-traffic) and [guardrails vs evaluations for AI agents](/blog/guardrails-vs-evaluations-for-ai-agents).

## Frequently asked questions

**How many test cases do we need?**
Enough to cover every intent and every past failure. For a focused agent, 50 to 200 cases is a sensible start; add cases after each incident.

**Can an LLM judge grade its own model's output?**
It can, but bias is a risk. Calibrate against human labels, and consider a different model family for judging.

**Do model upgrades need the same tests?**
Yes. Treat a model version change exactly like a prompt change and run the full suite.

To see evaluations on your own agents, [book a demo](https://www.lyzr.ai/book-demo/).

## Key facts

Regression-testing an AI agent: versioned golden dataset (real tasks, past failures, edge cases, policy cases, multi-turn scripts; start 50-200 cases); deterministic checks for schema, tool choice and facts; calibrated LLM judges for groundedness, completion, tone; thresholds set in advance, hard gates at 100% on schema and refusals, no slice regression, cost and latency budgets; gate in CI then canary. NIST AI RMF MEASURE. Opencontroller (by Lyzr): multi-turn and red-team evals, LLM judges on production traffic, groundedness and task-adherence guardrails, cost per agent and version. From $250K per deployment per year.

## Related

- [How do you use LLM judges to score production AI agent traffic?](https://opencontroller.lyzr.com/blog/use-llm-judges-on-production-agent-traffic.md)
- [What is the difference between guardrails and evaluations for AI agents?](https://opencontroller.lyzr.com/blog/guardrails-vs-evaluations-for-ai-agents.md)
- [How do you roll out and roll back a new AI agent version safely?](https://opencontroller.lyzr.com/blog/canary-releases-and-rollback-for-ai-agents.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
