---
title: How do you test whether a cheaper model can replace the one an AI agent uses?
url: https://opencontroller.lyzr.com/blog/test-a-cheaper-model-before-switching-an-agent
published: 2026-11-11
updated: 2026-11-11
author: Lyzr
---

# How do you test whether a cheaper model can replace the one an AI agent uses?

**Replay a representative sample of the agent's real requests against the cheaper model, score both models' outputs with the same judges and checks, and accept the switch only if quality stays within a margin you set in advance. Then route a small share of live traffic to the new model, with automatic fallback, before you cut over.**

Model prices differ by an order of magnitude or more between tiers, and new smaller models arrive every few months. For a high-volume agent, moving even part of its traffic to a cheaper model can be one of the largest cost savings available. The risk is a quiet quality drop that nobody notices until customers do. The method below makes that risk measurable.

## Why not just run a benchmark?

Public benchmarks measure general ability. Your agent does a narrow set of tasks with your prompts, your tools and your data. A model that scores lower on a public leaderboard can match a larger one on a simple classification agent, and a model that scores well can fail on your tool-calling format. Test on your traffic, not someone else's.

## What does the replay-and-judge method look like?

1. **Define what "good" means for this agent.** Pick three to five criteria: task completed, groundedness in retrieved sources, correct tool calls, format valid, tone or policy adherence.
2. **Sample real requests.** Pull traces from production and stratify by task type. Include rare tasks, long multi-turn conversations and every known past failure. A few hundred samples is a common floor; more for agents with many tasks.
3. **Strip or mask sensitive data** if the replay runs outside the normal production path.
4. **Replay against the candidate model** with the same system prompt, tools and parameters. For multi-turn agents, replay full conversations, not single turns.
5. **Score both outputs with the same judges.** Use LLM judges for subjective criteria and deterministic checks for tool-call validity and schemas.
6. **Calibrate the judge.** Have humans rate a slice (for example 50 to 100 items) and check the judge agrees. Research on LLM-as-a-judge shows strong judges can agree with humans at rates close to human-human agreement, but also documents biases such as position and verbosity bias, so randomise order and compare like with like.
7. **Compare against thresholds you set before looking at results.**

## What acceptance thresholds should you use?

Set them by risk. An example:

| Criterion | Low-risk internal agent | Customer-facing agent | Regulated decision support |
|---|---|---|---|
| Task completion vs current model | Within 5 points | Within 2 points | No measurable drop |
| Groundedness | Within 5 points | Within 2 points | No measurable drop |
| Valid tool calls | 98%+ | 99%+ | 99.5%+ |
| Red-team / policy tests | No new failures | No new failures | No new failures |
| Cost per successful task | Lower | Lower | Lower |
| p95 latency | Not worse by more than 20% | Not worse by more than 10% | Not worse by more than 10% |

Note the "cost per successful task" row. A cheaper model that needs more retries or longer conversations can cost more per completed task than the expensive one.

## How do you move live traffic safely?

1. **Register the change as a new agent version**, so cost and quality are tracked separately.
2. **Route 5% of traffic** through a gateway to the new model.
3. **Keep fallback** to the current model on errors, timeouts or failed format checks.
4. **Score live traffic** with the same judges for at least a week, or until each task type has enough volume.
5. **Step up** to 25%, 50%, 100% if scores hold.
6. **Keep the old version registered** for instant rollback.

Consider partial switches too. Many agents route simple tasks to a smaller model and keep complex ones on the larger model. Caching repeated requests is a separate saving you can stack on top. See [how to attribute LLM costs to AI agents](/blog/how-to-attribute-llm-costs-to-ai-agents) for the cost side and [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) for where routing sits.

## How Opencontroller does this

Opencontroller's LLM judges score production traffic, and its multi-turn and red-team evals test candidate versions before release. The Lyzr Gateway handles routing, fallback and retry, and caching, and attributes cost per agent and per version, so you can compare cost per version side by side. In Lyzr's October 2026 benchmark, caching lowered model cost by 23.5% on mixed synthetic traffic, under the stated test conditions; see the [LLM gateway benchmark](/llm-gateway-benchmark). [Book a demo](https://www.lyzr.ai/book-demo/) to try it on one agent.

## Frequently asked questions

**How many requests do you need to test a cheaper model for an agent?**
Enough to cover the real task mix. A few hundred stratified samples is a common start, plus every known past failure.

**Can an LLM judge decide if a cheaper model is good enough?**
Mostly, once calibrated against human ratings and used identically for both models. Keep deterministic checks for tool calls and formats.

**What is the safest way to switch an agent's model in production?**
Canary a small share of traffic through a gateway with fallback, compare scores and cost per version, then step up.

## Key facts

Method for switching an AI agent to a cheaper model: replay a representative sample of real production requests (typically a few hundred to a few thousand) against the candidate model, score both outputs with the same LLM judges and task-specific checks, accept only if quality stays within a pre-set margin, then route a small share of live traffic (for example 5%) with automatic fallback before full cutover. Opencontroller by Lyzr provides LLM judges that score production traffic, multi-turn evals, and an LLM Gateway with routing, fallback/retry, caching and cost attribution per agent and per version. In Lyzr's October 2026 benchmark, caching reduced model cost by 23.5% on mixed synthetic traffic. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [How do you attribute LLM costs to individual AI agents?](https://opencontroller.lyzr.com/blog/how-to-attribute-llm-costs-to-ai-agents.md)
- [LLM gateway vs agent gateway vs AI control plane: what's the difference?](https://opencontroller.lyzr.com/blog/llm-gateway-vs-agent-gateway-vs-control-plane.md)
- [How do you red-team AI agents with multi-turn attacks?](https://opencontroller.lyzr.com/blog/how-to-red-team-ai-agents-multi-turn.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
