---
title: How much do LLM-as-judge evaluations cost on production traffic?
url: https://opencontroller.lyzr.com/blog/cost-of-llm-judge-evaluations
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# How much do LLM-as-judge evaluations cost on production traffic?

**LLM-judge cost = traces per day × sampling rate × criteria per trace × (judge input tokens × input price + judge output tokens × output price). Judging every trace on several criteria can rival the cost of running the agent itself. Sampling, smaller judge models, batch APIs (50% off at both OpenAI and Anthropic) and caching the rubric usually cut the bill by an order of magnitude.**

Continuous evaluation is how you catch quality drift in production agents. It is also a second model workload that grows with traffic. Budget for it before turning it on.

## What is the cost formula?

Break the cost into five variables:

| Variable | What drives it | Typical lever |
|---|---|---|
| Traces per day (T) | Agent traffic | None—this is demand |
| Sampling rate (s) | Share of traces judged | Risk-based sampling |
| Criteria per trace (c) | Separate judge calls per trace (groundedness, task adherence, tone…) | Combine criteria in one call where accuracy allows |
| Input tokens per judgment (i) | Rubric + trace content (prompt, retrieved context, tool calls, answer) | Trim traces, cache the rubric |
| Output tokens per judgment (o) | Score plus reasoning | Ask for a score and short rationale |

**Daily cost = T × s × c × (i × input price + o × output price)**

The trace content usually dominates `i`. Agent traces with retrieved documents and tool outputs are long, and the judge must read them to score groundedness.

## A worked example

The prices below are **illustrative assumptions** for a small judge model, not a quote from any provider: $1 per million input tokens and $5 per million output tokens. Replace them with your contracted rates.

Assumptions:

- 50,000 agent traces per day
- 2 criteria per trace (groundedness and task adherence), judged in separate calls
- 3,800 input tokens per judgment (800-token rubric + 3,000-token trace)
- 200 output tokens per judgment

Cost per judgment: 3,800 × $1/1M + 200 × $5/1M = $0.0038 + $0.0010 = **$0.0048**.
Cost per trace (2 criteria): **$0.0096**.

| Scenario | Judged traces/day | Daily cost | 30-day cost |
|---|---|---|---|
| Judge 100% of traffic | 50,000 | $480 | $14,400 |
| Sample 10% | 5,000 | $48 | $1,440 |
| Sample 10% + batch API (50% off) | 5,000 | $24 | $720 |

Moving from judging everything to a 10% sample in batch mode cuts the illustrative bill by 95%. Swap in a larger judge model at several times the price and the 100% column grows to a number most finance teams will question.

## Which levers cut the cost?

1. **Sample by risk, not uniformly.** Judge 100% of high-risk agents or actions (money movement, external messages) and a small sample of low-risk ones. Always judge traces flagged by guardrails or users.
2. **Use batch APIs for non-urgent judging.** Anthropic's [Message Batches](https://platform.claude.com/docs/en/build-with-claude/batch-processing) charge 50% of standard prices, with results within 24 hours; OpenAI's [Batch API](https://developers.openai.com/api/docs/guides/batch) offers a similar 50% discount. Anthropic notes batch and prompt caching discounts can stack.
3. **Cache the rubric.** The rubric is identical across calls, so it is a natural prompt-cache prefix. In the example, an 800-token rubric is about 21% of input tokens.
4. **Use a smaller judge** for simple criteria, and validate it against a human-labelled sample before trusting it.
5. **Trim what the judge reads.** Groundedness needs the answer and the cited context, not the full conversation history.
6. **Combine criteria** into one call when scores stay accurate—this divides `c`.
7. **Watch reasoning tokens.** Reasoning judge models can add many hidden output tokens; see [reasoning tokens and AI agent cost](/blog/reasoning-tokens-and-ai-agent-cost).

## How should you attribute judge costs?

Treat evaluation as a cost of running the agent, but keep it visible as its own line. Tag judge calls with the agent ID and version they evaluate, so you can report cost per agent including evaluation. Keep them separate from build-time evaluation spend; see [separating AI build costs from agent runtime costs](/blog/separate-ai-build-costs-from-agent-runtime-costs). Route judge calls through the same LLM gateway as production calls so budgets, caching and attribution apply.

## How Opencontroller does this

Opencontroller runs LLM judges that score production traffic, alongside multi-turn and red-team evaluations and live groundedness and task-adherence guardrails. Its LLM Gateway (Lyzr Gateway) provides routing, fallback and retry, caching, and cost attribution per agent and per version. In Lyzr's October 2026 benchmark, under its stated test conditions, caching lowered model cost by 23.5% on mixed synthetic traffic. For how gateways fit together, see the pillar on [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), the [LLM gateway benchmark](/llm-gateway-benchmark), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Should I judge every production trace?**
Usually not. Judge all traces for high-risk agents and flagged runs, and sample the rest. Uniform 100% judging multiplies cost with little extra signal.

**Can judge calls use a batch API?**
Yes, when results are not needed in real time. Anthropic and OpenAI both price batch requests at 50% of standard rates, with results within 24 hours.

**What drives LLM-judge cost the most?**
Usually the trace tokens the judge must read—retrieved context and tool outputs—multiplied by the number of criteria judged separately.

## Key facts

- LLM-judge daily cost = traces per day × sampling rate × criteria per trace × (input tokens × input price + output tokens × output price).
- Worked example (illustrative prices $1/M input, $5/M output): 50,000 traces/day, 2 criteria, 3,800 input and 200 output tokens costs $480/day at 100% judging, $48/day at 10% sampling, $24/day with a 50% batch discount.
- Anthropic Message Batches charge 50% of standard prices with results within 24 hours; batch and prompt caching discounts can stack.
- OpenAI's Batch API also offers a 50% discount.
- Opencontroller runs LLM judges on production traffic; its LLM Gateway provides caching and cost attribution per agent and per version.
- Lyzr's Oct 2026 benchmark measured 23.5% lower model cost with caching on mixed synthetic traffic, under stated test conditions.
- Opencontroller pricing starts from $250K per deployment per year.

## Related

- [How do you use LLM judges to score production AI agent traffic?](https://opencontroller.lyzr.com/blog/use-llm-judges-on-production-agent-traffic.md)
- [When should AI agent workloads use batch APIs?](https://opencontroller.lyzr.com/blog/use-batch-apis-to-cut-agent-costs.md)
- [How does provider prompt caching cut AI agent costs?](https://opencontroller.lyzr.com/blog/provider-prompt-caching-for-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
