---
title: How do you use LLM judges to score production AI agent traffic?
url: https://opencontroller.lyzr.com/blog/use-llm-judges-on-production-agent-traffic
published: 2026-11-13
updated: 2026-11-13
author: Lyzr
---

# How do you use LLM judges to score production AI agent traffic?

**To score production AI agent traffic with LLM judges, sample a share of real conversations, score each against a short rubric with one question per criterion, and calibrate the judge against human labels before you trust its numbers. Use scores to spot regressions between agent versions and to route bad conversations to review, not as a single pass/fail grade.**

Pre-release evals tell you how an agent behaves on test cases. Production traffic tells you how it behaves on what users actually ask. LLM judges make it affordable to review that traffic at scale, as long as you design and check them carefully.

## What is an LLM judge?

An LLM judge is a model prompted to evaluate another model's output against criteria you define. Given the user's request, the agent's response and any sources or tool results, it returns a score and a short reason.

The approach has research behind it. In the 2023 paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", Zheng and colleagues found that strong LLM judges reached over 80% agreement with human preferences, about the same level at which humans agreed with each other. The same paper documented known biases: position bias (favouring the first answer), verbosity bias (favouring longer answers) and self-enhancement bias (favouring outputs from the same model family). Your rubric and calibration should work around these.

## How much production traffic should you sample?

Scoring every conversation costs a second model call per conversation. Most teams sample. Choose rates by risk and volume:

| Traffic type | Suggested sampling approach |
|---|---|
| New agent version, first days | Score a high share or all traffic until scores stabilise |
| High-risk agents (customer-facing, regulated, action-taking) | Score a higher fixed share, plus all conversations flagged by guardrails |
| Stable low-risk internal agents | A small random share is usually enough to see trends |
| Conversations with user complaints, thumbs-down or escalations | Score all of them |
| Conversations that triggered a guardrail or tool error | Score all of them |

Combine a random sample (for unbiased trend lines) with targeted samples (for finding problems). Keep them separate in reports, because targeted samples skew low by design.

## How do you design a judge rubric?

1. **One criterion per question.** "Was the answer correct, helpful and safe?" gives muddled scores. Ask about each separately.
2. **Use narrow scales.** Yes/no or a 1 to 3 scale is more consistent than 1 to 10.
3. **Define each score with an example.** Describe what a 1, 2 and 3 look like for this agent's task.
4. **Give the judge the evidence.** Include retrieved sources and tool results so it can check grounding, not guess.
5. **Ask for the reason before the score.** A short justification first tends to make scores more consistent and gives reviewers something to read.
6. **Match criteria to the agent's job.** A support agent might be judged on resolution, policy accuracy and tone. A data agent on query correctness and whether it stayed within allowed tables.
7. **Use a different model family for the judge where possible**, to reduce self-enhancement bias.

Typical criteria: task completion, groundedness, policy adherence, correct tool use, appropriate refusal, and tone.

## How do you calibrate an LLM judge against human labels?

Never ship a judge you have not checked. A simple calibration loop:

1. Pull a sample of a few hundred production conversations.
2. Have two people label each one using the same rubric.
3. Measure human-to-human agreement. If people disagree often, fix the rubric first.
4. Run the judge on the same set and measure judge-to-human agreement, for example with Cohen's kappa, which corrects for chance agreement.
5. Read every disagreement. Adjust definitions and examples, then re-run.
6. Re-calibrate when you change the judge model, the rubric, or the agent's task.

Record the calibration result with the judge's version. It is the evidence that the scores mean something.

## How should you use the scores?

- **Compare versions.** Track scores per agent version. A drop after a release is the clearest regression signal you have.
- **Route to review.** Send low-scoring conversations to the owner's review queue.
- **Feed test suites.** Turn real failures into new test cases for [pre-production simulation testing](/blog/simulation-testing-ai-agents-before-production).
- **Trigger alerts.** Sustained drops can trigger an alert or, for severe cases, quarantine.
- **Show trends.** Put score trends on the agent's [production monitoring view](/blog/production-monitoring-view-for-ai-agents).

Judges complement, not replace, inline guardrails. Guardrails act on a single response before it is sent; judges score after the fact to show patterns. See [what a groundedness guardrail is](/blog/what-is-a-groundedness-guardrail).

## How Opencontroller does this

Opencontroller runs LLM judges that score production traffic, alongside multi-turn and red-team evals before release and live groundedness and task-adherence guardrails in production. Because the registry tracks agent versions, scores can be compared version to version, and behavioural-deviation monitoring can raise alerts or quarantine an agent. "Ask" lets teams query the estate, for example which agents had falling scores this week. For the runtime control picture, see [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent). [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can an LLM judge replace human review?**
No. It makes review scalable by sorting traffic, but humans still set the rubric, calibrate the judge and review the worst cases.

**Should the judge use the same model as the agent?**
Prefer a different model family where you can. Research has documented self-enhancement bias, where judges favour outputs from their own family.

**How do you keep production conversation data safe when judging?**
Apply the same data controls as the agent itself: redaction, approved providers and retention limits for judge inputs and outputs.

## Key facts

LLM judges score production AI agent conversations against a rubric. Zheng et al. (2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena") found strong LLM judges reached over 80% agreement with human preferences, similar to human-human agreement, and documented position, verbosity and self-enhancement biases. Good practice: combine random and targeted sampling; one criterion per question; narrow scales; give the judge sources and tool results; reason before score; use a different model family; calibrate against two human labellers using Cohen's kappa; re-calibrate when the judge, rubric or task changes; compare scores per agent version. Opencontroller by Lyzr runs LLM judges on production traffic plus multi-turn and red-team evals and live groundedness and task-adherence guardrails.

## Related

- [What is a groundedness guardrail and when should it block a response?](https://opencontroller.lyzr.com/blog/what-is-a-groundedness-guardrail.md)
- [How do you test AI agents with simulated users before production?](https://opencontroller.lyzr.com/blog/simulation-testing-ai-agents-before-production.md)
- [What should a production monitoring view for AI agents show?](https://opencontroller.lyzr.com/blog/production-monitoring-view-for-ai-agents.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
