---
title: What evidence should you keep when a human approves an AI agent's output?
url: https://opencontroller.lyzr.com/blog/evidence-for-human-review-of-ai-agent-outputs
published: 2026-11-12
updated: 2026-11-12
author: Lyzr
---

# What evidence should you keep when a human approves an AI agent's output?

**Keep a review record that lets an auditor reconstruct the decision: what the agent proposed, the inputs and sources it used, its confidence, which version produced it, who reviewed it, what they decided, why, and how long they spent. Then keep aggregate measures, such as approval and override rates, that show the review is real rather than a rubber stamp.**

Many enterprise agent workflows end with a person approving the result. A document-comparison agent extracts values from PDFs, compares them with master data in a spreadsheet, scores its confidence and passes the result to an underwriter. The approval is the control. If you cannot show how it happened, the control is hard to prove.

## Why does the review record matter?

Three reasons:

1. **Regulation expects it for high-risk uses.** The EU AI Act requires high-risk systems to support effective human oversight (Article 14) and automatic logging (Article 12). Deployers must assign oversight to people with the competence, training and authority to do it, and keep logs for at least six months (Article 26). After the Digital Omnibus, these high-risk duties apply from 2 December 2027 for Annex III systems.
2. **Disputes need reconstruction.** When a customer challenges a decision, you need the agent's proposal and the reviewer's action side by side.
3. **Automation bias is real.** Article 14 itself names the risk of people over-relying on system output. Evidence that reviewers sometimes disagree is evidence the oversight works.

## What fields belong in every review record?

| Field | Example | Why |
|---|---|---|
| Case or transaction ID | Policy or claim reference | Links to the business record |
| Agent and version | Comparison agent v14 | Ties the output to a tested version |
| Model and provider | As routed by the gateway | Shows what produced the output |
| Inputs | Document IDs, master-data row IDs | Reconstructs what the agent saw |
| Agent proposal | Extracted values, recommended action | The thing being approved |
| Confidence score | Per field or overall | Explains why it was routed to review |
| Sources cited | Page and field references | Lets the reviewer verify quickly |
| Guardrail results | Groundedness score, policy checks | Shows automated checks ran |
| Reviewer identity | From SSO, not a free-text name | Proves who decided |
| Reviewer role or authority | Underwriter level 2 | Shows competence and authority |
| Decision | Approve, reject, edit, request info | The control outcome |
| Edits made | Field-level diff | Shows what the human changed |
| Reason | Required for reject or edit | Explains disagreement |
| Timestamps | Presented, opened, decided | Measures real review time |

Store the record immutably, with a retention period agreed with legal. See [how long to retain AI agent logs](/blog/how-long-to-retain-ai-agent-logs).

## How do you prove the review is not a rubber stamp?

Single records prove a decision happened. Aggregates prove the process works. Track these monthly per agent and per reviewer:

1. **Approval rate.** A rate near 100% over thousands of cases deserves a look.
2. **Edit and override rate.** How often reviewers change the agent's output.
3. **Median time to decision.** Seconds per complex case suggests no real reading.
4. **Confidence calibration.** Are low-confidence cases actually wrong more often?
5. **Sampled second review.** A small share re-reviewed by a second person, with agreement measured.
6. **Escalations.** Cases sent up for more information.

When a pattern looks wrong, act on it: retrain reviewers, change the routing threshold, or slow the queue.

## When should a case go to a human at all?

Route by risk, not by habit. Common triggers:

- confidence below a set threshold on any required field,
- a value outside tolerance against master data,
- a failed groundedness or policy check,
- a high-value or high-impact case regardless of confidence,
- a random sample of high-confidence cases, to catch silent errors.

Document the thresholds and the reason for each. Change them through the same review process as code. Our post on [human approval for high-risk AI agent actions](/blog/human-approval-for-high-risk-ai-agent-actions) covers approval design; this post is about the evidence.

## How does this map to other frameworks?

The same records serve several frameworks. ISO/IEC 42001 expects documented operational controls and records. The NIST AI RMF calls for human oversight processes to be defined and assessed. For a crosswalk, see the [framework mapping hub](/blog/how-to-map-ai-agent-controls-to-compliance-frameworks). For assembling it into one pack, see the [regulator inspection evidence pack](/blog/regulator-inspection-evidence-pack-for-ai-agents).

## How Opencontroller does this

Opencontroller provides evidence for oversight duties; it does not make you compliant. It records agent versions in a governed registry, logs model calls through its LLM Gateway, applies live groundedness and task-adherence guardrails, and scores production traffic with LLM judges. Policy as code includes built-in mappings to OWASP, MITRE ATLAS, NIST and ISO 42001. Human-approval hold workflows are not an Opencontroller gateway feature; reviews happen in your business application, and these records sit alongside them. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Does every AI agent output need a human review record?**
No. Keep records where a human review is the control, typically for high-impact or high-risk decisions.

**How long should human review records be kept?**
At least as long as the underlying business record requires. For high-risk systems under the EU AI Act, deployers keep logs for at least six months.

**What shows that human oversight is effective?**
Edit and override rates, realistic review times, sampled second reviews and calibrated confidence thresholds.

## Key facts

A human review record for an AI agent output should include: case ID, agent and version, model, inputs, the agent's proposal, confidence score, sources, guardrail results, reviewer identity from SSO, reviewer authority, decision, edits, reason, and timestamps. Aggregate measures (approval rate, override rate, median review time, confidence calibration, sampled second review) show oversight is effective rather than a rubber stamp. EU AI Act Article 14 requires effective human oversight for high-risk systems and names automation bias; Article 12 requires automatic logging; Article 26 requires deployers to assign competent overseers and keep logs at least six months. After the Digital Omnibus (Regulation (EU) 2026/1744), Annex III high-risk duties apply from 2 December 2027. Opencontroller by Lyzr provides evidence (versioned registry, gateway logs, groundedness guardrails, LLM judges, framework mappings) but does not offer human-approval hold workflows in the gateway. Pricing starts from $250K per deployment per year.

## Related

- [How do you map AI agent controls to compliance frameworks?](https://opencontroller.lyzr.com/blog/how-to-map-ai-agent-controls-to-compliance-frameworks.md)
- [Which AI agent actions need human approval?](https://opencontroller.lyzr.com/blog/human-approval-for-high-risk-ai-agent-actions.md)
- [What evidence will a regulator ask for about your AI agents?](https://opencontroller.lyzr.com/blog/regulator-inspection-evidence-pack-for-ai-agents.md)
- [More on Evidence and frameworks](https://opencontroller.lyzr.com/blog/cluster/evidence-and-frameworks)
