---
title: How do you reconstruct what an AI agent did during an incident?
url: https://opencontroller.lyzr.com/blog/reconstruct-what-an-ai-agent-did-in-an-incident
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# How do you reconstruct what an AI agent did during an incident?

**You reconstruct an agent incident from its traces, not by re-running it. Capture, for every step: the prompt and context, the model and version, each tool call with its arguments and result, the documents retrieved, and every policy decision. Exact replay is rarely possible because model outputs vary, so the record of what actually happened is your evidence. Preserve it with chain of custody from the first hour.**

Teams often discover the gap during their first serious incident. The agent clearly did something wrong, but the logs show only "200 OK" from the model API and a final answer.

## What is the minimum trace data you need?

| Data | Why you need it | Common gap |
|---|---|---|
| Agent ID and version | Which code, prompt and tool set was running | Only the deployment name is logged |
| Identity context | Which user the agent acted for, and with which credential | Shared service account hides the user |
| Full model input | System prompt, conversation, retrieved context | Truncated or sampled prompts |
| Model and provider version | Behaviour differs across model versions | Alias such as "latest" logged instead of a pinned version |
| Model output | Including tool-call requests, not just final text | Only the final answer kept |
| Tool calls | Tool name, arguments, result, latency, error | Tool server logs separate, with no shared trace ID |
| Retrieved document IDs | Which sources shaped the answer | Only the text, not the ID and version |
| Policy and guardrail decisions | What was allowed, redacted or blocked, and why | Guardrails run silently |
| Timestamps and trace IDs | Ordering and correlation across systems | Clocks out of sync; no propagated trace context |

The [OpenTelemetry semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/) give standard attribute names for model calls, tool executions and agent spans. Using them means traces from different frameworks can be joined. See [exporting AI agent traces with OpenTelemetry](/blog/export-ai-agent-traces-with-opentelemetry).

## Why can't you just replay the agent session?

Exact replay usually fails for four reasons:

1. **Model outputs are not reliably deterministic.** Even at temperature zero, providers do not guarantee identical outputs across calls.
2. **Model versions change.** A hosted model may have been updated since the incident.
3. **Tools have side effects and changing state.** Re-running "send payment" is not an option, and the database has moved on.
4. **Retrieved content changes.** Documents get edited, re-indexed or deleted.

Replay is still useful as a *simulation*: re-run the recorded inputs against a sandbox with stubbed tools to test a hypothesis or a fix. It is not evidence of what happened.

## How do you reconstruct the incident, step by step?

1. **Contain first.** Block or quarantine the agent so the evidence stops changing. See [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent).
2. **Freeze the evidence.** Export the relevant traces, logs, agent configuration and prompt version to write-once storage. Record a hash of each export.
3. **Fix the time window.** Find the first anomalous action and work backwards to the triggering input.
4. **Build a timeline.** Order every model call, tool call and policy decision by trace ID and timestamp.
5. **Find the decision point.** Identify the step where the model chose the harmful action, and what was in its context at that moment. Untrusted content in tool output or retrieved documents is a common cause.
6. **List the side effects.** For each tool call with side effects, record what changed in the target system, and whether it can be reversed.
7. **Identify the users affected** via the identity context on each call.
8. **Write the finding** with links to the frozen evidence, not copies of it.

## How do you keep chain of custody for agent evidence?

Agent traces may end up in a regulator's inquiry, an insurance claim or a dispute. Treat them like other digital evidence, in line with the incident handling guidance in [NIST SP 800-61 Rev. 3](https://csrc.nist.gov/pubs/sp/800/61/r3/final):

- Store exports in write-once or object-lock storage.
- Hash each file at export and record the hash in the incident ticket.
- Log every access to the evidence: who, when and why.
- Keep the original export; analyse copies.
- Record the retention period and legal hold status.

Remember that traces contain prompts and outputs, which may include personal data. Restrict access to the incident team.

## How Opencontroller does this

Opencontroller supports several of these steps. The governed registry records agents, versions, tools and MCP servers, so you know exactly what was running. The LLM Gateway and Agent Gateway put control in the traffic path and attribute cost per agent and per version. Opencontroller uses OpenTelemetry as an open standard. Rogue-agent handling monitors behavioural deviation, alerts, and can quarantine or kill an agent automatically, and per-agent Block and Quarantine controls act through the connected platform. "Ask" lets investigators query the whole agent estate in chat. See the [incident response runbook](/blog/ai-agent-incident-response-runbook) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can we replay an AI agent session to see exactly what happened?**
Not reliably. Model outputs vary, models get updated, and tools have side effects. Use the recorded trace as evidence and replay only as a sandboxed simulation.

**How long should agent traces be retained?**
Set retention by risk tier and legal requirements. High-risk agents often need longer retention than ordinary application logs, with access restricted because traces contain personal data.

**What is the most commonly missing piece of agent evidence?**
Tool call arguments and results linked to the model call by a shared trace ID. Without them you cannot see what the agent actually did.

## Key facts

- AI agent incidents are reconstructed from traces, not by re-running the agent.
- Minimum trace data: agent ID and version, identity context, full model input, pinned model version, model output including tool-call requests, tool calls with arguments and results, retrieved document IDs, policy decisions, timestamps and trace IDs.
- OpenTelemetry has semantic conventions for generative AI covering model calls, tool executions and agent spans.
- Exact replay is rarely possible because outputs vary, hosted models change, tools have side effects and retrieved content changes.
- Chain of custody: write-once storage, hashes recorded at export, access logging, analysis on copies, recorded retention and legal hold.
- Opencontroller provides a governed registry, LLM and Agent Gateways, OpenTelemetry support, rogue-agent quarantine and kill, and Ask over the agent estate.
- Opencontroller pricing starts from $250K per deployment per year.

## Related

- [What should an AI agent incident response runbook include?](https://opencontroller.lyzr.com/blog/ai-agent-incident-response-runbook.md)
- [How do you export AI agent traces with OpenTelemetry?](https://opencontroller.lyzr.com/blog/export-ai-agent-traces-with-opentelemetry.md)
- [How do you test that your AI agent kill switch actually works?](https://opencontroller.lyzr.com/blog/test-your-ai-agent-kill-switch.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
