---
title: What should a production monitoring view for AI agents show?
url: https://opencontroller.lyzr.com/blog/production-monitoring-view-for-ai-agents
published: 2026-11-12
updated: 2026-11-12
author: Lyzr
---

# What should a production monitoring view for AI agents show?

**A production view for AI agents should answer five questions at a glance: which agents are running or paused, what is going wrong ranked by severity, which agents are slowing down or failing, what each agent costs and who owns it, and whether answers are still good. Build it agent-first, not model-first: every panel should click through to one agent, its version and its owner.**

Most teams start with an LLM dashboard: tokens, latency and errors per model. That is useful for the platform team, but it does not tell a risk lead which agent misbehaved or who should fix it. Here is a panel-by-panel spec for a view that does.

## Why is an LLM dashboard not enough for agents?

Agents add things a model dashboard cannot show. One agent can call several models. One model can serve dozens of agents. Agents also call tools, act for users and change version. When something goes wrong, the question is "which agent, which version, whose?" A model-first view cannot answer it.

So the unit of the view is the **agent**, keyed by agent ID and version, with an owner attached.

## What panels should the view have?

| Panel | What it shows | Why it matters | Click-through |
|---|---|---|---|
| 1. Estate status | Count of agents by state: running, paused, quarantined, retired; new since last week | Shows what is live and what was stopped | Agent list filtered by state |
| 2. Issues by severity | Open policy violations and incidents ranked critical, high, medium, low, with age | Puts attention on the worst first | Issue detail with trace |
| 3. Behaviour deviation | Agents whose tool calls, volume or targets moved away from their normal pattern | Early sign of a rogue or broken agent | Agent behaviour timeline |
| 4. Performance alerts | Latency p95, error rate and timeouts per agent, against its own baseline | Separates one slow agent from a provider outage | Traces for the slow window |
| 5. Quality scores | Groundedness and task-adherence scores from judges on production traffic, trend by version | Catches answers getting worse after a change | Lowest-scoring conversations |
| 6. Cost and owner | Spend per agent and per version, budget used, owner and team | Makes cost someone's job | Cost breakdown by model and call |
| 7. Posture | Agents with risky permissions or config, such as broad data access or unauthenticated tools | Fixes risk before it becomes an incident | Finding detail and fix |
| 8. Changes | Recent deployments, version changes, policy changes and new tools or MCP servers | Most incidents follow a change | Change diff |

## What should each panel get right?

1. **Baselines per agent, not global thresholds.** A support agent at 2 seconds p95 may be normal; a routing agent at 2 seconds may be broken.
2. **Severity defined in policy.** Use the same levels in the view, alerts and tickets. See [how to set severity levels for agent policies](/blog/how-to-set-severity-levels-for-ai-agent-policies).
3. **Version on every metric.** Quality and cost should be comparable before and after a change.
4. **Owner on every row.** An issue with no owner will not be fixed.
5. **Actions next to data.** From the issue panel, a reviewer should be able to pause, quarantine or kill the agent, with the action logged. See [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent).
6. **Traces behind every number.** Use OpenTelemetry so each metric drills down to the spans that produced it; see [exporting agent traces with OpenTelemetry](/blog/export-ai-agent-traces-with-opentelemetry).
7. **Cost attributed at the call.** Tag each model call with agent and version at the gateway rather than splitting a bill afterwards; see [attributing LLM costs to agents](/blog/how-to-attribute-llm-costs-to-ai-agents).

## Who uses which part of the view?

- **Platform and SRE teams** watch panels 1, 4 and 8.
- **Risk and security teams** watch panels 2, 3 and 7.
- **Agent owners** watch their own row across panels 4, 5 and 6.
- **Finance** watches panel 6, rolled up by team.

One view with role-based filters beats four separate dashboards that disagree.

## How Opencontroller does this

Opencontroller, Lyzr's open control plane for enterprise AI agents, puts the agent estate in one registry and monitors it at runtime: behavioural-deviation monitoring with alerts, automatic quarantine and kill, LLM judges scoring production traffic, live groundedness and task-adherence guardrails, Posture findings for risky permissions and configuration, and cost attribution per agent and per version through the LLM Gateway. Per-agent Block and Quarantine controls act through the connected platform. "Ask" lets teams query the whole estate in chat, for example "which agents had high-severity issues this week?". [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Should the monitoring view be per model or per agent?**
Per agent, with version and owner. Model views stay useful for the platform team, but incidents and fixes happen at the agent level.

**How often should quality scores update?**
Continuously or daily for high-volume agents, using judges on sampled production traffic, and always after a version change.

**What is the first panel to build?**
Estate status with owners. You cannot monitor agents you have not listed, and every other panel depends on that list.

## Key facts

A production monitoring view for AI agents should be agent-first (keyed by agent, version and owner), with eight panels: estate status (running, paused, quarantined, retired), issues ranked by severity, behaviour deviation, performance alerts against per-agent baselines, quality scores from LLM judges (groundedness, task adherence), cost per agent and owner, posture findings, and recent changes. Every metric should carry the version, every row an owner, and every number should drill down to OpenTelemetry traces. Opencontroller by Lyzr provides behavioural-deviation monitoring with automatic quarantine and kill, LLM judges on production traffic, Posture, per-agent cost attribution through the LLM Gateway and "Ask" chat over the estate. Pricing starts from $250K per deployment per year.

## Related

- [How to stop a rogue AI agent: kill switches, quarantine and runtime controls](https://opencontroller.lyzr.com/blog/how-to-stop-a-rogue-ai-agent.md)
- [How should you set severity levels for AI agent policies?](https://opencontroller.lyzr.com/blog/how-to-set-severity-levels-for-ai-agent-policies.md)
- [How do you export AI agent traces with OpenTelemetry?](https://opencontroller.lyzr.com/blog/export-ai-agent-traces-with-opentelemetry.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
