---
title: What does Amazon Bedrock AgentCore Evaluations measure?
url: https://opencontroller.lyzr.com/blog/amazon-bedrock-agentcore-evaluations
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# What does Amazon Bedrock AgentCore Evaluations measure?

**AgentCore Evaluations scores AI agents at three levels: whole sessions (did the agent reach the user's goal), individual traces (was the response correct, faithful, helpful, safe), and tool calls (did it pick the right tool with the right parameters). It runs online against a sample of production traces or on demand from CI, using 13 built-in evaluators plus your own LLM-based or Lambda code evaluators. It became generally available on 31 March 2026.**

For AWS teams this answers the question "how do we know the agent is still good after go-live?" without building a judging pipeline from scratch. It covers agents instrumented for AgentCore; anything outside that path needs another approach.

## What do the built-in evaluators cover?

AWS describes 13 built-in evaluators across response quality, safety, task completion and tool use. Examples by level:

| Level | Example evaluators | Question answered |
|---|---|---|
| Session | Goal Success Rate | Did the conversation achieve what the user wanted? |
| Trace | Correctness, Faithfulness, Helpfulness, Instruction Following, Response Relevance | Was this response right, grounded and on task? |
| Trace (safety) | Harmfulness, Stereotyping, Refusal | Did the response cross safety lines, or refuse when it should not? |
| Tool | Tool Selection Accuracy, Tool Parameter Accuracy | Did the agent call the right tool with sensible arguments? |

Built-in evaluators use fixed evaluator models and prompt templates. You reference them by ID, such as `Builtin.Helpfulness`, and cannot modify them.

## How do online and on-demand evaluation differ?

| Mode | Use | How it runs | Where results go |
|---|---|---|---|
| Online | Production monitoring | Sampling rules score a share of live traces, no code change | AgentCore Observability dashboard on CloudWatch, plus a CloudWatch log group in JSON |
| On-demand | Development and CI/CD | API call on selected sessions, traces or spans | API response, up to 10 evaluations per call |

Ground truth support lets you compare against reference answers, session-level behavioural assertions and expected tool sequences. That is what makes on-demand runs useful as regression tests.

## How do custom evaluators work?

- **LLM-based:** your prompt, criteria, scoring schema and choice of judge model.
- **Code-based:** Python or JavaScript in an AWS Lambda function, for deterministic checks such as format, policy keywords or a lookup against a system of record.

Custom evaluators are private to your account and controlled with IAM resource-based policies. AWS sets a quota of up to 1,000 evaluation configurations per Region per account.

## What do you need in place first?

1. **Instrumentation.** Evaluations read traces. AgentCore documents integration with Strands and LangGraph through OpenTelemetry and OpenInference instrumentation.
2. **A supported Region.** At GA AWS listed nine: N. Virginia, Ohio, Oregon, Mumbai, Singapore, Sydney, Tokyo, Frankfurt and Ireland.
3. **A sampling budget.** Each judged trace costs model tokens. Start with a low sampling rate on high-volume agents and raise it for high-risk ones. If evaluations throttle, AWS advises lowering the rate.
4. **Thresholds and owners.** A score with no alert threshold and no owner is a dashboard nobody acts on.

## Where does AgentCore Evaluations stop?

- **Scope.** It scores agents whose traces reach AgentCore. Agents on Microsoft Foundry, Vertex AI, Salesforce or a developer laptop are outside it.
- **Judging is not enforcement.** A low faithfulness score tells you after the fact. Blocking a bad response needs a guardrail or policy in the request path. See [guardrails vs evaluations](/blog/guardrails-vs-evaluations-for-ai-agents).
- **Judge quality.** LLM judges can be wrong. Spot-check a sample of judged traces with people, especially before using scores for release gates.
- **Evidence format.** Results sit in CloudWatch. If audit evidence must cover every platform, you need to export and normalise them.

For the wider approach to scoring live traffic, read [using LLM judges on production agent traffic](/blog/use-llm-judges-on-production-agent-traffic). For what changes when teams move from Bedrock Agents, see [Bedrock Agents vs AgentCore governance](/blog/bedrock-agents-vs-agentcore-governance), and for trace retention see [AgentCore Observability for audit](/blog/agentcore-observability-for-agent-audit).

## How Opencontroller does this

Opencontroller runs multi-turn and red-team evals and LLM judges that score production traffic across platforms, not only AWS, and adds live groundedness and task-adherence guardrails in the path. Its runtime adapter for Bedrock AgentCore brings AgentCore agents into the same registry and policy model as agents on Vertex AI, Microsoft Foundry and third-party platforms, and behavioural-deviation monitoring can quarantine an agent automatically. See the [AI agent control plane](/blog/what-is-an-ai-agent-control-plane) model or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**When did AgentCore Evaluations become generally available?**
AWS announced general availability on 31 March 2026, after a preview launched at re:Invent 2025.

**Can I change a built-in evaluator's prompt?**
No. Built-in evaluator configurations are fixed. Create a custom evaluator instead.

**Can AgentCore Evaluations run in CI?**
Yes. On-demand evaluation is an API for development and CI/CD regression tests.

## Key facts

Amazon Bedrock AgentCore Evaluations: GA 31 March 2026 (preview at re:Invent 2025); 13 built-in evaluators across response quality, safety, task completion, tool use, at session (Goal Success Rate), trace (Correctness, Faithfulness, Helpfulness, Harmfulness, etc.) and tool level (Tool Selection Accuracy, Tool Parameter Accuracy). Online evaluation samples live traces; results in AgentCore Observability on CloudWatch and a CloudWatch log group. On-demand API for CI, up to 10 evaluations per call. Custom LLM-based and Lambda code-based evaluators (Python/JavaScript). Up to 1,000 evaluation configurations per Region per account; 9 Regions at GA. Opencontroller (Lyzr) runs evals and LLM judges across platforms and has a Bedrock AgentCore runtime adapter. Pricing starts from $250K per deployment per year.

## Related

- [How do you use LLM judges to score production AI agent traffic?](https://opencontroller.lyzr.com/blog/use-llm-judges-on-production-agent-traffic.md)
- [How do you use AgentCore Observability for AI agent audit trails?](https://opencontroller.lyzr.com/blog/agentcore-observability-for-agent-audit.md)
- [How do you govern AI agents built on Amazon Bedrock AgentCore?](https://opencontroller.lyzr.com/blog/govern-amazon-bedrock-agentcore-agents.md)
- [More on Platform Guides](https://opencontroller.lyzr.com/blog/cluster/platform-guides)
