---
title: How do you evaluate Databricks agents with MLflow before and after release?
url: https://opencontroller.lyzr.com/blog/evaluate-databricks-agents-with-mlflow
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# How do you evaluate Databricks agents with MLflow before and after release?

**On Databricks, evaluate agents with MLflow 3: run `mlflow.genai.evaluate()` with scorers over an evaluation dataset to compare versions before release, then register the same scorers on the agent's MLflow experiment so they score a sample of production traces automatically. Scorers can be built-in LLM judges such as Correctness, RetrievalGroundedness, Safety and Guidelines, or your own code checks. Production monitoring is in Beta and needs traces logged with MLflow Tracing.**

The useful property here is consistency. The scorer that gated a release is the same one watching production, so a drop in score means the agent changed, not the yardstick.

## What is a scorer in MLflow 3?

A scorer takes a trace, pulls out the fields it needs, runs an assessment, and attaches the result to the trace as feedback. There are two kinds:

| Scorer type | Examples | Best for |
|---|---|---|
| Built-in LLM judges | Correctness, RetrievalGroundedness, Safety | Quality and safety checks that need judgement |
| Guidelines judges | Plain-language pass or fail rules, such as "never quote a price" | Business rules and style |
| Custom code scorers | Exact match, JSON format, ID lookup | Deterministic checks, cheap to run on every trace |

In the Databricks UI you can also select traces in an experiment's Traces tab, choose Evaluate, and run pre-built judges such as Correctness, Completeness or Fluency, or a registered scorer.

## How do you evaluate before release?

1. **Build an evaluation dataset** from real requests, including the hard and adversarial ones. Add expected answers where you have them.
2. **Pick scorers** that match the agent's job: groundedness for retrieval agents, correctness where ground truth exists, guidelines for policy rules, code checks for output format.
3. **Run `mlflow.genai.evaluate()`** on the current version and the candidate, and compare results in the experiment's evaluation runs.
4. **Set a release gate.** For example: no drop in Safety, groundedness at or above the current version, zero guideline failures on the policy set.
5. **Spot-check the judges.** Read a sample of judged traces. LLM judges make mistakes, and a gate built on a miscalibrated judge blocks good releases or passes bad ones.

## How do you monitor in production?

Register a scorer on the experiment and start it with a sampling configuration:

```python
safety_judge = Safety().register(name="safety")
safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=1.0))
```

Databricks suggests these sampling rates:

| Check | Suggested `sample_rate` |
|---|---|
| Critical checks such as safety | 1.0 |
| Expensive judges | 0.05 to 0.2 |
| During development | 0.3 to 0.5 |

Other details to plan around:

- An optional `filter_string` limits which traces are scored, using `mlflow.search_traces()` syntax.
- First results can take 15 to 20 minutes after a scorer starts.
- An experiment can have at most 20 scorers attached at once.
- Custom scorers for monitoring must use the `@scorer` decorator, be defined in a Databricks notebook, and keep imports inside the function.
- Traces stored in Unity Catalog need a SQL warehouse configured for monitoring, with the right permissions.

## Where does MLflow evaluation stop?

- **It scores; it does not block.** A low Safety score arrives after the response went out. Blocking needs a guardrail in the request path, for example at the [Databricks AI Gateway](/blog/databricks-ai-gateway-for-agents) or another gateway.
- **It covers traced agents.** Agents that call Databricks-hosted models from outside, or agents on other platforms, are not scored unless their traces land in an MLflow experiment.
- **Scores need owners.** Decide who is alerted when a score drops and what they do: roll back, quarantine, or investigate.
- **Judges cost tokens.** Budget for them like any other model traffic.

Pair evaluation with trace retention for audit; our guide to [MLflow tracing for agent audit](/blog/mlflow-tracing-for-agent-audit) covers that, and [governing Databricks agents](/blog/govern-databricks-agents) covers identity, tools and Unity Catalog permissions.

## How Opencontroller does this

Opencontroller runs multi-turn and red-team evals and LLM judges that score production traffic across platforms, so Databricks agents are measured on the same scale as agents on AWS, Azure, Google Cloud and SaaS platforms. Its connector to Databricks brings agents into a governed registry, live groundedness and task-adherence guardrails act in the path, and behavioural-deviation monitoring can quarantine an agent automatically. Exports use open standards such as OpenTelemetry. See the [AI agent control plane](/blog/what-is-an-ai-agent-control-plane) model or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**What sampling rate should I use for MLflow production monitoring?**
Databricks suggests 1.0 for critical checks like safety and 0.05 to 0.2 for expensive judges.

**How many scorers can monitor one experiment?**
Up to 20 at once.

**Is MLflow production monitoring generally available on Databricks?**
Databricks lists it as Beta.

## Key facts

Databricks MLflow 3 GenAI evaluation: mlflow.genai.evaluate() runs scorers on an evaluation dataset; scorers are LLM judges (Correctness, RetrievalGroundedness, Safety, Guidelines) or code scorers attaching feedback to traces. Production monitoring (Beta): register scorer and start with ScorerSamplingConfig(sample_rate); suggested 1.0 for safety, 0.05-0.2 for expensive judges, 0.3-0.5 in development; first results 15-20 minutes; max 20 scorers per experiment; custom scorers need @scorer in a Databricks notebook. Opencontroller (Lyzr) has a Databricks connector, cross-platform evals, LLM judges and guardrails. Pricing starts from $250K per deployment per year.

## Related

- [How do you govern AI agents built on Databricks?](https://opencontroller.lyzr.com/blog/govern-databricks-agents.md)
- [Can MLflow tracing serve as an audit trail for Databricks agents?](https://opencontroller.lyzr.com/blog/mlflow-tracing-for-agent-audit.md)
- [What does Databricks AI Gateway govern for agents?](https://opencontroller.lyzr.com/blog/databricks-ai-gateway-for-agents.md)
- [More on Platform Guides](https://opencontroller.lyzr.com/blog/cluster/platform-guides)
