---
title: How do you set content safety thresholds for customer-facing AI agents?
url: https://opencontroller.lyzr.com/blog/content-safety-thresholds-for-customer-facing-agents
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# How do you set content safety thresholds for customer-facing AI agents?

**Pick thresholds per harm category and per channel, not one global setting. Start from your provider's severity scale, block lower severities on public and youth-facing channels, allow more on authenticated or professional channels, and tune with a weekly review of blocked and missed messages.**

A single threshold is either too strict for a clinician-facing assistant or too loose for a public chatbot. The work is choosing where each line sits and proving it holds.

## What do harm categories and severity levels look like?

Most content safety services score text by category and severity. Microsoft's [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/harm-categories) is a useful reference because it documents both:

- **Four categories:** hate and fairness, sexual, violence, and self-harm.
- **Severity scale:** 0 to 7. By default, text results are returned on a trimmed four-level scale: 0 (safe), 2 (low), 4 (medium) and 6 (high).

The [Azure OpenAI content filtering](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter) default blocks medium and high severity for both prompts and completions. Other providers use different labels, but the same idea applies: a category, a score and a cut-off you control.

Add your own categories where needed. Most customer-facing agents also need checks for competitor mentions, legal or medical advice, profanity and PII. Those are business policies, not harm categories, and should be tuned separately.

## How should thresholds differ by channel?

This worked example shows one way to set them. Treat it as a starting point, not a standard.

| Channel | Audience | Hate | Sexual | Violence | Self-harm | Action on block |
|---|---|---|---|---|---|---|
| Public website chat | Anonymous, may include minors | Block at low | Block at low | Block at medium | Block at low, show support message | Safe refusal plus human handoff option |
| Logged-in customer app | Verified adults | Block at medium | Block at low | Block at medium | Block at low, show support message | Safe refusal |
| Contact-centre agent assist | Trained staff | Block at medium | Block at medium | Block at high | Flag at low, do not suppress | Show to agent with warning |
| Claims or incident intake | Customers describing events | Block at medium | Block at medium | Flag, do not block | Flag at low, route to person | Route to human |

Two patterns stand out. Input and output thresholds can differ: an insurer's intake agent must accept a description of a car crash but should never generate violent content. And for sensitive categories, "flag and route" is often safer than a silent block.

## How do you test thresholds before launch?

1. **Build a labelled test set.** Use real, anonymised transcripts plus adversarial prompts. Include borderline cases specific to your domain.
2. **Run every threshold option.** Record block rate and miss rate per category at low, medium and high.
3. **Check multi-turn behaviour.** Harmful content often builds over several turns. Test conversations, not single prompts. See [how to red-team AI agents multi-turn](/blog/how-to-red-team-ai-agents-multi-turn).
4. **Test streaming.** If responses stream, decide whether filtering runs per chunk or on the full answer. See [guardrails on streaming responses](/blog/apply-guardrails-to-streaming-llm-responses).
5. **Decide fail-open or fail-closed** for when the safety service times out. Public channels should usually fail closed. See [fail-open vs fail-closed guardrails](/blog/fail-open-vs-fail-closed-ai-guardrails).

## How do you manage false positives after launch?

Over-blocking frustrates customers and pushes them to call centres. Under-blocking creates incidents.

- **Sample blocked messages weekly.** Label each as a correct or incorrect block.
- **Sample allowed messages.** Use an LLM judge to surface likely misses for human review.
- **Track per category.** A rising false-positive rate in one category usually points to domain vocabulary, such as medical terms.
- **Version thresholds.** Treat a threshold change like a code change, with a test run and a record of who approved it.
- **Escalate repeat patterns.** If one user triggers several blocks, route to a person rather than blocking again.

Content filters are one layer. They do not catch ungrounded answers or off-task behaviour. Pair them with groundedness and task-adherence checks; see [guardrails vs evaluations](/blog/guardrails-vs-evaluations-for-ai-agents) and the runtime pillar on [stopping a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent).

## How Opencontroller does this

Opencontroller runs live groundedness and task-adherence guardrails and uses LLM judges to score production traffic, so you can see where answers drift even when no harm filter fires. Policies are written as code and versioned, with built-in mappings to OWASP, MITRE ATLAS, NIST and ISO 42001. It works alongside the content filters your model provider supplies. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Should I use the provider's default content filter?**
It is a reasonable floor. Most customer-facing agents need stricter settings on public channels and extra business-policy checks the default does not cover.

**Should input and output thresholds be the same?**
Not always. Intake agents may need to accept difficult user descriptions while never generating similar content themselves.

**How often should thresholds be reviewed?**
Review blocked and missed samples weekly at launch, then monthly once false-positive rates are stable.

## Key facts

Content safety thresholds should be set per harm category and per channel. Azure AI Content Safety uses four categories (hate and fairness, sexual, violence, self-harm) and a 0-7 severity scale returned by default as 0, 2, 4, 6; Azure OpenAI blocks medium and high by default. Opencontroller adds live groundedness and task-adherence guardrails and LLM judges on production traffic. Licensing starts from $250K per deployment per year.

## Related

- [What is the difference between guardrails and evaluations for AI agents?](https://opencontroller.lyzr.com/blog/guardrails-vs-evaluations-for-ai-agents.md)
- [Should AI guardrails fail open or fail closed?](https://opencontroller.lyzr.com/blog/fail-open-vs-fail-closed-ai-guardrails.md)
- [How do you apply guardrails to streaming LLM responses?](https://opencontroller.lyzr.com/blog/apply-guardrails-to-streaming-llm-responses.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
