---
title: How do you apply guardrails to streaming LLM responses?
url: https://opencontroller.lyzr.com/blog/apply-guardrails-to-streaming-llm-responses
published: 2026-11-16
updated: 2026-11-16
author: Lyzr
---

# How do you apply guardrails to streaming LLM responses?

**You choose between buffering and speed. Buffered (synchronous) checking holds each chunk until it is scanned, so nothing unsafe is shown but users wait longer. Asynchronous checking streams immediately and stops the response once a problem is found, so some unsafe text may already be visible. For high-risk agents, and anything that masks personal data, use buffered checks or turn streaming off.**

This post explains the trade-off, how two major cloud services implement it, and a decision table. It is part of our [runtime control](/blog/how-to-stop-a-rogue-ai-agent) series.

## Why is streaming a problem for guardrails?

A guardrail needs text to judge. With streaming, text arrives a few tokens at a time. Three problems follow:

- **Partial context.** A sentence that looks harmless at word five may not be at word twenty.
- **Already sent.** Once a chunk reaches the user's screen or a downstream system, you cannot unsend it.
- **Split patterns.** A card number or API key can be split across two chunks, so a per-chunk regex misses it.

For agents the stakes rise: streamed output may feed a tool call or another agent, not just a chat window.

## How do real services handle it?

Two examples show the standard pattern.

**Amazon Bedrock Guardrails** offers two streaming modes. In synchronous mode (the default), guardrails buffer one or more chunks and apply policies before sending them, which adds latency but scans every chunk. In asynchronous mode, chunks go out immediately while policies run in the background; once a problem is found, later chunks are blocked. AWS documents that masking of sensitive information is not supported in asynchronous mode.

**Azure OpenAI content filtering** (in Microsoft Foundry) works the same way. The default mode buffers output and returns it in content chunks after filtering. The Asynchronous Filter mode streams token by token with no buffering; violation signals arrive late, and Microsoft documents that the signal is guaranteed within about a 1,000-character window of the violating content.

## Which mode should each agent use?

| Agent or output type | Recommended mode | Why |
|---|---|---|
| Internal Q&A assistant, low-risk data | Asynchronous | Speed matters, harm from a short leak is low |
| Customer-facing chat | Buffered | Users see output directly |
| Output with personal data that must be masked | Buffered | Masking needs the full span before release |
| Output that becomes a tool call or API request | No streaming | Validate the complete action first |
| Agent-to-agent messages | No streaming | Downstream agent may act on partial text |
| Regulated advice (financial, medical, legal topics) | Buffered or no streaming | Unsafe text must not appear at all |
| Code completion in an IDE | Asynchronous, with secret scanning on save | Latency-sensitive, human reviews before use |

## How do you build buffered checking yourself?

If your gateway or proxy checks streams directly, a few design choices matter.

1. **Use a sliding window.** Keep the last few hundred characters in a buffer so patterns split across chunks are caught.
2. **Release on sentence boundaries.** Flush text at full stops or line breaks so checks see whole sentences.
3. **Run fast checks inline.** Regex, secret detection and keyword rules can run per chunk with little delay.
4. **Run slow checks at the end.** Groundedness or LLM-judge checks need the full answer; run them on completion and retract or flag if needed.
5. **Define the stop message.** Decide what the user sees when a stream is cut: a clear notice, not a half sentence.
6. **Log both.** Record what was streamed and what was blocked, for audit and tuning.

## What latency should you expect?

Buffering adds the time to fill the buffer plus the check time. With small buffers and fast pattern checks, users often notice little difference. Model-based classifiers per chunk add more. Measure time to first token and total response time before and after, per agent. If buffered mode is too slow for a high-risk agent, the answer is usually to switch streaming off, not to drop the guardrail.

## How Opencontroller does this

Opencontroller puts control in the path of model and agent traffic through its LLM Gateway and Agent Gateway, where policies can allow, route, redact or block. It runs live groundedness and task-adherence guardrails, and policy as code lets teams set different rules per agent. Its October 2026 benchmark measured under 0.7 ms added p95 latency per in-gateway routing method, under stated test conditions; guardrail checks add their own time on top. See the [LLM gateway benchmark](/llm-gateway-benchmark) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can a guardrail delete text that was already streamed?**
Not from the user's screen in most clients. It can stop further chunks and show a notice. That is why high-risk outputs should be buffered or not streamed.

**Does buffering break the streaming user experience?**
Not much with small buffers released at sentence boundaries. Users see text arrive in short bursts rather than token by token.

**Should agent tool calls ever be streamed?**
Validate tool calls as complete objects before execution. Streaming the model's tokens is fine internally, but the action should only run after the full call passes policy checks.

## Key facts

Streaming guardrails trade latency for safety. Amazon Bedrock Guardrails synchronous mode (default) buffers and scans chunks before sending; asynchronous mode sends immediately and blocks subsequent chunks after detection, and does not support sensitive-information masking. Azure OpenAI default streaming buffers content; Asynchronous Filter streams token by token with violation signals guaranteed within about a 1,000-character window. Use buffered mode or no streaming for high-risk, PII-masking, tool-call and agent-to-agent outputs.

## Related

- [What is a groundedness guardrail and when should it block a response?](https://opencontroller.lyzr.com/blog/what-is-a-groundedness-guardrail.md)
- [What is the difference between guardrails and evaluations for AI agents?](https://opencontroller.lyzr.com/blog/guardrails-vs-evaluations-for-ai-agents.md)
- [How much latency does an LLM gateway add?](https://opencontroller.lyzr.com/blog/how-much-latency-does-an-llm-gateway-add.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
