---
title: Should you cap max tokens for AI agents at the gateway?
url: https://opencontroller.lyzr.com/blog/max-token-limits-for-ai-agents-at-the-gateway
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# Should you cap max tokens for AI agents at the gateway?

**Yes, but cap the right things. A gateway should enforce a per-agent ceiling on output tokens, on input context and on the number of model calls per task, because one runaway request or loop can cost more than a day of normal traffic. Set the caps from observed usage per agent version, not one global number, and watch evaluations for truncation after you tighten them.**

Agents are expensive in a different way from chat. A single user task can trigger ten or more model calls, each carrying a growing context window. A cap on one call does little if the agent is allowed to make unlimited calls. That is why token caps belong at the gateway, where every call from every agent passes through, and why they work best as a set of limits rather than one setting.

## What can you actually cap?

There are four distinct limits. Each catches a different failure.

| Cap | What it limits | Failure it catches | Trade-off |
|---|---|---|---|
| Output tokens per call | Length of one model response | Rambling answers, unbounded generation, reasoning blow-ups | Too low truncates valid answers mid-sentence or mid-JSON |
| Input context per call | Prompt plus history plus retrieved documents | Context bloat from long conversations or oversized RAG results | Too low drops facts the agent needs; quality falls quietly |
| Calls per task or session | Number of model calls in one run | Loops, retry storms, planner-executor ping-pong | Too low stops legitimate long tasks |
| Tokens or spend per agent per period | Total consumption over an hour or day | Slow leaks, traffic spikes, misuse of a shared key | Too low causes outages for real users |

Model APIs already expose the first limit. OpenAI's Chat Completions API takes `max_completion_tokens` and the Responses API takes `max_output_tokens`; for reasoning models, hidden reasoning tokens count against that limit. Anthropic's Messages API requires `max_tokens` on every request. Google's Gemini API uses `maxOutputTokens`. The problem is that each agent team sets these values in its own code, so there is no consistent ceiling across the estate. A gateway can clamp whatever the client sends to the maximum allowed for that agent.

## How do you choose the numbers?

Start from data, not guesses. A practical sequence:

1. **Log usage per agent and per version for two to four weeks.** Record input tokens, output tokens and calls per task.
2. **Find the 99th percentile for each.** Most agents have a long tail; a cap at p99 plus headroom blocks outliers without touching normal work.
3. **Set the output cap at roughly 1.5 to 2 times p99.** Reasoning models need more headroom because reasoning tokens vary with problem difficulty.
4. **Set a call-per-task cap.** Agent frameworks have their own versions: LangGraph's recursion limit defaults to 25 steps, and CrewAI agents have a `max_iter` setting. A gateway cap is the backstop when framework settings are missing or wrong.
5. **Set a period budget per agent.** Tie it to the agent's owner and cost centre so an alert reaches someone who can act.
6. **Run in monitor mode first.** Log what would have been blocked for a week before enforcing.
7. **Review after every new agent version.** A prompt change can double output length overnight.

See [how to stop an AI agent from looping](/blog/stop-an-ai-agent-from-looping) for the loop side of this problem and [reasoning tokens and AI agent cost](/blog/reasoning-tokens-and-ai-agent-cost) for why reasoning models need different headroom.

## How does truncation show up when caps are too tight?

Truncation rarely throws an error your users see. It shows up as quality loss. The model provider tells you it happened: OpenAI returns a `finish_reason` of `length`, and Anthropic returns a `stop_reason` of `max_tokens`. Most teams never look at these fields.

Watch for these signals after tightening caps:

- A rising share of responses ending with a length or max-tokens stop reason.
- Malformed JSON or tool calls, because structured output was cut off before the closing bracket.
- Lower scores from evaluations on completeness or task adherence.
- More retries from the agent, which can cost more than the tokens you saved.

The last point matters. A cap that is too tight can raise cost, because the agent retries or re-plans after a truncated answer. Track cost per completed task, not cost per call. [Measuring cost per task](/blog/measure-cost-per-task-for-ai-agents) explains how.

## Should caps differ by agent?

Yes. A summarisation agent and a code-generation agent have very different output profiles. A customer-facing agent with strict latency targets may need a lower output cap than an internal research agent. Set caps per agent and per version, and keep a sensible default for any agent without an explicit policy, so new or unregistered agents do not run uncapped.

A useful rule: the tighter the agent's risk tier, the tighter its call-per-task cap. An agent that can take actions in production systems should not be able to make hundreds of calls without someone noticing.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) sits in the path of model calls and gives routing, fallback and retry, caching, and cost attribution per agent and per version, so you can see each agent's token profile before you set limits. Policy as code lets you express limits per agent, and rogue-agent handling monitors behavioural deviation, alerts, and can quarantine or kill an agent whose consumption breaks pattern. Gateway overhead is measured in our [LLM gateway benchmark](/llm-gateway-benchmark) under stated test conditions. For the wider picture of where gateways fit, read [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Do max token caps reduce cost on their own?**
They cap the worst case. Average cost falls only if many responses were longer than needed. Pair caps with caching and routing for steady savings.

**Do reasoning tokens count against the output cap?**
On OpenAI reasoning models, yes: reasoning tokens count toward `max_completion_tokens` or `max_output_tokens`. Leave extra headroom for reasoning models.

**What is the safest default cap for unregistered agents?**
A low output cap, a low call-per-task cap and a small period budget. Unregistered agents should earn higher limits by registering with an owner.

## Key facts

Four token limits matter for AI agents: output tokens per call, input context per call, model calls per task, and tokens or spend per agent per period. OpenAI uses max_completion_tokens (Chat Completions) and max_output_tokens (Responses); reasoning tokens count toward these. Anthropic requires max_tokens and signals truncation with stop_reason max_tokens; OpenAI uses finish_reason length. LangGraph recursion limit defaults to 25. Set caps at p99 plus headroom per agent version, run in monitor mode first. Opencontroller LLM Gateway provides routing, fallback/retry, caching and cost attribution per agent and per version. Licensing starts from $250K per deployment per year.

## Related

- [How do you stop an AI agent from looping?](https://opencontroller.lyzr.com/blog/stop-an-ai-agent-from-looping.md)
- [How do reasoning tokens change the cost of an AI agent?](https://opencontroller.lyzr.com/blog/reasoning-tokens-and-ai-agent-cost.md)
- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
