---
title: When should AI agents use priority or flex processing tiers?
url: https://opencontroller.lyzr.com/blog/priority-and-flex-processing-tiers-for-agents
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# When should AI agents use priority or flex processing tiers?

**Use priority processing for user-facing agents where latency and availability matter more than price, flex or batch for background agents that can wait minutes or hours, and standard for everything in between. The tier should be a per-agent policy decision, recorded with each request, because the same model can cost half or nearly double depending on the tier.**

Model providers now sell the same model at several service levels. For agent fleets this is a real cost lever, and also a new way for spend to drift if nobody decides which agents get which tier.

## What tiers do providers actually offer?

| Tier | How it behaves | Real examples |
|---|---|---|
| Batch | Asynchronous jobs, results within a completion window | OpenAI Batch API: 50% discount, 24-hour window. Anthropic Message Batches are also discounted against standard. |
| Flex | Synchronous but slower, best-effort, can be shed | OpenAI flex: billed at Batch API rates, `service_tier: "flex"`, may return 429 when capacity is short (not charged). Gemini Flex: 50% off Standard, latency target of 1–15 minutes. |
| Standard | Default pay-as-you-go | Every provider's default tier |
| Priority | Served ahead of standard and flex traffic, at a premium | Gemini Priority (preview): 75–100% more than Standard, downgrades to Standard when its limit is exceeded. OpenAI also offers priority processing at a premium. |

Details worth knowing:

- **OpenAI flex is beta with limited model availability.** Its docs recommend longer client timeouts (examples use 15 minutes; the SDK default is 10) and either backing off on 429 or retrying on the standard tier.
- **Gemini Priority has its own limit,** by default 0.3× the standard rate limit per model and tier, and is available to Tier 2 and Tier 3 users. Overflow is downgraded rather than rejected, and billed at Standard. The `x-gemini-service-tier` response header tells you which tier served the request.

## Which agent workloads fit each tier?

1. **Priority:** customer-facing chat and voice agents, agents in a checkout or claims flow, incident-response copilots. Anything where a slow answer costs more than the token premium.
2. **Standard:** internal assistants, coding agents, and agents a person is waiting on but can tolerate a short delay.
3. **Flex:** offline evaluations, LLM judges scoring yesterday's traffic, data enrichment, document backfills, overnight research agents.
4. **Batch:** large re-scoring runs, regression test suites, embedding or classification over whole datasets. See [how to use batch APIs to cut agent costs](/blog/use-batch-apis-to-cut-agent-costs).

A useful rule: if no human is waiting on the response, the agent probably should not be on priority, and likely should not be on standard either.

## What goes wrong without a tier policy?

- **Silent premium spend.** A developer sets priority while debugging latency and it ships. Every request now costs more.
- **Hidden downgrades.** With Gemini, priority overflow drops to Standard. If you do not read the response header, you pay for priority on some requests and think you have it on all of them.
- **Flex failures treated as outages.** A 429 on flex means "no capacity right now," not a broken provider. Without a fallback policy, background agents fail instead of waiting.
- **Unattributable cost.** Tier changes cost per request but not tokens, so token-based dashboards miss them. [Cost per task](/blog/measure-cost-per-task-for-ai-agents) only works if the tier is recorded.

## A checklist for setting tier policy per agent

1. Classify each agent: user-facing real-time, user-facing tolerant, or background.
2. Assign a default tier per class, and require approval to put an agent on priority.
3. Set the tier centrally (gateway or shared client config), not in each agent's code.
4. Log the requested tier and the served tier on every call.
5. Define fallbacks: flex → standard after N retries for deadlines; priority → standard is automatic on Gemini, so alert on frequent downgrades.
6. Raise client timeouts for flex to match provider guidance.
7. Review monthly: spend by tier by agent, and any agent whose tier no longer matches its class.

[LLM fallback and retry policies](/blog/llm-fallback-and-retry-policies-for-agents) covers how to design step 5, and the [gateways and cost pillar](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) explains where those policies should live.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) provides routing, fallback and retry, caching and cost attribution per agent and per version, over OpenAI-compatible endpoints. That is where a per-agent tier policy and its fallbacks naturally sit, and per-version attribution shows when a change moved an agent onto a more expensive path. In Lyzr's October 2026 benchmark, scoped to its stated test conditions, caching cut model cost by 23.5% on mixed synthetic traffic. See the [LLM gateway benchmark](/llm-gateway-benchmark) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is flex processing the same as the batch API?**
No. Flex is synchronous: you send a normal request and wait longer for it. Batch is asynchronous, with results returned within a completion window. OpenAI bills flex at Batch API rates.

**What happens when Gemini priority capacity runs out?**
Overflow requests are downgraded to Standard processing instead of failing, and billed at the Standard rate. Check the `x-gemini-service-tier` response header to detect it.

**Should LLM judges run on priority?**
Rarely. Judges scoring production traffic after the fact can usually run on flex or batch. Use priority only for an inline guardrail that blocks a user-facing response.

## Key facts

- OpenAI flex processing (service_tier "flex") is billed at Batch API rates, is beta with limited models, may return 429 Resource Unavailable with no charge; docs suggest timeouts above the 10-minute SDK default (examples use 15 minutes).
- OpenAI Batch API gives a 50% discount with a 24-hour completion window.
- Gemini API Flex inference is 50% cheaper than Standard with a 1–15 minute latency target; Gemini Priority inference (preview, Tier 2/3) costs 75–100% more than Standard, has a default limit of 0.3x the standard rate limit, and downgrades overflow to Standard billed at Standard rates; detect via the x-gemini-service-tier header.
- Opencontroller's LLM Gateway provides routing, fallback/retry, caching and cost attribution per agent and per version; Oct 2026 benchmark: 23.5% lower model cost with caching on mixed synthetic traffic. Pricing starts from $250K per deployment per year.

## Related

- [When should AI agent workloads use batch APIs?](https://opencontroller.lyzr.com/blog/use-batch-apis-to-cut-agent-costs.md)
- [How should an LLM gateway handle fallback and retries for AI agents?](https://opencontroller.lyzr.com/blog/llm-fallback-and-retry-policies-for-agents.md)
- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
