---
title: When should AI agent workloads use batch APIs?
url: https://opencontroller.lyzr.com/blog/use-batch-apis-to-cut-agent-costs
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# When should AI agent workloads use batch APIs?

**Use batch APIs for agent work that does not need an answer within seconds: evaluations, backfills, bulk classification, enrichment and nightly reports. OpenAI's Batch API and Anthropic's Message Batches API both bill at 50% of standard prices in exchange for results within 24 hours. Interactive, multi-step agent loops do not qualify, because each step depends on the last.**

Batch pricing is one of the few cost levers that needs no model change and no quality trade-off. The catch is latency, and knowing which parts of an agent programme can wait. This post is part of the gateways and cost series; the pillar is [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## What do the main batch APIs offer?

| Feature | OpenAI Batch API | Anthropic Message Batches API |
|---|---|---|
| Price | 50% less than synchronous calls | 50% of standard API prices |
| Completion window | Must finish within 24 hours | Expires if not complete within 24 hours; most finish in under 1 hour |
| Batch size limit | 50,000 requests or a 200 MB input file | 100,000 requests or 256 MB |
| Result retention | Output file in your account storage | Results downloadable for 29 days |
| Prompt caching | See provider docs | Discounts stack; cache hits are best-effort |

Sources: the [OpenAI Batch API guide](https://developers.openai.com/api/docs/guides/batch) and Anthropic's [batch processing documentation](https://platform.claude.com/docs/build-with-claude/batch-processing). Check both before you plan around them; limits change.

## Which agent workloads qualify?

A workload fits batch when three things are true: each request is independent, nobody is waiting on the result, and a delay of up to 24 hours is acceptable.

| Workload | Batch fit | Why |
|---|---|---|
| Offline evaluations and regression suites | Strong | Thousands of independent test cases; results reviewed later |
| LLM-judge scoring of yesterday's traffic | Strong | Retrospective by design |
| Backfills (re-tagging, re-summarising history) | Strong | Large, independent, no user waiting |
| Bulk classification and extraction | Strong | Each document stands alone |
| Embedding refreshes | Strong (where supported) | Independent and large |
| Nightly reports and digests | Good | Deadline is hours away |
| Red-team prompt sweeps | Good | Independent attacks scored afterwards |
| Customer-facing chat agents | No | A user is waiting |
| Multi-step tool-using agent tasks | No | Each step needs the previous output |
| Real-time guardrails | No | Must act in the request path |

## Can part of an agent use batch?

Often, yes. Split the agent's work into the interactive loop and the work around it.

1. **Pre-compute.** Classify or summarise incoming documents overnight, so the live agent reads results instead of generating them.
2. **Post-process.** Score, label or audit completed tasks in a nightly batch.
3. **Fan-out stages.** If an agent plans a task and then runs hundreds of independent sub-steps that nobody needs immediately, submit those sub-steps as a batch and resume when results arrive.

## How much can you save?

The arithmetic is simple. If 30% of an agent programme's token spend is batch-eligible, moving it to batch cuts total model spend by about 15%. To find your eligible share:

1. Tag every model call with the agent, the workload type and whether a user was waiting.
2. Sum spend by workload type for a month.
3. Mark each type as batch-eligible using the table above.
4. Multiply eligible spend by 0.5 for the saving.

This works only if costs are attributed per agent and workload. See [how to measure cost per task for AI agents](/blog/measure-cost-per-task-for-ai-agents) and [how to forecast AI agent spend](/blog/forecast-ai-agent-spend).

## What are the risks?

- **Partial failures.** Some requests in a batch can fail. Build retry for failed items, not the whole batch.
- **Stale inputs.** A batch submitted at midnight uses midnight's data.
- **Governance gaps.** Batch jobs often bypass the gateway that real-time traffic uses, so they escape cost attribution and data policies. Route them through the same controls, or log them with the same agent and version tags.
- **Data handling.** Batch inputs and outputs are stored files. Apply the same retention and access rules you apply to logs.

## How Opencontroller does this

The Lyzr Gateway in Opencontroller attributes cost per agent and per version, which is the data you need to find batch-eligible spend. Opencontroller runs multi-turn and red-team evaluations and LLM judges that score production traffic; these are the kinds of workloads that suit batch processing at the provider level. Opencontroller does not itself change provider pricing. [Book a demo](https://www.lyzr.ai/book-demo/) or see how to [test a cheaper model before switching an agent](/blog/test-a-cheaper-model-before-switching-an-agent).

## Frequently asked questions

**How much cheaper are batch APIs?**
OpenAI and Anthropic both price batch requests at 50% of their standard synchronous prices, with results returned within 24 hours.

**Can a tool-using AI agent run on a batch API?**
Not its interactive loop, because each step depends on the previous one. Its independent pre-processing, scoring and backfill work often can.

**What agent workloads are best for batch?**
Offline evaluations, LLM-judge scoring of past traffic, backfills, bulk classification, embedding refreshes and nightly reports.

## Key facts

OpenAI Batch API: 50% less than synchronous calls, 24-hour completion window, up to 50,000 requests or 200 MB per batch. Anthropic Message Batches API: 50% of standard prices, up to 100,000 requests or 256 MB, most batches finish in under 1 hour, expire after 24 hours, results downloadable for 29 days, discounts stack with prompt caching. Batch suits evaluations, LLM-judge scoring, backfills, bulk classification and nightly reports; not interactive or multi-step tool-using loops. Opencontroller's Lyzr Gateway attributes cost per agent and version to size batch-eligible spend.

## Related

- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [How do you forecast AI agent spend for next year's budget?](https://opencontroller.lyzr.com/blog/forecast-ai-agent-spend.md)
- [How do you test whether a cheaper model can replace the one an AI agent uses?](https://opencontroller.lyzr.com/blog/test-a-cheaper-model-before-switching-an-agent.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
