---
title: How much does retrieved context add to AI agent cost?
url: https://opencontroller.lyzr.com/blog/rag-context-cost-for-ai-agents
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# How much does retrieved context add to AI agent cost?

**Retrieved context is often the largest share of an agent's input tokens, and it compounds when chunks stay in the conversation history. Multiply chunks per retrieval by tokens per chunk by the number of turns that carry them. Keeping retrieved text in history can multiply this cost several times over across a conversation.**

Teams usually budget for the model's answer. The bill is driven by what goes in. In retrieval-augmented agents, that is mostly documents.

## How do you calculate retrieval cost per conversation?

Use this formula for input tokens from retrieval:

`retrieval tokens = chunks per retrieval × tokens per chunk × retrievals × turns each chunk is resent`

The last factor is the one people miss. If retrieved chunks are appended to the message history, every later turn pays for them again.

## What does a worked example look like?

Assumptions, chosen to be round and illustrative rather than measured:

- 5 chunks per retrieval, 500 tokens each = 2,500 tokens per retrieval
- One retrieval per turn, 6 turns per conversation
- 100,000 conversations a month
- An illustrative input price of $3 per million tokens

| Pattern | Retrieval tokens per conversation | Monthly retrieval tokens | Monthly retrieval cost |
|---|---|---|---|
| Chunks dropped after each turn | 6 × 2,500 = 15,000 | 1.5 billion | $4,500 |
| Chunks kept in history | 2,500 × (1+2+3+4+5+6) = 52,500 | 5.25 billion | $15,750 |
| Kept in history, top 3 chunks after reranking | 1,500 × 21 = 31,500 | 3.15 billion | $9,450 |
| Dropped after each turn, top 3 chunks | 6 × 1,500 = 9,000 | 0.9 billion | $2,700 |

Same model, same questions. The cost spread is close to six times, purely from context handling. Substitute your own model price and traffic; the ratios hold.

This excludes the system prompt, tool definitions and the user's messages, which add more per turn. See [the token cost of MCP tool definitions](/blog/token-cost-of-mcp-tool-definitions) for that part.

## Which levers cut retrieval cost the most?

1. **Drop or summarise old context.** Keep the answer, not the source chunks, in history. Re-retrieve if the user returns to the topic. This is usually the largest saving. See [cutting the cost of long conversations](/blog/cut-cost-of-long-ai-agent-conversations).
2. **Rerank, then send fewer chunks.** Retrieve broadly, then rerank and send only the top few. Fewer, better chunks often improve groundedness too.
3. **Right-size chunks.** Very large chunks carry irrelevant text. Very small ones need more of them. Test chunk size against answer quality and token count together.
4. **Retrieve only when needed.** Let the agent decide whether a turn needs retrieval. Small-talk and clarifying turns do not.
5. **Cache stable prefixes.** Provider prompt caching discounts repeated prefixes such as the system prompt and tool definitions. Anthropic's [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and OpenAI's [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) both work on the leading part of the prompt, so put stable content first and retrieved chunks after it. See [provider prompt caching for agents](/blog/provider-prompt-caching-for-agents).
6. **Cache repeated answers.** For FAQ-style traffic, a gateway response cache can skip the model call entirely. Check freshness rules so cached answers do not outlive document updates.

## How do you know which agent is paying for retrieval?

Measure at the task level, not the account level. Tag each model call with the agent, its version and the task. Then report input tokens per task, split into system prompt, tools, retrieved context and conversation. When retrieved context dominates, the levers above apply. See [measuring cost per task for AI agents](/blog/measure-cost-per-task-for-ai-agents).

Track one quality metric alongside cost, such as groundedness. Cutting chunks that the answer depended on saves money and loses accuracy.

## How Opencontroller does this

The LLM Gateway in Opencontroller (Lyzr Gateway) provides routing, fallback and retry, caching, and cost attribution per agent and per version. In an October 2026 benchmark on mixed synthetic traffic, caching lowered model cost by 23.5% under the stated test conditions. Live groundedness guardrails show whether trimming context affects answer quality. For where the gateway fits, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) and the [gateway benchmark](/llm-gateway-benchmark). [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Does prompt caching help with retrieved chunks?**
Only partly. Provider caching works on repeated prompt prefixes. Retrieved chunks change per query, so put them after stable content to keep the prefix cacheable.

**How many chunks should an agent retrieve?**
There is no fixed number. Retrieve broadly, rerank, and send the smallest set that keeps groundedness scores stable in your evals.

**Is a bigger context window cheaper than retrieval?**
Usually not. Filling a large context window with whole documents sends far more tokens per call than targeted retrieval.

## Key facts

In retrieval-augmented agents, retrieved chunks are often the largest share of input tokens. Retrieval tokens = chunks × tokens per chunk × retrievals × turns resent; keeping chunks in history can multiply cost several times. Opencontroller's LLM Gateway provides caching and cost attribution per agent and version; caching lowered model cost by 23.5% on mixed synthetic traffic in an October 2026 benchmark. Licensing starts from $250K per deployment per year.

## Related

- [Why do long AI agent sessions get so expensive and how do you cut the cost?](https://opencontroller.lyzr.com/blog/cut-cost-of-long-ai-agent-conversations.md)
- [How does provider prompt caching cut AI agent costs?](https://opencontroller.lyzr.com/blog/provider-prompt-caching-for-agents.md)
- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
