---
title: How much can LLM caching save, and when is it unsafe?
url: https://opencontroller.lyzr.com/blog/llm-caching-savings-and-risks
published: 2026-11-13
updated: 2026-11-13
author: Lyzr
---

# How much can LLM caching save, and when is it unsafe?

**LLM caching can cut model spend noticeably when agents send repeated or overlapping prompts, but the saving depends on how much of your traffic actually repeats. Provider prompt caching discounts repeated prompt prefixes, exact-match caching skips a model call entirely for identical requests, and semantic caching reuses answers for similar requests. Caching becomes unsafe when the cache key ignores who is asking, what they are allowed to see, or how fresh the answer must be.**

For AI agents the question is not "should we cache?" but "which layer, for which agent, with which key?". This post compares the three layers and lists the cases where you should turn caching off.

## What are the three kinds of LLM caching?

| Type | Where it runs | What is reused | Saving | Main risk |
|---|---|---|---|---|
| Provider prompt caching | At the model provider | The processed prompt prefix (system prompt, tools, long context) | Lower input-token price on cache hits; the model still generates a fresh answer | Low; output is still generated per request |
| Exact-match response caching | In your gateway | The full response for a byte-identical request | The whole model call is skipped | Serving a response to someone who should not see it |
| Semantic response caching | In your gateway | The response for a request judged "similar enough" | The whole model call is skipped | Wrong answer for a subtly different question, plus the same access risk |

Provider caching is the safest because the model still answers each request. Response caching saves more per hit but takes on the responsibility of deciding whether an old answer is right for a new caller.

## How much does provider prompt caching save?

It depends on the provider's published pricing. Two examples from current vendor documentation:

- **Anthropic:** cache reads are billed at 10% of the base input-token price. Cache writes cost more than normal input: 1.25 times base for the 5-minute cache and 2 times base for the 1-hour cache. Caching pays off when the same prefix is read several times within the cache lifetime.
- **OpenAI:** prompt caching applies automatically to prompts of 1,024 tokens or more, with cached input tokens billed at a discount that varies by model.

Agents are good candidates because they resend long, stable prefixes: system prompts, tool definitions and retrieved policy documents. Put stable content first and variable content last so the prefix stays identical.

## How much does response caching save?

There is no universal number. Savings equal the share of requests that hit the cache times the cost of the calls you skip, minus the cost of running the cache. In our own October 2026 benchmark, caching produced 23.5% lower model cost on mixed synthetic traffic. That figure is scoped to the stated test conditions; real traffic with more unique requests will save less, and highly repetitive traffic such as FAQ-style agents may save more. See the [LLM gateway benchmark](/llm-gateway-benchmark) for the method.

Measure your own hit rate before turning caching on broadly. Run the cache in "shadow" mode, record would-be hits, and estimate savings per agent. Then compare with your cost attribution data; see [how to attribute LLM costs to AI agents](/blog/how-to-attribute-llm-costs-to-ai-agents).

## When is LLM caching unsafe?

Turn response caching off, or narrow its key, in these cases:

1. **Answers depend on the caller's permissions.** If the agent retrieves documents the user is entitled to, a cached answer can leak content to a user without access. Include the user or permission scope in the cache key, or do not cache.
2. **Multi-tenant traffic.** Never share response caches across tenants or business units with different data boundaries.
3. **Answers depend on live data.** Prices, balances, stock levels and ticket status go stale. Set short time-to-live values or skip caching.
4. **Tool-calling steps.** Caching a model response that decides to call a tool can replay an action. Cache only final, read-only answers.
5. **Regulated decisions.** If you must show how each decision was made, a cache hit needs to record the original request it reused.
6. **Semantic near-misses.** "Cancel my order" and "can I cancel my order?" may look similar but need different handling. Set a strict similarity threshold and test it against real traffic.
7. **Timing side channels.** Research on prompt caching in commercial APIs has shown that cache hits can be detected from response timing, which can reveal that another user sent a similar prompt when caches are shared. Prefer per-organisation cache isolation.

## How should you roll out caching for agents?

1. Enable provider prompt caching first; reorder prompts so stable parts come first.
2. Classify agents: read-only and non-personal answers are candidates for response caching; permissioned, live-data and action-taking agents are not.
3. Shadow-test hit rates per agent for a week.
4. Turn on exact-match caching with a short time-to-live for the candidates.
5. Add semantic caching only where you have tested thresholds against real queries.
6. Report savings per agent and watch quality scores for regressions.

Caching is one of several cost controls; see [where AI agent costs leak](/blog/where-ai-agent-costs-leak) and the [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) overview.

## How Opencontroller does this

The Lyzr Gateway, Opencontroller's LLM Gateway, provides routing, fallback and retry, caching and cost attribution per agent and per version through OpenAI-compatible endpoints. Because cost is attributed per agent version, you can see whether caching changed spend for a specific agent, and LLM judges scoring production traffic help show whether quality moved. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is provider prompt caching safe for sensitive data?**
It is lower risk than response caching because the model still generates each answer. Check the provider's documentation on cache isolation and retention for your account type.

**Does caching change model output quality?**
Provider prefix caching should not. Response caching can, because a reused answer may not fit the new question. Track quality scores before and after.

**Should every agent use semantic caching?**
No. Use it only for read-only, non-personal answers where you have tested the similarity threshold on real traffic.

## Key facts

Three kinds of LLM caching: provider prompt caching (discounted repeated prompt prefixes; model still generates each answer), exact-match response caching and semantic response caching (gateway skips the model call). Anthropic bills cache reads at 10% of base input price; cache writes cost 1.25x base (5-minute) or 2x (1-hour). OpenAI applies prompt caching automatically to prompts of 1,024+ tokens. Response caching is unsafe when answers depend on caller permissions, across tenants, with live data, for tool-calling steps, for regulated decisions without trace, for semantic near-misses, and where shared caches expose timing side channels. Opencontroller's Lyzr Gateway benchmark (Oct 2026, scoped to test conditions): 23.5% lower model cost with caching on mixed synthetic traffic; under 0.7 ms added p95 per routing method; 5,134 req/s on one 4-vCPU gateway.

## Related

- [Where do AI agent costs usually leak?](https://opencontroller.lyzr.com/blog/where-ai-agent-costs-leak.md)
- [How do you attribute LLM costs to individual AI agents?](https://opencontroller.lyzr.com/blog/how-to-attribute-llm-costs-to-ai-agents.md)
- [How do you test whether a cheaper model can replace the one an AI agent uses?](https://opencontroller.lyzr.com/blog/test-a-cheaper-model-before-switching-an-agent.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
