---
title: When is self-hosting an LLM cheaper than an API for AI agents?
url: https://opencontroller.lyzr.com/blog/self-hosted-llm-vs-api-cost-for-agents
published: 2026-11-16
updated: 2026-11-16
author: Lyzr
---

# When is self-hosting an LLM cheaper than an API for AI agents?

**Self-hosting an open model is cheaper than a pay-per-token API only when your GPUs stay busy. The break-even depends on sustained utilisation, not on list prices: a GPU you pay for around the clock costs the same whether it serves one request or a million. Steady, high-volume, latency-tolerant agent workloads can cross the line; spiky or low-volume workloads usually do not.**

The question comes up as soon as agent traffic grows. Token invoices are visible and rising. Open-weight models look free. They are not, and the comparison needs the full cost on both sides.

## What costs go into each option?

| Cost component | Pay-per-token API | Self-hosted open model |
|---|---|---|
| Inference compute | Included in token price | GPU instances or hardware, paid per hour whether used or not |
| Idle capacity | None; you pay per call | Full cost; nights, weekends and quiet hours still billed |
| Peak capacity | Provider absorbs it (within rate limits) | You provision for peak or accept queuing |
| Engineering and operations | Integration only | Serving stack, autoscaling, upgrades, on-call |
| Model updates | Provider ships new versions | You evaluate, re-deploy and re-test |
| Security and compliance | Vendor assessment, data terms | Your own patching, isolation, access control |
| Evaluation | Needed either way | Needed either way, plus regression testing on every model change |

The two lines teams forget are idle capacity and operations. Both are fixed costs. They only shrink per token when volume rises.

## How do you work out the break-even?

Use one formula and your own numbers.

1. **Measure demand.** Tokens per day for the agents you would move, split into input and output, and the peak-to-average ratio.
2. **Measure throughput.** Benchmark the candidate model on your target GPU with a serving engine such as [vLLM](https://docs.vllm.ai/). Record sustained tokens per second at your latency target, not peak marketing figures.
3. **Size the fleet.** GPUs needed = peak tokens per second ÷ tokens per second per GPU, plus redundancy.
4. **Compute the self-hosted cost per million tokens.** (GPU hourly cost × GPUs × hours per month + monthly ops cost) ÷ (tokens served per month ÷ 1,000,000).
5. **Compare** with the API's blended price per million tokens for the same input/output mix, after any caching and batch discounts.

A worked illustration with round, hypothetical inputs: if a fleet costs 20,000 per month all-in and serves 2,000 million tokens, that is 10 per million tokens. If real traffic is only 500 million tokens, the same fleet costs 40 per million. Nothing changed except utilisation.

## When does self-hosting tend to win?

- **Steady, predictable volume.** Back-office agents that process documents all day keep GPUs busy.
- **Latency-tolerant work.** Batch jobs can queue, so you size for average rather than peak.
- **Smaller models that are good enough.** A well-evaluated small model on fewer GPUs can serve classification, extraction or routing steps.
- **Data residency or network constraints.** Sometimes the decision is not cost; the model must run inside your boundary.

## When does an API tend to win?

- **Spiky or growing traffic** where you cannot predict peak.
- **Frontier reasoning tasks** where open models do not meet your quality bar on evals.
- **Small teams** without people to run a GPU serving stack.
- **Workloads that benefit from provider features** such as prompt caching and batch pricing. See [provider prompt caching for agents](/blog/provider-prompt-caching-for-agents).

Many estates end up hybrid: high-volume, simple steps on a self-hosted model; complex reasoning on an API.

## What governance do you need either way?

Moving a model in-house does not remove the need for control. You still need:

- **Per-agent cost attribution**, including the amortised GPU cost, so internal models are not treated as free.
- **One endpoint for both.** An [OpenAI-compatible endpoint](/blog/what-is-an-openai-compatible-endpoint) in front of internal and external models lets agents switch without code changes.
- **Evaluations before switching.** Test the cheaper model on real tasks first.
- **Fallback.** If the internal fleet is saturated, route overflow to an API rather than failing.

Our pillar on [LLM gateways, agent gateways and control planes](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) covers where routing decisions sit.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) sits in front of external and internal models with OpenAI-compatible endpoints, routing, fallback and retry, caching, and cost attribution per agent and per version. Multi-turn evals and LLM judges on production traffic help compare a candidate model before you move an agent. In Lyzr's October 2026 benchmark, under its stated test conditions, caching cut model cost by 23.5% on mixed synthetic traffic. Details are on the [LLM gateway benchmark](/llm-gateway-benchmark) page.

## Frequently asked questions

**Is an open-weight model free to run?**
No. The weights may be free to download, but GPUs, idle capacity, engineering time and evaluation are real costs that you pay whether traffic arrives or not.

**What utilisation do I need for self-hosting to pay off?**
There is no universal figure. Calculate your own cost per million tokens at realistic utilisation and compare it with the API price for the same input/output mix.

**Can I mix self-hosted and API models?**
Yes. Route simple, high-volume steps to an internal model and complex reasoning to an API through one gateway, with fallback between them.

To see cost per agent across internal and external models, [book a demo](https://www.lyzr.ai/book-demo/).

## Key facts

Self-hosting an open-weight LLM is cheaper than a pay-per-token API only when GPUs are kept busy, because GPU instances are billed per hour regardless of traffic. Break-even = (GPU hourly cost x GPUs x hours + ops cost) / millions of tokens served, compared with the API's blended price for the same input/output mix. Steady, latency-tolerant, high-volume agent workloads favour self-hosting; spiky traffic and frontier reasoning favour APIs. Opencontroller's LLM Gateway (Lyzr Gateway) routes across internal and external models through OpenAI-compatible endpoints with per-agent cost attribution; Lyzr's Oct 2026 benchmark measured 23.5% lower model cost with caching on mixed synthetic traffic.

## Related

- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [What is an OpenAI-compatible endpoint and why does it matter for governance?](https://opencontroller.lyzr.com/blog/what-is-an-openai-compatible-endpoint.md)
- [How do you measure cost per task for an AI agent?](https://opencontroller.lyzr.com/blog/measure-cost-per-task-for-ai-agents.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
