---
title: How do you attribute GPU costs of self-hosted models to AI agents?
url: https://opencontroller.lyzr.com/blog/attribute-gpu-costs-of-self-hosted-models-to-agents
published: 2026-11-18
updated: 2026-11-18
author: Lyzr
---

# How do you attribute GPU costs of self-hosted models to AI agents?

**Take the fully loaded monthly cost of the inference cluster, measure each agent's weighted token share from gateway logs, and allocate cost in proportion. Then decide, explicitly, who pays for idle capacity: the agents pro rata, a platform budget, or the teams that reserved it.** Token share is the practical unit because GPU seconds cannot be cleanly split per request when a serving engine batches many agents together.

## Why can't you just read GPU usage per agent?

Modern serving engines such as vLLM use continuous batching. One forward pass on a GPU serves requests from many agents at once, so GPU time is shared at the step level. Kubernetes can tell you which pod used a GPU, but every agent calls the same pod. You need a request-level signal, and the most reliable one is tokens:

- **The gateway** knows which agent and version sent each request, and records input and output token counts.
- **The serving engine** knows total tokens processed. vLLM, for example, exposes Prometheus counters such as `vllm:prompt_tokens_total` and `vllm:generation_tokens_total` per model.

Use the gateway for the split, and the engine counters to reconcile totals.

## Why weight output tokens differently?

Input (prefill) tokens are processed in parallel. Output (decode) tokens are generated one at a time and hold GPU memory for the whole generation. Per token, output is usually the more expensive resource, which is why API providers price output tokens several times higher than input.

Pick a weight and document it. A simple approach is to copy the input:output price ratio of a comparable hosted model. Better, measure it: run a load test with input-heavy and output-heavy traffic and compare throughput. If your engine uses prefix caching, consider a lower weight for cached input tokens.

## A worked example

All numbers below are **illustrative assumptions**, not benchmarks.

- Fully loaded cluster cost: **$60,000 per month** (GPU nodes, storage, networking, on-call share).
- Output weight: **1 output token = 4 input-token units.**
- Measured busy utilisation: **55%.**

| Agent | Input tokens | Output tokens | Weighted units | Share |
|---|---|---|---|---|
| A: support triage | 400M | 40M | 560M | 50.9% |
| B: report drafting | 150M | 60M | 390M | 35.5% |
| C: code review | 50M | 25M | 150M | 13.6% |
| **Total** | 600M | 125M | **1,100M** | 100% |

Weighted units = input + 4 × output. Agent B uses far fewer input tokens than A, but its long outputs give it over a third of the bill.

## How should idle capacity be handled?

GPU clusters are rarely fully used. Choose one of three policies and apply it consistently.

| Policy | Agent A | Agent B | Agent C | Platform | When it fits |
|---|---|---|---|---|---|
| 1. Full pro-rata | $30,545 | $21,273 | $8,182 | $0 | Platform is fully chargeback-funded |
| 2. Used share only | $16,800 | $11,700 | $4,500 | $27,000 | Platform team owns sizing decisions |
| 3. Reserved capacity | Fixed fee for reserved replicas, plus policy 2 for shared pool | | | Remainder | Teams demand dedicated latency or throughput |

Policy 2 charges agents for 55% of the cost (their used share) and leaves the 45% idle cost with the platform. It gives the platform team a reason to right-size. Policy 1 makes agent costs move when other agents leave, which owners find confusing.

## What data do you need?

A checklist for a monthly allocation run:

1. **Cluster cost** from the cloud bill or internal hardware amortisation, tagged to the inference cluster.
2. **Per-request logs** from the gateway: agent ID, agent version, model, input tokens, output tokens, cached tokens, timestamp.
3. **Engine counters** per model for reconciliation. Gateway totals should match within a small, explained variance.
4. **Utilisation** over the month, from GPU metrics or engine throughput versus measured capacity.
5. **Agent-to-cost-centre mapping**, ideally set when the agent is approved. See [tying agents to cost centres at approval](/blog/tie-ai-agents-to-cost-centres-at-approval).
6. **Unattributed traffic report.** Any calls that bypassed the gateway, or arrived without an agent ID, are listed and assigned to an owner.

OpenTelemetry's GenAI semantic conventions define standard attributes for input and output token usage, which makes it easier to join gateway data with traces.

## How does this compare to API spend?

Once self-hosted models are costed per weighted token, you can compare them to hosted APIs on the same basis. Our guide to [self-hosted LLM vs API cost](/blog/self-hosted-llm-vs-api-cost-for-agents) covers that decision, and [routing agents to internal models through an LLM gateway](/blog/route-agents-to-internal-models-through-llm-gateway) covers the traffic path. The [gateway vs control plane overview](/blog/llm-gateway-vs-agent-gateway-vs-control-plane) shows where cost attribution sits.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) routes traffic to hosted and internal models through OpenAI-compatible endpoints and attributes cost per agent and per version. Combined with auto-discovery of models and API keys across AWS, Azure and Google Cloud, that gives one per-agent view of hosted and self-hosted spend. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Why not allocate GPU cost by request count?**
Requests vary hugely in size. One long-output request can use more GPU time than hundreds of short classification calls, so weighted tokens are a fairer unit.

**Who should pay for idle GPUs?**
It is a policy choice. Charging agents only for used share and leaving idle cost with the platform team keeps agent costs stable and rewards right-sizing.

**How do we check the gateway numbers are complete?**
Compare gateway token totals per model with the serving engine's own counters each month, and investigate any traffic that reached the engine without passing the gateway.

## Key facts

To attribute self-hosted GPU cost to AI agents: allocate fully loaded cluster cost by each agent's weighted token share from gateway logs (output tokens weighted higher than input), reconcile totals against serving-engine counters such as vLLM's vllm:prompt_tokens_total and vllm:generation_tokens_total, and pick an explicit idle-capacity policy (full pro-rata, used share with idle to the platform, or reserved capacity). Continuous batching means GPU time cannot be split per request directly. Opencontroller attributes cost per agent and per version through its LLM Gateway; licensing starts from $250K per deployment per year.

## Related

- [When is self-hosting an LLM cheaper than an API for AI agents?](https://opencontroller.lyzr.com/blog/self-hosted-llm-vs-api-cost-for-agents.md)
- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [How do you allocate shared AI platform costs to divisions?](https://opencontroller.lyzr.com/blog/allocate-shared-ai-platform-costs-to-divisions.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
