Blog / Topic
Gateways and cost
Start here
All articles
How do you autoscale an LLM gateway on Kubernetes?
Scale an LLM gateway on Kubernetes with the Horizontal Pod Autoscaler: which metric to use, how to handle streaming connections, and how to size replicas.
How do you attribute LLM usage when many users share one application key?
Three ways to see who is behind traffic on a shared LLM key: per-user virtual keys, end-user identifier fields, and signed token pass-through.
How are images and documents priced in AI agent calls?
Images and PDFs are billed as input tokens. The Claude formula, worked examples for screenshot agents, and the levers that cut multimodal agent cost.
Do structured outputs work the same across LLM providers behind a gateway?
OpenAI, Anthropic and Gemini each support a different JSON Schema subset for structured outputs. What breaks when a gateway fails over, and how to plan for it.
How do you attribute GPU costs of self-hosted models to AI agents?
Split self-hosted GPU cost across AI agents using weighted token shares from gateway logs, with a worked example and three ways to handle idle capacity.
How much does storing AI agent traces cost?
A formula and worked example for AI agent trace volume, why payloads dominate size, and how sampling and tiering cut cost without losing audit evidence.
Can an LLM gateway govern realtime voice APIs?
Realtime voice agents use WebRTC, WebSocket and SIP, not plain HTTP. What a gateway can see and enforce for each transport, and the control points that remain.
What is an agent gateway?
An agent gateway mediates agent-to-tool (MCP), agent-to-agent (A2A) and inbound agent calls. What it enforces and how it differs from an LLM gateway.
Does extended thinking work through an LLM gateway?
Claude thinking blocks must pass back unmodified with signatures in tool loops. Where LLM gateways break this, cross-platform rules, cost and a test set.
When should AI agents use priority or flex processing tiers?
Flex, standard, priority and batch tiers trade price for latency and reliability. Which agent workloads fit each, with real OpenAI and Gemini tier behaviour.
Should you cap max tokens for AI agents at the gateway?
Cap output tokens, input context, calls per task and spend per agent at the gateway. How to set the numbers and spot truncation.
What should an LLM gateway log, and what should it never store?
Log metadata on every request; treat prompts and responses as sensitive data that is sampled, redacted and access-controlled. A field-by-field table.
How does long-context pricing change the cost of AI agents?
Some models charge more per token once a prompt passes 200K tokens. Current examples, a worked agent cost calculation, and six ways to stay under the line.
How much do LLM-as-judge evaluations cost on production traffic?
A worked cost formula for LLM-as-judge evaluation of agent traffic (traces × sampling × criteria × tokens × price) and the levers that cut it.
How do you detect anomalies in AI and LLM spend?
Budgets catch overspend in days; anomaly detection catches a runaway agent in an hour. Baselines per agent version, five signals, thresholds and a worked example.
How do you plan for LLM model retirements in production agents?
Where Anthropic, Azure OpenAI, OpenAI and Vertex AI publish model retirement dates, their notice periods, and a migration checklist with evaluation gates.
What are virtual keys in an LLM gateway?
A virtual key is a gateway-issued credential that maps to a provider key plus an owner, budget and model allow-list. How they work and how to rotate them.
How do you upgrade a self-hosted LLM gateway without breaking agents?
Checklist for upgrading a self-hosted LLM gateway: rollout pattern, config and key compatibility, SSO re-checks, provider smoke tests and rollback.
How much does retrieved context add to AI agent cost?
Retrieved chunks are often the biggest share of an agent's input tokens. A worked example of chunks × tokens × turns, and the levers that cut it.
Why do multi-agent systems use so many more tokens?
Anthropic reported its multi-agent research system used about 15x the tokens of chat. Where the tokens go, when it is worth it, and how to control cost.
How do you tie every AI agent to a cost centre when it is approved?
The seven approval fields that make AI agent chargeback automatic, how the cost centre flows to keys, gateway routes and cloud tags, and a backfill plan.
What is a denial-of-wallet attack on AI agents?
A denial-of-wallet attack runs up your model and tool bill through an AI agent. Attack patterns, the OWASP reference, and the gateway controls that stop them.
How do you attribute Microsoft Copilot agent costs to teams and cost centres?
Use Copilot Credit spending policies, the Consumption view and the Cost Management API to attribute Copilot agent costs to teams and cost centres.
When is self-hosting an LLM cheaper than an API for AI agents?
Self-hosting an LLM beats a per-token API only at high, steady GPU utilisation. Cost components, a break-even formula, and the governance needed either way.
Should AI budgets be set per user, per agent or per team?
Set AI budgets per agent first, roll them up to teams, and add per-user caps only where people drive spend. A comparison of budget scopes and how to layer them.
How do reasoning tokens change the cost of an AI agent?
Reasoning models bill hidden thinking tokens as output. How effort and thinking budgets work, how to measure them per task, and a worked cost example.
How much do MCP tool definitions add to token costs?
MCP tool schemas are sent as input tokens on every model call. How to measure the overhead per agent, a worked example, and ways to cut it.
How does provider prompt caching cut AI agent costs?
Anthropic and OpenAI price cached prompt tokens differently. Compare write and read multipliers, minimums and TTLs, and how it differs from gateway caching.
When should AI agent workloads use batch APIs?
OpenAI and Anthropic batch APIs cost 50% of standard prices with results within 24 hours. Which agent workloads qualify, which do not, and how to size savings.
How do you forecast AI agent spend for next year's budget?
A driver-based model for AI agent spend: agents × runs × tokens × price, with scenarios for price drops, model mix and adoption, plus seat lines.
Should embedding calls go through the LLM gateway?
Embedding calls send whole documents to a provider, spike during re-indexing and are often missing from cost reports. When to gateway them and when a direct path is fine.
Why do long AI agent sessions get so expensive and how do you cut the cost?
Agents resend the whole context every turn, so session cost grows roughly quadratically. A worked example and seven levers to cut it.
How much latency does an LLM gateway add?
Where LLM gateway latency comes from, how to measure p95 overhead fairly, and which features cost the most milliseconds, with scoped benchmark figures.
Why does AI spend in the gateway not match the provider invoice?
Gateway cost figures and model provider invoices rarely match. The usual causes, and a monthly reconciliation checklist with a tolerance threshold.
How do you make an LLM gateway highly available?
Run stateless gateway replicas behind a load balancer, health-check them, fail over across regions and model providers, and size from your own load tests.
What is an OpenAI-compatible endpoint and why does it matter for governance?
An OpenAI-compatible endpoint accepts OpenAI API-shaped requests, so apps switch providers by changing base_url. Why it helps governance and what differs.
How do you separate AI coding assistant costs from AI agent runtime costs?
Build costs (coding assistants, premium requests, CI agents) and runtime costs (tokens per agent run) have different drivers. A tagging scheme to split them.
What does a CISO security review of an LLM gateway cover?
The nine areas a CISO checks before an LLM gateway goes live, from SSO-only admin access to prompt logging and egress IPs, with pass criteria for each.
How do you measure productivity when several AI tools contribute to the same outcome?
When assistants and agents touch the same work, vendor dashboards double count. Measure outcomes at the process level with cohorts and holdouts.
How do you keep your prompts, evaluations and logs when you switch model providers?
Own the prompts, eval sets, traces and routing rules that make your agents work, and avoid depending on state the model provider holds for you.
How do you attribute AI spend to teams when your identity provider has no cost-centre data?
Fallback sources for AI cost attribution when the IdP lacks cost centres: HR system, manager chain, code-host org, cloud tags, agent owner, with confidence flags.
How do you allocate shared AI platform costs to divisions?
Split AI costs into direct and shared pools, allocate shared costs by usage, fixed split or headcount, and send each division a clear monthly statement.
How do you rate-limit AI agents?
Rate-limit AI agents with per-agent token and request quotas, burst allowances and priority tiers, so one agent cannot drain the shared provider quota.
What is LLM model routing?
LLM model routing sends each request to the model best suited by rule, cost, availability or difficulty. Four strategies and when each pays off.
How do you measure cost per task for an AI agent?
Cost per task = all model, retry, tool and human-escalation costs for an agent, divided by tasks completed successfully. Worked example inside.
Can Azure API Management act as an AI gateway for agents?
Azure API Management has AI gateway policies for token limits, token metrics, semantic caching and content safety, plus MCP support. What it covers for agents.
How do you track OpenAI API costs by project with the Usage and Costs APIs?
Use an admin key to pull daily costs by project and line item from the Costs API, tokens by API key and model from the Usage API, and give each agent a project.
How should users and applications authenticate to an LLM gateway?
Separate console login from machine access: SSO for people, and scoped virtual keys, OAuth client credentials or workload identity for apps.
How do you attribute GitHub Copilot usage and cost to employees across several GitHub enterprises?
Cover every GitHub enterprise and org, then map GitHub logins to employees using EMU naming rules, SAML identities or verified emails. Gaps included.
How do you control which teams can use expensive frontier models?
Tier your models, default every agent to the cheapest tier that passes its evals, give each tier a budget, and grant frontier access by exception.
Should you charge AI costs back to business units or just show them?
Showback vs chargeback for AI spend: when each fits, allocation keys when business-unit tags are missing, and a staged path from one to the other.
How do you combine AI seat licences and token spend into one cost view?
Join per-seat AI licences with metered token spend per person, team and agent. A formula, the data each vendor exposes, and a worked example.
How do you find unused GitHub Copilot, ChatGPT Enterprise and Claude seats?
Which activity fields GitHub Copilot, ChatGPT Enterprise and Claude expose for idle seats, their limits, and an inactivity policy you can run monthly.
How should an LLM gateway handle fallback and retries for AI agents?
Which LLM errors to retry, how to set retry budgets, six equivalence rules for fallback models, and how to stop retries from repeating tool calls.
How much can LLM caching save, and when is it unsafe?
Provider prompt caching vs exact-match vs semantic caching for AI agents: what each saves, a scoped benchmark figure, and seven cases where caching is unsafe.
Can our existing API gateway govern AI agents?
An API gateway can enforce auth, quotas and token limits on agent traffic. It cannot see agents that bypass it, their tools, their owners or their behaviour.
How do you see AI costs inside data platforms like Snowflake and Databricks?
The system views and billing tables in Snowflake and Databricks that expose AI functions, agent and model serving spend, and how to roll them up for finance.
How do you monitor LLM provider latency and errors across agents?
A per-provider SLO set for LLM traffic: p95 latency, time to first token, 429 and 5xx rates, plus the fallback triggers that should act on them.
Can an LLM gateway sit behind your existing API gateway?
Yes. Which layer handles auth, routing, token budgets and cost attribution, what breaks (streaming, timeouts) and the latency budget for both hops.
How do you manage IP allow-lists for an LLM gateway without infrastructure drift?
Stop LLM gateway allow-list drift: one owner per list, managed CIDR lists fed by vendor egress feeds via pull requests, and scheduled drift checks.
What network setup does a self-hosted LLM gateway need?
A ten-step network checklist for a self-hosted LLM gateway: DNS, private CA certificates, ingress, egress allow-lists, proxy bypass and owners.
What AI spend reports does a CFO actually need?
Five AI spend report definitions for finance, with grain, fields, owner and cadence, separating seat, consumption and infrastructure spend.
Where do AI agent costs usually leak?
Eight AI agent cost leak patterns, from retry storms to orphaned API keys, with the metric that exposes each and a monthly cost-leak review.
How do you set budgets and alerts for AI agent spend?
A four-level budget hierarchy for AI agent spend (organisation, team, agent, version) with alert thresholds, hard-stop rules and what provider limits miss.
How do you test whether a cheaper model can replace the one an AI agent uses?
A replay-and-judge method to test a cheaper model for an AI agent: sample traffic, score with LLM judges, set acceptance thresholds, then canary live traffic.
How do you route AI agents to self-hosted or internal models through an LLM gateway?
Register an OpenAI-compatible internal model with your LLM gateway, add health checks, and set routing and fail-closed fallback rules. Five steps.
How do you attribute LLM costs to individual AI agents?
Tag every model call with agent ID and version at the gateway, record input, cached and output tokens, price them per model, and roll up by agent and team.
Put the framework into practice