---
title: How do you route AI agents to self-hosted or internal models through an LLM gateway?
url: https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway
published: 2026-11-11
updated: 2026-11-11
author: Lyzr
---

# How do you route AI agents to self-hosted or internal models through an LLM gateway?

**Expose the internal model behind an OpenAI-compatible endpoint, register that endpoint with the LLM gateway as another provider, add health checks, and write routing rules that send chosen agents or data classes to it, with fallback to an approved external model when it is unhealthy or overloaded. Agents keep calling the gateway; they never need to know where the model runs.**

Enterprises add internal models for three reasons: data that may not leave their network, cost at high volume, and models fine-tuned on their own domain. The hard part is not serving the model. It is making it a first-class route that gets the same policy, logging and cost attribution as every hosted provider.

## Why route internal models through the gateway at all?

If agents call an internal model directly, you lose the controls you built at the gateway: content policy, token budgets, traces, cost attribution and fallback. You also end up with two integration paths per agent. Routing through the gateway keeps one endpoint, one policy set and one log for every model, wherever it runs. For how gateways relate to other control layers, see [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## Step 1: Serve the model with an OpenAI-compatible API

Most self-hosting stacks already speak the OpenAI API shape, which makes them easy to register:

- **vLLM** ships an [OpenAI-compatible server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). `vllm serve <model>` exposes `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` and `/v1/models`, and a `/health` endpoint, on port 8000 by default.
- **Ollama** offers [OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md) under `/v1` on its local API.
- **Cloud model services** that host open-weight models in your own account typically expose either an OpenAI-compatible route or their own API, which the gateway adapts.

Check three things before registering: streaming works, the `usage` object returns token counts (you need them for cost attribution), and tool or function calling behaves the same way your agents expect.

## Step 2: Register the endpoint as a provider

Treat the internal model like any provider. Record:

| Field | Example | Why it matters |
|---|---|---|
| Provider name | `internal-vllm-eu` | Appears in traces and cost reports |
| Base URL | private DNS name, not a public IP | Keeps traffic on your network |
| Auth | mTLS or a service token | Internal does not mean unauthenticated |
| Model IDs | the names returned by `/v1/models` | Routing rules reference these |
| Region / data zone | `eu-internal` | Lets residency rules route to it |
| Unit cost | internal cost per 1M tokens | Makes cost attribution comparable with hosted models |
| Limits | max concurrent requests, max context | Prevents the gateway overloading the server |

Set a unit cost even for self-hosted models. Divide the monthly cost of the GPUs and operations by tokens served; it does not need to be exact, but without it the internal model looks free and cost reports mislead.

## Step 3: Add health checks

Two kinds of check catch different failures:

1. **Liveness:** poll the health endpoint every few seconds. If it fails, remove the route from rotation.
2. **Synthetic completions:** send a tiny prompt every minute or so and check the response and latency. A server can report healthy while the model is stuck loading or the GPU is out of memory.

Also watch queue depth or time-to-first-token. Self-hosted servers degrade gradually under load rather than failing cleanly, so latency is often the first signal.

## Step 4: Write routing and fallback rules

Typical rules, in priority order:

1. **Data-class rules:** requests carrying restricted data go only to the internal model. If it is down, **fail closed** (return an error) rather than falling back externally.
2. **Agent rules:** named agents or versions use the internal model by default.
3. **Cost rules:** high-volume, low-complexity tasks such as classification or summarisation go internal.
4. **Fallback:** for unrestricted traffic, if the internal route is unhealthy, exceeds a latency threshold or returns errors, retry once, then fall back to an approved hosted model.

Fail-closed versus fail-open is the decision that matters most. Write it down per data class. Residency rules work the same way; see [how to enforce data residency for LLM and agent traffic](/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic).

## Step 5: Check quality before shifting traffic

An internal model is a different model. Before routing production traffic, run the same evaluation set against it and the current hosted model, then shift a small percentage of traffic and compare quality scores, latency and cost per agent. Keep the fallback rule live during the shift.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) routes model traffic with fallback and retry, caching and cost attribution per agent and per version, using OpenAI-compatible endpoints, so internal and hosted models sit behind one interface. Multi-turn and red-team evals and LLM judges that score production traffic help compare models before and after a shift. Opencontroller can run on Lyzr SaaS or on customer-controlled infrastructure, which matters when the internal models sit on a private network. See [how to attribute LLM costs to AI agents](/blog/how-to-attribute-llm-costs-to-ai-agents), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can an LLM gateway route to self-hosted models?**
Yes, if the model is served behind an API the gateway understands. OpenAI-compatible servers such as vLLM and Ollama are the simplest to register.

**What happens if the internal model goes down?**
For unrestricted traffic, fall back to an approved hosted model. For restricted data, fail closed and return an error rather than sending it externally.

**How do you compare internal and hosted model cost?**
Assign the internal model a unit cost per million tokens from GPU and operations spend, so the gateway's cost attribution covers both on one basis.

## Key facts

To route agents to internal models: serve the model behind an OpenAI-compatible API (e.g. vLLM, Ollama), register it as a gateway provider with a unit cost, add liveness and synthetic health checks, and set routing with fail-closed rules for restricted data and fallback for other traffic. Opencontroller's LLM Gateway (Lyzr Gateway) provides routing, fallback/retry, caching and per-agent cost attribution behind OpenAI-compatible endpoints. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [LLM gateway vs agent gateway vs AI control plane: what's the difference?](https://opencontroller.lyzr.com/blog/llm-gateway-vs-agent-gateway-vs-control-plane.md)
- [How do you attribute LLM costs to individual AI agents?](https://opencontroller.lyzr.com/blog/how-to-attribute-llm-costs-to-ai-agents.md)
- [How do you enforce data residency for LLM and AI agent traffic?](https://opencontroller.lyzr.com/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
