---
title: What is the Kubernetes Gateway API Inference Extension?
url: https://opencontroller.lyzr.com/blog/kubernetes-gateway-api-inference-extension
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# What is the Kubernetes Gateway API Inference Extension?

**The Gateway API Inference Extension is a Kubernetes project that turns a Gateway API gateway into an "inference gateway" for self-hosted models. You replace a Service with an InferencePool, and an Endpoint Picker sends each request to the best model-server pod based on signals such as prefix cache status and loaded LoRA adapters. It governs load and model routing inside the cluster; it does not know which agent is calling, what it may do, or what it costs.**

Platform teams running open-weight models on GPUs meet this project quickly, because ordinary load balancing wastes expensive accelerators. Governance teams then ask whether it is "the AI gateway". It is not — but it is a useful layer underneath one.

## How does it work?

The project, driven by Kubernetes WG-Serving and SIG-Network, extends gateways that support both Gateway API and Envoy's external processing (ext-proc) protocol. Its stated goals are to standardise routing to inference workloads, reduce latency and improve GPU utilisation.

Four parts matter:

| Component | What it is |
|---|---|
| **InferencePool** | A backend resource that selects model-server pods. You use it in place of a Kubernetes Service. GA at `v1` in `inference.networking.k8s.io`. |
| **InferenceObjective** | Describes a workload's serving goals, such as `priority`. It replaced `InferenceModel` in the v1 API. |
| **Endpoint Picker (EPP)** | The inference router. It picks the endpoint for each request and tells the gateway where to send it. |
| **Gateway implementation** | Envoy Gateway, kgateway and GKE Gateway are named by the project. |

The EPP uses metrics and capabilities from model servers. The project names prefix cache status and LoRA adapter availability as examples. That is why a request can land on the pod that already has the right adapter loaded, instead of the next pod in round-robin order.

The project also supports **model-aware routing**: requests to OpenAI-compatible chat completion endpoints can be routed by model name, not only by URL path. That enables traffic splitting between model versions during a rollout.

## What does it govern — and what does it not?

| Question | Inference Extension | Needs another layer |
|---|---|---|
| Which pod serves this request? | Yes | — |
| Which model version gets 10% of traffic? | Yes (by model name) | — |
| Which requests get served first under load? | Yes (priority) | — |
| Which agent sent the request, on whose behalf? | No | Agent identity |
| Is this agent allowed to call this model? | No | Policy |
| Does the prompt contain data that must be redacted? | No | Guardrails |
| What did each agent and version cost? | No | Cost attribution |
| Can we stop a misbehaving agent now? | No | Runtime control |

The extension sees requests, not agents. Unless every agent has its own credential and something checks it, a request from a test agent and one from a production payments agent look identical to the EPP.

## Where does it sit in a governed architecture?

A practical layering for self-hosted models:

1. **Agents** call one OpenAI-compatible endpoint with their own workload identity — see [Kubernetes workload identity for agent pods](/blog/kubernetes-workload-identity-for-agent-pods).
2. **An LLM gateway** authenticates the agent, applies policy (allow, route, redact, block), records cost per agent and falls back to a hosted model if needed.
3. **The inference gateway** (Gateway API plus Inference Extension) spreads the gateway's traffic across GPU pods efficiently.
4. **Model servers** run the open-weight models.

This keeps each layer doing one job. The LLM gateway decides **whether and where** a request may go; the inference gateway decides **which pod** serves it. For the model-routing concept, see [what LLM model routing is](/blog/what-is-llm-model-routing). For a worked pattern, see [routing agents to internal models through an LLM gateway](/blog/route-agents-to-internal-models-through-llm-gateway).

## What should you check before adopting it?

1. **API version.** Migrate any `v1alpha2` resources to the GA `v1` API; the API group changed from `inference.networking.x-k8s.io` to `inference.networking.k8s.io`.
2. **EPP choice.** The project ships a reference EPP for conformance testing and recommends a production EPP for real workloads.
3. **Gateway support.** Confirm your gateway implementation supports ext-proc and the extension.
4. **Identity at the edge.** Do not expose the inference gateway directly to agents without authentication per agent.
5. **Telemetry.** Export request metrics with OpenTelemetry so cost and latency can be tied back to agents upstream.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) exposes OpenAI-compatible endpoints and provides routing, fallback and retry, caching and cost attribution per agent and per version, in front of hosted or self-hosted models. Agent workload identity federation ties each request to a registered agent. In Lyzr's October 2026 benchmark, under the stated test conditions, in-gateway routing added under 0.7 ms at p95 per routing method. See [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), the [benchmark](/llm-gateway-benchmark), the [cross-platform governance guide](/blog/how-to-govern-ai-agents-across-platforms), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**What replaced InferenceModel?**
InferenceObjective, in the v1 API. Its `poolRef` points to a v1 InferencePool.

**Which gateways support the Inference Extension?**
The project names Envoy Gateway, kgateway and GKE Gateway, and expects more of the 25+ Gateway API implementations to add support.

**Does it work with hosted APIs such as OpenAI?**
It targets self-hosted model servers in your clusters. Hosted APIs are reached through an LLM gateway instead.

## Key facts

- The Gateway API Inference Extension is a Kubernetes project, driven by WG-Serving and SIG-Network, that turns Gateway API gateways supporting Envoy external processing into inference gateways for self-hosted models.
- InferencePool is GA at v1 in the API group inference.networking.k8s.io; the v1 API replaced InferenceModel with InferenceObjective.
- An Endpoint Picker (EPP) chooses the model-server pod for each request using model-server signals such as prefix cache status and LoRA adapter availability.
- Named gateway implementations include Envoy Gateway, kgateway and GKE Gateway.
- The extension does not provide per-agent identity, policy decisions, guardrails or cost attribution per agent.
- Opencontroller's LLM Gateway offers routing, fallback/retry, caching and cost attribution per agent and per version, with OpenAI-compatible endpoints.
- Opencontroller starts from $250K per deployment per year.

## Related

- [How do you govern AI agents running on Kubernetes?](https://opencontroller.lyzr.com/blog/govern-ai-agents-on-kubernetes.md)
- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [What is LLM model routing?](https://opencontroller.lyzr.com/blog/what-is-llm-model-routing.md)
- [More on Platform Guides](https://opencontroller.lyzr.com/blog/cluster/platform-guides)
