---
title: What network setup does a self-hosted LLM gateway need?
url: https://opencontroller.lyzr.com/blog/self-hosted-llm-gateway-network-checklist
published: 2026-11-11
updated: 2026-11-11
author: Lyzr
---

# What network setup does a self-hosted LLM gateway need?

**A self-hosted LLM gateway needs six network changes: a DNS name, a TLS certificate your clients trust, an ingress path from agents to the gateway, an egress allow-list from the gateway to model providers, a proxy bypass or proxy configuration for that traffic, and health checks. Most delays come from these steps being owned by four different teams, so assign owners before the install date.**

The gateway software is usually the quick part. Activation stalls on a DNS ticket, a certificate request or a firewall change waiting in another team's queue. This checklist lists every change and who typically owns it.

## What is the full network checklist?

| # | Step | What to configure | Usually owned by |
|---|---|---|---|
| 1 | DNS | Internal name for the gateway (for example `llm-gateway.internal.example.com`), private zone record | Network or DNS team |
| 2 | TLS certificate | Certificate for that name from your private CA or public CA; renewal automated | PKI or platform team |
| 3 | Client trust | Private CA root installed in agent runtimes and containers | Platform team and app owners |
| 4 | Ingress | Load balancer or Kubernetes Ingress exposing the gateway to agent networks only | Platform or cloud team |
| 5 | Source restrictions | Security groups or network policies allowing only agent subnets | Security or cloud team |
| 6 | Egress allow-list | Outbound access from the gateway to each model provider endpoint and internal model hosts | Network security |
| 7 | Proxy handling | Either route gateway egress through the corporate proxy, or bypass it with documented approval | Network security |
| 8 | Proxy bypass for clients | Add the gateway name to `NO_PROXY` for agents so internal calls do not hairpin through the proxy | App owners |
| 9 | Health checks | Load balancer health check on the gateway's health endpoint | Platform team |
| 10 | Logging | Access logs and traces shipped to your log store | Observability team |

## Why do certificates cause the most failures?

Agents fail with TLS errors when they do not trust the CA that signed the gateway's certificate. Each runtime finds trusted CAs differently:

- **Python `requests`** uses its own bundle unless `REQUESTS_CA_BUNDLE` points to yours.
- **Node.js** adds extra CAs through `NODE_EXTRA_CA_CERTS`.
- **Java** uses the JVM trust store.
- **Containers** need the CA baked into the image or mounted at runtime.

Test from each agent runtime type before go-live, not just from a laptop. If you run on Kubernetes, [cert-manager](https://cert-manager.io/docs/) can issue and renew certificates from your private CA automatically.

## Should gateway egress go through the corporate proxy?

Both options work. The trade-off is visibility versus latency and fragility.

| Option | Pros | Cons |
|---|---|---|
| Through the proxy | Existing inspection and logging; no new firewall rules | TLS inspection can break streaming responses; proxy becomes a dependency for every model call |
| Direct egress with allow-list | Lower latency; simpler streaming | Needs new firewall rules per provider; less central inspection |

If you use TLS inspection, exempt model provider endpoints or test streaming responses carefully. Server-sent events are often the first thing to break. Either way, the gateway itself becomes the inspection point for prompts and responses, which is usually more useful than proxy logs.

## How do you avoid ingress mistakes?

- **Do not expose the gateway publicly** unless agents outside your network must reach it. Use an internal load balancer.
- **Limit source ranges** to agent subnets and build pipelines. See [how to manage IP allow-lists without drift](/blog/manage-llm-gateway-ip-allowlists-without-drift).
- **Set idle timeouts high enough for streaming.** Long model responses can exceed default load balancer idle timeouts.
- **Keep one hostname per environment** (dev, test, prod), so agents do not cross environments by accident.

The [Kubernetes Ingress documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) covers the controller-specific settings for timeouts and source ranges.

## What order should the steps happen in?

1. Book owners for steps 1, 2, 4, 6 and 7 two weeks before install.
2. Create DNS and certificate first; they have the longest lead time.
3. Deploy the gateway and ingress with source restrictions.
4. Open egress for one provider and test end to end.
5. Install CA trust in agent runtimes and set `NO_PROXY`.
6. Move one low-risk agent, then expand.

If you already run an API gateway, read [putting an LLM gateway behind your existing API gateway](/blog/llm-gateway-behind-existing-api-gateway). For routing to internal models, see [route agents to internal models through an LLM gateway](/blog/route-agents-to-internal-models-through-llm-gateway).

## How Opencontroller does this

Opencontroller, by Lyzr, can run as Lyzr SaaS or in customer-controlled infrastructure. In a self-hosted deployment, Lyzr engineers embed with your team to work through these network steps. The LLM Gateway (Lyzr Gateway) exposes OpenAI-compatible endpoints, so most agents only need a base URL change, and it exports traces with OpenTelemetry. In an October 2026 benchmark, under stated test conditions, it added under 0.7 ms p95 per in-gateway routing method and handled 5,134 requests per second on one 4-vCPU gateway. See the [benchmark](/llm-gateway-benchmark), the pillar [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Does a self-hosted LLM gateway need a public IP?**
Usually not. Agents inside your network reach it through an internal load balancer; only egress to model providers leaves your network.

**Why do agents get TLS errors after switching to the gateway?**
Their runtime does not trust the CA that signed the gateway's certificate. Install the CA in each runtime's trust store.

**Should the LLM gateway hostname be in NO_PROXY?**
Yes, for internal clients, so calls to the gateway do not route through the corporate proxy and back.

## Key facts

A self-hosted LLM gateway needs DNS, a TLS certificate clients trust, client CA trust (REQUESTS_CA_BUNDLE for Python requests, NODE_EXTRA_CA_CERTS for Node.js, the JVM trust store for Java), internal ingress restricted to agent subnets, an egress allow-list to model providers, a decision on corporate proxy routing, NO_PROXY entries for clients, health checks and logging. Owners usually span network, PKI, platform and security teams. TLS inspection can break streaming responses. Opencontroller by Lyzr runs as SaaS or in customer infrastructure; its LLM Gateway exposes OpenAI-compatible endpoints and in an October 2026 benchmark added under 0.7 ms p95 per routing method and handled 5,134 req/s on one 4-vCPU gateway under stated conditions. Pricing starts from $250K per deployment per year.

## Related

- [Can an LLM gateway sit behind your existing API gateway?](https://opencontroller.lyzr.com/blog/llm-gateway-behind-existing-api-gateway.md)
- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [How do you enforce data residency for LLM and AI agent traffic?](https://opencontroller.lyzr.com/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
