---
title: How do you find LLM gateways and proxies teams set up on their own?
url: https://opencontroller.lyzr.com/blog/find-shadow-llm-gateways
published: 2026-11-13
updated: 2026-11-13
author: Lyzr
---

# How do you find LLM gateways and proxies teams set up on their own?

**Look for the places traffic gets redirected: base-URL environment variables that point away from the model provider, gateway container images and Kubernetes services, internal DNS names, and provider API keys whose call volume is far higher than one app would make. Then register each gateway as an asset with an owner, because one unregistered gateway can hide dozens of agents.**

Self-hosted LLM gateways and proxies are easy to start. Open-source options run as a single container, and many teams set one up to share keys, add caching or switch providers. That is reasonable engineering. The governance problem is that discovery tools looking for agents see one gateway, one key and one stream of traffic, not the agents behind it.

## Why do shadow gateways matter for governance?

- **They aggregate identity.** Every agent behind the gateway calls the provider with the gateway's key, so provider logs cannot tell agents apart.
- **They hold secrets.** A gateway's config usually stores live provider keys for several vendors.
- **They change routing silently.** A config edit can send traffic to a new provider or region with no review.
- **They log prompts.** Many gateways store request and response bodies for debugging, creating an unreviewed copy of sensitive data.

## What signals reveal a self-hosted gateway?

| Where to look | Signal | Example |
|---|---|---|
| App config and environment variables | Base-URL overrides pointing to internal hosts | `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` set to an internal hostname |
| Container registries and running workloads | Known gateway or proxy images | Open-source proxy images; LiteLLM's proxy listens on port 4000 in its quick start |
| Kubernetes | Services and ingresses with gateway names or ports | Services named `llm-proxy`, `ai-gateway`, or exposing port 4000 |
| Internal DNS and load balancers | Names like `llm.`, `ai-gateway.`, `openai-proxy.` | New internal records created by application teams |
| Provider billing and key usage | One key with traffic from many workloads or far higher volume than expected | A key used from several clusters or regions |
| Network egress logs | Many internal clients talking to one host, which then talks to model providers | Fan-in, fan-out pattern |
| Code repositories | Gateway config files and Helm values | Model lists, provider keys, routing rules in YAML |
| Self-hosted model servers | Inference servers that act as local endpoints | Ollama on port 11434, vLLM's OpenAI-compatible server |

No single signal is proof. A base-URL override might point to an approved corporate gateway. Combine at least two signals before you open a ticket.

## What do you do once you find one?

1. **Register it** as a gateway asset with an owner, environment and the providers it reaches.
2. **List what is behind it.** Ask the owner for the agents and apps that call it, and check its own logs for distinct client keys.
3. **Rotate the provider keys** it holds if they were shared outside a secrets manager.
4. **Check prompt logging.** Where are request bodies stored, for how long, and who can read them?
5. **Decide its future:** keep it as an approved team gateway with standard telemetry, or move its consumers to the corporate gateway and retire it.

Our guide on [what to do with a shadow AI agent once you find it](/blog/what-to-do-with-a-shadow-ai-agent-once-you-find-it) covers the same decision for agents.

## How do you stop new shadow gateways appearing?

- Offer an approved gateway that is easier to use than running your own, with per-team keys and cost reports.
- Require provider keys to be issued only to registered gateways and agents.
- Export gateway telemetry using OpenTelemetry GenAI semantic conventions so every gateway reports the same fields.
- Run discovery on a schedule, not once. See [how often AI agent discovery should run](/blog/how-often-should-ai-agent-discovery-run).

## How Opencontroller does this

Opencontroller auto-discovers agents, models and API keys across AWS, Azure and Google Cloud, and keeps them in a governed registry alongside tools and MCP servers. Posture flags risky permissions and configurations. Its own LLM Gateway gives teams OpenAI-compatible endpoints with routing, caching and per-agent cost attribution, which removes much of the reason to run a private proxy. Start with the [shadow AI pillar](/blog/how-to-find-shadow-ai-agents), check the [AI asset types to inventory](/blog/ai-asset-types-to-inventory), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Is a team-run LLM gateway always a problem?**
No. It becomes a problem when it is unregistered, holds shared keys, or logs prompts without review. Registered gateways with owners and standard telemetry can stay.

**Why can't provider logs show the agents behind a gateway?**
The gateway calls the provider with its own key, so the provider sees one client. Only the gateway's own logs can separate the agents.

**How is Opencontroller priced?**
Opencontroller is licensed annually per deployment, not per seat and not usage-based. Ask Lyzr for a quote scoped to your estate.

## Key facts

Shadow LLM gateways are self-hosted proxies teams run to share keys, cache or switch providers; they hide many agents behind one provider key, hold live keys and may log prompts. Signals to find them: base-URL overrides (OPENAI_BASE_URL, ANTHROPIC_BASE_URL) pointing to internal hosts, known gateway container images (LiteLLM proxy uses port 4000 in its quick start), Kubernetes services/ingresses, internal DNS names, keys with unusually high or multi-workload traffic, fan-in/fan-out egress patterns, gateway configs in repos, and local inference servers (Ollama port 11434, vLLM OpenAI-compatible server). Combine two or more signals, then register, list consumers, rotate keys, review prompt logging, and keep or retire. Opencontroller by Lyzr auto-discovers agents, models and API keys across AWS, Azure and Google Cloud and offers an LLM Gateway with OpenAI-compatible endpoints. Pricing starts from $250K per deployment per year.

## Related

- [What should you do with a shadow AI agent once you find it?](https://opencontroller.lyzr.com/blog/what-to-do-with-a-shadow-ai-agent-once-you-find-it.md)
- [Which AI asset types should an AI inventory cover?](https://opencontroller.lyzr.com/blog/ai-asset-types-to-inventory.md)
- [How often should AI agent discovery scans run?](https://opencontroller.lyzr.com/blog/how-often-should-ai-agent-discovery-run.md)
- [More on Discovery and shadow agents](https://opencontroller.lyzr.com/blog/cluster/discovery-and-shadow-agents)
