---
title: Should embedding calls go through the LLM gateway?
url: https://opencontroller.lyzr.com/blog/should-embedding-calls-go-through-the-llm-gateway
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# Should embedding calls go through the LLM gateway?

**Usually, yes. Embedding and rerank calls send whole documents to a model provider, so they carry more data-exposure and residency risk than most chat prompts, and their volume spikes during re-indexing in ways that surprise budgets. Routing them through the same gateway as generation calls gives you one place for data rules, cost attribution and keys. A direct path is acceptable for a self-hosted embedding model inside your network, as long as usage is still logged.**

This post belongs to the gateways and cost cluster; the pillar is [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## Why do teams leave embeddings out of the gateway?

Three common reasons:

- **They look cheap.** Per-token prices for embedding models are typically far lower than for generation models.
- **They run in batch jobs.** Ingestion pipelines are owned by data teams, not the team that set up the gateway.
- **They feel harmless.** An embedding is a vector, not text, so it does not look like data leaving.

Each of these is weaker than it seems.

## What risks do embedding calls carry?

| Risk | Why it matters for embeddings | What to check |
|---|---|---|
| Data exposure | Ingestion sends full documents — contracts, tickets, HR files — to the provider | Which data classes are embedded, and by which provider |
| Residency | Batch jobs often use a default region or endpoint | Endpoint region per index |
| Inversion | Research has shown embeddings can be inverted to recover much of the original text | Treat stored vectors as sensitive as the source |
| Cost spikes | Re-indexing a corpus after a model or chunking change re-embeds everything | Spend by job, not only by month |
| Key sprawl | Pipelines often hold their own provider keys outside central control | Keys discovered vs keys issued |
| Missing attribution | Embedding spend lands in a shared data budget | Cost by index and by consuming agent |

On inversion: the 2023 paper "Text Embeddings Reveal (Almost) As Much As Text" showed that a trained model could reconstruct a large share of short input texts exactly from their embeddings. Vector stores therefore deserve the same access controls as the documents they came from; see [controlling agent retrieval from vector databases](/blog/control-ai-agent-retrieval-from-vector-databases).

## How do embedding costs catch budgets out?

Embedding costs scale with corpus size, not user traffic. A single re-indexing run can embed millions of chunks in hours. Common triggers:

1. Switching embedding model (old and new vectors are not comparable, so everything is re-embedded).
2. Changing chunk size or overlap.
3. Adding a large new source system.
4. A bug that re-ingests unchanged documents on every run.

The fourth is the most expensive because it repeats silently. A gateway that logs embedding calls per job makes it visible within a day. See [where AI agent costs leak](/blog/where-ai-agent-costs-leak).

## When is a direct path acceptable?

A direct path, bypassing the gateway, is reasonable when all of these are true:

- The embedding model is self-hosted inside your network, so no data leaves.
- Usage is still logged with job, index and volume.
- Access to the model endpoint is restricted to approved pipelines.
- Spend or compute is attributed to the owning team.

If the embedding model is a third-party API, route it through the gateway.

## What should you log for each embedding call?

- Calling job or agent ID and version.
- Target index or collection.
- Model and endpoint region.
- Number of inputs and tokens.
- Data classification of the source, where known.
- Outcome and latency.

Most embedding APIs follow the OpenAI-compatible `/v1/embeddings` shape, so a gateway that speaks that format can usually take embedding traffic with a base-URL change; see [what an OpenAI-compatible endpoint is](/blog/what-is-an-openai-compatible-endpoint).

## How do you move embedding traffic to the gateway?

1. **Find the callers.** Search pipelines and code for embedding endpoints and provider keys.
2. **Issue gateway keys per pipeline.** One key per job or index, not one shared key.
3. **Switch base URLs.** Start with the largest ingestion job.
4. **Apply data and region rules.** Route by data class to approved providers and regions; see [enforcing data residency for LLM traffic](/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic).
5. **Revoke the old direct keys.** Otherwise traffic drifts back.

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) exposes OpenAI-compatible endpoints with routing, fallback and retry, caching, and cost attribution per agent and per version. Auto-discovery finds models and API keys across AWS, Azure and Google Cloud, which helps surface pipeline keys that bypass the gateway. In the October 2026 benchmark, the gateway added under 0.7 ms at p95 per in-gateway routing method under the stated test conditions. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Are embeddings sensitive data?**
Treat them as sensitive as the source text. Research has shown embeddings can be inverted to recover much of the original input.

**Why did our embedding bill spike?**
Most often a re-index: a new embedding model, changed chunking, a large new source, or a bug re-ingesting unchanged documents.

**Can a self-hosted embedding model skip the gateway?**
Yes, if data stays inside your network and usage is still logged and attributed to an owner.

## Key facts

- Embedding calls send whole documents to a model provider, so they carry data-exposure and residency risk.
- The 2023 paper "Text Embeddings Reveal (Almost) As Much As Text" showed embeddings can be inverted to recover much of the original text.
- Embedding costs scale with corpus size; re-indexing after a model or chunking change re-embeds everything.
- A direct path is acceptable for a self-hosted embedding model inside the network if usage is still logged and attributed.
- Opencontroller's LLM Gateway exposes OpenAI-compatible endpoints with cost attribution per agent and per version.
- Opencontroller pricing starts from $250K per deployment per year.

## Related

- [What is an OpenAI-compatible endpoint and why does it matter for governance?](https://opencontroller.lyzr.com/blog/what-is-an-openai-compatible-endpoint.md)
- [How do you enforce data residency for LLM and AI agent traffic?](https://opencontroller.lyzr.com/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic.md)
- [Where do AI agent costs usually leak?](https://opencontroller.lyzr.com/blog/where-ai-agent-costs-leak.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
