---
title: How do you control what an AI agent can retrieve from a vector database?
url: https://opencontroller.lyzr.com/blog/control-ai-agent-retrieval-from-vector-databases
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# How do you control what an AI agent can retrieve from a vector database?

**Enforce access in the vector store, not in the prompt. Give each agent its own credential, partition data by tenant or classification using namespaces or collections, apply metadata filters on the server based on the end user's identity, and log the IDs of every chunk returned. A direct connection that skips these steps lets any agent read any embedded document.**

Many teams put relational databases behind MCP connectors with query controls, then connect vector databases straight to the agent through a client library. The vector path ends up with fewer controls than the SQL path, even though it often holds the same sensitive documents in a different form. This guide sits under our [guide to agent identity and on-behalf-of delegation](/blog/ai-agent-identity-on-behalf-of-delegation).

## Why is the vector path easy to miss?

Three reasons. First, retrieval feels like reading, not acting, so it gets less review. Second, ingestion pipelines often embed whole document stores with one service account, dropping the original file permissions. Third, the client library call happens inside the agent code, so a gateway in front of the model never sees it.

OWASP lists this as LLM08:2025, Vector and Embedding Weaknesses, in its Top 10 for LLM Applications. The risks it names include unauthorised access, cross-context leakage between users or tenants, and data poisoning through the knowledge base.

## What controls should sit in front of a vector store?

| Control | What it does | Where it is enforced |
|---|---|---|
| Per-agent credential | Ties every query to one agent | Vector database auth |
| Partitioning | Separates tenants or classifications into namespaces, collections or indexes | Vector database |
| Server-side metadata filters | Restricts results to documents the user may see | Query layer, never the model |
| Permission sync at ingestion | Carries source-system access lists into chunk metadata | Ingestion pipeline |
| Result logging | Records chunk and document IDs per query | Retrieval service |
| Write controls | Limits who can add or change embeddings | Vector database and pipeline |

The key word is server-side. If the agent builds the filter itself, a prompt injection can change it. Put a thin retrieval service between the agent and the store that adds filters from the authenticated user's identity and refuses unfiltered queries.

## How do you make retrieval follow user permissions?

Permission-aware retrieval has two halves.

1. **At ingestion,** copy access information from the source system onto each chunk: allowed groups, classification label, tenant ID, data owner. Without this, nothing downstream can filter.
2. **At query time,** resolve the end user's groups from your identity provider and add them as a mandatory filter. When the agent acts on behalf of a user, use the user's identity, not the agent's broader one.

If your vector data lives in PostgreSQL with pgvector, you can use PostgreSQL row-level security policies, so the same database rules apply to similarity search as to any other query. Managed vector databases offer namespaces, collections and metadata filtering. Check whether your product enforces filters before or after the nearest-neighbour search, because post-filtering can return fewer results than requested and push teams to remove the filter.

Keep permissions fresh. When a document's access list changes in the source system, re-sync the metadata. A nightly job is a common starting point, with immediate updates for revocations.

## Are embeddings themselves sensitive?

Yes. Research has shown that text can be partly reconstructed from its embeddings. Treat an embedding with the same classification as the text it came from. That means encryption at rest, the same residency rules, and no copying embeddings of restricted data into shared indexes for convenience.

## What should you log?

For each retrieval, log the agent identity, the user it acted for, the query time, the filters applied and the IDs of returned chunks and their source documents. You do not need the full query text if it holds personal data, but you do need enough to answer one question after an incident: which documents did this agent see when it gave that answer? Our guide to [linking AI agents to the data catalogue](/blog/link-ai-agents-to-data-catalogue) shows how to connect those IDs back to data owners.

## How does this compare with MCP database connectors?

An MCP server in front of a relational database can apply query allowlists, row limits and read-only roles, as covered in our guide to [governing MCP servers that query databases](/blog/govern-mcp-servers-that-query-databases). The same thinking applies to vectors. Either wrap the vector store in a retrieval tool with fixed filters, or enforce the filters in the database. What you should avoid is two standards: strict for SQL and open for embeddings of the same documents.

## How Opencontroller does this

Opencontroller keeps a governed registry of agents, tools and MCP servers, supports delegated on-behalf-of identity and agent workload identity federation, and runs Posture checks that flag risky permissions and configurations. Policies are written as code and can be extended for your data controls. See the [capability reference](/capability-reference) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can I rely on the system prompt to stop an agent reading restricted documents?**
No. Instructions in a prompt are not access control. Enforce filters in the retrieval service or database where the model cannot change them.

**Should each tenant get its own vector index?**
Separate namespaces or collections per tenant are the simplest strong boundary. Shared indexes with tenant filters work, but need tests that prove one tenant cannot retrieve another's data.

**Do embeddings need the same protection as the source text?**
Yes. Text can be partly reconstructed from embeddings, so give them the same classification, encryption and residency rules.

## Key facts

- Enforce vector retrieval permissions in the store or a retrieval service, never in the prompt.
- OWASP Top 10 for LLM Applications 2025 lists LLM08:2025 Vector and Embedding Weaknesses, covering unauthorised access, cross-context leakage and poisoning.
- Copy source-system access lists onto chunks at ingestion and apply the end user's groups as a mandatory filter at query time.
- pgvector runs inside PostgreSQL, so row-level security policies apply to similarity search.
- Log agent, user, filters and returned chunk and document IDs for every retrieval.
- Opencontroller supports delegated on-behalf-of identity, workload identity federation and Posture checks for risky permissions.
- Opencontroller starts from $250K per deployment per year.

## Related

- [How do you govern an MCP server that turns natural language into SQL?](https://opencontroller.lyzr.com/blog/govern-mcp-servers-that-query-databases.md)
- [How do you link AI agents to the data assets in your data catalogue?](https://opencontroller.lyzr.com/blog/link-ai-agents-to-data-catalogue.md)
- [What does least privilege look like for an AI agent?](https://opencontroller.lyzr.com/blog/least-privilege-checklist-for-ai-agents.md)
- [More on Agent identity and access](https://opencontroller.lyzr.com/blog/cluster/agent-identity-and-access)
