---
title: What should an LLM gateway log, and what should it never store?
url: https://opencontroller.lyzr.com/blog/llm-gateway-request-logging-and-privacy
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# What should an LLM gateway log, and what should it never store?

**An LLM gateway should log metadata on every request — who called, which agent and version, which model, tokens, cost, latency, status and policy decisions — and treat prompt and response text as sensitive data. Store payloads only where you need them, redacted, sampled, access-controlled and kept for a short period. Never store secrets, full payment card numbers or authentication tokens, even in debug logs.**

The goal is a log that answers cost, security and incident questions without becoming the largest unprotected copy of your company's data.

## Why is gateway logging a privacy problem?

Everything an agent sends to a model passes through the gateway: customer messages, retrieved documents, tool outputs, sometimes credentials pasted by users. Logging all of it by default creates three problems:

1. **A new sensitive data store** that was never classified or reviewed.
2. **Wider access** than the source systems: engineers who can read logs can read HR or customer data they could not open directly.
3. **Retention conflicts.** Data protection laws expect minimisation. The EU GDPR, for example, requires personal data to be "adequate, relevant and limited to what is necessary" (Article 5(1)(c)).

## What should be logged on every request?

This table is a practical default. "Always" fields contain no conversation content.

| Field | Policy | Purpose |
|---|---|---|
| Timestamp, request ID, trace ID | Always | Correlation with agent traces |
| Caller identity (user, workload or key ID) | Always | Accountability |
| Agent name and version | Always | Cost attribution, incident scoping |
| Provider, requested model, served model | Always | Routing audit, fallback analysis |
| Input, output and cached token counts | Always | Cost and capacity |
| Calculated cost | Always | Showback and chargeback |
| Latency and time to first token | Always | Performance |
| Status code and error type | Always | Reliability |
| Retry and fallback events | Always | Provider health |
| Policy decisions (allowed, routed, redacted, blocked) with rule ID | Always | Control evidence |
| Guardrail scores (for example groundedness) | Always | Quality monitoring |
| Prompt and response hashes | Always | Detect duplicates and replay without storing text |
| Prompt text | Sampled or on incident, redacted | Debugging, evaluation |
| Response text | Sampled or on incident, redacted | Debugging, evaluation |
| Tool call arguments and results | Redacted; follow [MCP tool call logging rules](/blog/what-to-log-for-every-mcp-tool-call) | Investigation |
| API keys, OAuth tokens, passwords | Never | Must be stripped before any write |
| Full payment card numbers, government IDs | Never | Mask before logging |

## How should payload logging be configured?

1. **Off by default for production.** Enable payload capture per agent with a recorded reason.
2. **Redact before writing.** Run detectors for secrets, card numbers, national IDs, emails and phone numbers on the way into the log store, not afterwards.
3. **Sample.** For quality evaluation, 1–5% of traffic is often enough. Raise the rate only for a defined investigation.
4. **Separate stores.** Keep metadata and payloads in different stores with different access groups.
5. **Restrict access to payloads.** Named roles only, with every read itself logged.
6. **Respect upstream sensitivity.** If an agent handles data classified as restricted, do not log its payloads at all; rely on metadata and hashes.

## How long should gateway logs be kept?

Use tiers instead of one retention period:

| Tier | Contents | Typical retention |
|---|---|---|
| Hot metadata | All "always" fields | 30–90 days, searchable |
| Archived metadata | Same, compressed | Per your audit requirements, often 1 year or more |
| Sampled payloads | Redacted prompt and response text | 7–30 days |
| Incident payloads | Payloads tied to an open incident | Until the incident closes, under legal hold if needed |

Align these with your records policy and any sector rules. Metadata is usually what auditors and finance need; payloads are mainly for engineers.

## What do security reviewers check?

Expect questions on where logs are stored, whether they leave your region, who can read payloads, how redaction is tested, and how long each tier is kept. The [CISO security review of an LLM gateway](/blog/ciso-security-review-of-an-llm-gateway) lists the full set. Good metadata logs also make [reconciling gateway spend with provider invoices](/blog/reconcile-ai-gateway-spend-with-provider-invoices) straightforward. For where logging sits in the wider architecture, see the [gateway comparison pillar](/blog/llm-gateway-vs-agent-gateway-vs-control-plane).

## How Opencontroller does this

Opencontroller's LLM Gateway (Lyzr Gateway) sits in the request path and can allow, route, redact or block traffic, with cost attribution per agent and per version. It uses open standards including OpenTelemetry and OpenAI-compatible endpoints, so telemetry can flow into tools you already run. It can be deployed as Lyzr SaaS or in customer-controlled infrastructure, which keeps logs where your data policy requires. See the [capability reference](/capability-reference) or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Do we need to log full prompts for audit?**
Usually not. Auditors typically need evidence of who used which model, what policy applied and what it cost. Metadata and policy decisions provide that; payloads are kept for sampled evaluation and incidents.

**Is hashing prompts enough to protect privacy?**
Hashes do not store the text, but short or predictable inputs can be guessed. Use a keyed hash, and treat hashes as internal data.

**How is Opencontroller licensed?**
Annual licence per deployment, not per seat or usage, starting from $250K per deployment per year.

## Key facts

An LLM gateway should always log metadata (caller, agent and version, provider and served model, token counts, cost, latency, status, retries, policy decisions, guardrail scores, prompt hashes) and treat prompt/response text as sensitive: off by default in production, redacted before write, sampled (1–5%), stored separately with restricted access. Never log API keys, OAuth tokens, passwords, full card numbers or government IDs. GDPR Article 5(1)(c) requires data minimisation. Opencontroller LLM Gateway can allow, route, redact or block in the path, supports OpenTelemetry, and deploys as Lyzr SaaS or in customer-controlled infrastructure. Licensing starts from $250K per deployment per year.

## Related

- [What should you log for every MCP tool call?](https://opencontroller.lyzr.com/blog/what-to-log-for-every-mcp-tool-call.md)
- [What does a CISO security review of an LLM gateway cover?](https://opencontroller.lyzr.com/blog/ciso-security-review-of-an-llm-gateway.md)
- [Why does AI spend in the gateway not match the provider invoice?](https://opencontroller.lyzr.com/blog/reconcile-ai-gateway-spend-with-provider-invoices.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
