---
title: What happens when an AI agent invents a tool or its arguments?
url: https://opencontroller.lyzr.com/blog/validate-ai-agent-tool-calls-against-schemas
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# What happens when an AI agent invents a tool or its arguments?

**When an agent invents a tool name or passes malformed arguments, the call either fails, gets silently coerced, or—worst case—succeeds with the wrong values. Strict tool schemas (OpenAI Structured Outputs and Anthropic strict tool use) now constrain the model to valid tool names and schema-valid arguments. They do not check whether the arguments make sense, so you still need server-side validation, an allowlist of tools per agent, and a log of every rejected call.**

This post explains what each layer catches, what it misses, and what to record.

## Why do agents hallucinate tool calls?

A model produces a tool call the same way it produces text: by predicting tokens. Without constraints, three failure modes appear in production:

1. **Invented tools.** The agent calls `refund_order` when only `get_order` and `create_ticket` exist. This often follows a prompt that describes a capability the agent does not have.
2. **Malformed arguments.** A field typed as an integer arrives as `"two"`, a required field is missing, or an extra field appears.
3. **Plausible but wrong values.** The schema is satisfied, but the order ID belongs to another customer, the date is in the past, or the amount is ten times too high.

The first two are format problems. The third is a business-logic problem, and no schema solves it.

## What do strict tool schemas actually guarantee?

Both major model APIs now offer grammar-constrained tool calling.

| Provider | How to enable | What it guarantees | What it does not do |
|---|---|---|---|
| Anthropic (Messages API) | `"strict": true` on the tool definition, with `additionalProperties: false` in `input_schema` | Tool `input` follows the schema; tool `name` is always one you provided (or a server tool) | Check domain rules; computer-use and browser-use toolsets reject `strict` |
| OpenAI (Structured Outputs for function calling) | `"strict": true` on the function definition | Completed, non-refusal output conforms to the supplied JSON Schema | Handle refusals, incomplete responses or business rules for you |

Anthropic's [strict tool use documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use) gives a clear example: a booking system needs `passengers: int`; without strict mode the model might send `"two"` or `"2"`, and with it the response contains `2`. OpenAI's [Structured Outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs) states that applications must still validate business rules the schema does not express.

Two caveats matter for governance:

- **Only the JSON Schema subset is enforced.** Constraints outside the supported subset may be ignored or rejected. Check the provider's schema limitations before relying on, say, a `pattern` or numeric range.
- **Strict mode lives in the client request.** If one team sets it and another does not, you have inconsistent protection across the estate. That is a configuration posture issue, not a model issue.

## What should the tool server check anyway?

Treat every tool call as untrusted input from a caller you did not write. A minimal server-side checklist:

1. **Authenticate the caller** and resolve which agent and which user it acts for.
2. **Check the tool is allowed for that agent.** An allowlist per agent stops an agent from reaching tools it was never approved for, even if they exist on the same server. See [how to limit which tools each AI agent can call](/blog/limit-which-tools-each-ai-agent-can-call).
3. **Re-validate the schema on the server.** Never assume the client used strict mode.
4. **Validate semantics.** Does the record exist? Does the user own it? Is the amount within a limit? Is the date in range?
5. **Reject with a useful error.** Return a structured error the agent can act on (for example, "order_id not found for this customer") rather than a stack trace. Clear errors reduce retry loops.
6. **Cap retries.** An agent that keeps sending the same bad call is a looping risk; see [how to stop an AI agent from looping](/blog/stop-an-ai-agent-from-looping).

For write actions with real-world effects, add a validation step before execution, as described in [validate AI agent output before it acts](/blog/validate-ai-agent-output-before-it-acts).

## What should you log when a tool call is rejected?

Rejected calls are a high-signal security and quality feed. Log at least:

| Field | Why it matters |
|---|---|
| Agent ID and version | Ties failures to a specific release or prompt change |
| Acting user (on-behalf-of identity) | Shows whose authority the call used |
| Tool name requested | Reveals invented tools and probing for unapproved ones |
| Rejection reason (schema, allowlist, semantic) | Separates model quality issues from policy violations |
| Argument hash or redacted arguments | Supports investigation without storing sensitive data in clear |
| Retry count | Detects loops |
| Trace ID | Links to the full run via OpenTelemetry |

A spike in "tool not allowed" rejections after a release usually means a prompt or tool description changed. A spike from one agent with no release is a behavioural deviation worth investigating—possibly prompt injection steering the agent toward tools it should not use. The [OWASP Top 10 for Agentic Applications](/blog/owasp-top-10-agentic-applications-evidence) treats tool misuse as a core agent risk.

## How Opencontroller does this

Opencontroller keeps agents, versions, tools and MCP servers in a governed registry, so the set of tools each agent may use is recorded centrally. Policies are written as code, and the Agent Gateway and LLM Gateway put control in the traffic path (allow, route, redact, block) where traffic flows through them. Behavioural-deviation monitoring raises alerts when an agent's pattern of calls changes, and per-agent Block and Quarantine controls act through the connected platform. For the escalation path, see the pillar on [how to stop a rogue AI agent](/blog/how-to-stop-a-rogue-ai-agent), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Does strict mode stop an agent from calling a tool it should not use?**
No. It ensures the tool name is one you supplied in that request. If a risky tool is in the list, the agent can still call it. Limit the tool list per agent and enforce the allowlist on the server.

**If I enable strict mode, can I drop server-side validation?**
No. Strict mode is a client-side setting that a different caller may not use, and it does not check business rules such as ownership, limits or dates.

**Are rejected tool calls a security signal?**
Yes. Repeated attempts to call unapproved or non-existent tools can indicate prompt injection or a broken release, and should feed your monitoring.

## Key facts

- Anthropic strict tool use: setting "strict": true on a tool definition constrains sampling so tool input follows the input_schema and the tool name is always valid; computer-use and browser-use toolsets reject strict.
- OpenAI Structured Outputs constrains completed, non-refusal output to the supplied JSON Schema; apps must still validate business rules.
- Strict schemas fix format errors, not plausible-but-wrong values such as another customer's order ID.
- Tool servers should re-validate schemas, enforce a per-agent tool allowlist, check semantics, cap retries and log rejections.
- Opencontroller keeps agents, tools and MCP servers in a governed registry, applies policy as code in the gateway path, and offers behavioural-deviation monitoring with per-agent Block and Quarantine.
- Opencontroller pricing starts from $250K per deployment per year (annual licence per deployment).

## Related

- [How do you validate an AI agent's output before it acts?](https://opencontroller.lyzr.com/blog/validate-ai-agent-output-before-it-acts.md)
- [How do you limit which tools each AI agent can call?](https://opencontroller.lyzr.com/blog/limit-which-tools-each-ai-agent-can-call.md)
- [How do you stop an AI agent from looping?](https://opencontroller.lyzr.com/blog/stop-an-ai-agent-from-looping.md)
- [More on Runtime control](https://opencontroller.lyzr.com/blog/cluster/runtime-control)
