---
title: How do you tell when external AI agents are calling your APIs and website?
url: https://opencontroller.lyzr.com/blog/find-ai-agents-calling-your-public-apis
published: 2026-11-17
updated: 2026-11-17
author: Lyzr
---

# How do you tell when external AI agents are calling your APIs and website?

**Look for three kinds of evidence: what the client says it is (user-agent tokens), what it can prove (a Web Bot Auth signature built on RFC 9421 HTTP Message Signatures), and how it behaves. Declared identity alone is weak because anyone can copy a user-agent string. Signatures and published IP ranges turn a claim into something you can verify.**

Most AI governance work looks inward, at [agents your own people run](/blog/how-to-find-shadow-ai-agents). The reverse question matters too. Other companies' crawlers, assistants and browsing agents now fetch your pages and call your APIs, sometimes on behalf of your own customers. You need to know which is which before you can set a policy.

## What kinds of external AI traffic exist?

| Type | What it does | Example declared tokens |
|---|---|---|
| Training crawlers | Collect content for model training | GPTBot, ClaudeBot |
| Search and answer crawlers | Index content for AI search results | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-triggered fetchers | Fetch a page because a user asked an assistant | ChatGPT-User, Claude-User, Perplexity-User |
| Browsing agents | Click, fill forms and complete tasks in a cloud browser | ChatGPT agent (signed requests) |
| API-calling agents | Call your documented APIs with a customer's credentials | Your normal API keys or OAuth tokens |

Note that Google-Extended is a robots.txt control token, not a separate crawler user-agent. Check each vendor's documentation, because names and behaviour change.

## How does Web Bot Auth verify an agent?

Web Bot Auth lets an automated client sign each request so the server can check who sent it. It builds on RFC 9421, HTTP Message Signatures. The current IETF draft, draft-ietf-webbotauth-httpsig-protocol-01, was published on 29 October 2026 and is still an Internet-Draft, not an RFC.

The mechanics are concrete:

- Requests carry `Signature`, `Signature-Input` and `Signature-Agent` headers.
- `Signature-Agent` points to an HTTPS location where the operator publishes its keys.
- Keys sit in a JSON Web Key Set at `/.well-known/http-message-signatures-directory`.
- The signature must carry the tag `web-bot-auth`, a `keyid`, and `created` and `expires` times, and must cover the request's authority or target URI.

OpenAI documents that ChatGPT agent requests use `Signature-Agent: "https://chatgpt.com"` with keys published at `chatgpt.com/.well-known/http-message-signatures-directory`. Several CDNs verify these signatures for you. If you run your own edge, make sure proxies preserve all three headers.

## How do you build a detection process?

1. **Log the right fields.** Keep user-agent, the signature headers, source IP, path, status code and referer for web and API traffic.
2. **Classify declared bots.** Match user-agent tokens against vendors' published lists.
3. **Verify claims.** Check signatures where present. Otherwise compare source IPs with the operator's published ranges. Treat unverified claims as unknown.
4. **Look at behaviour.** Very regular intervals, no asset loading, high page depth per session and odd hours point to automation even without a declaration.
5. **Separate API agents by credential.** On APIs, the key or OAuth client tells you which customer integration is calling. Add a field to your developer terms asking customers to declare agentic use.
6. **Report weekly.** Volume by bot type, top paths and error rates show what AI systems read about you.

## What policy options do you have?

| Option | Use when | Mechanism |
|---|---|---|
| Allow and monitor | You want to appear in AI answers | robots.txt (RFC 9309) allow rules, logging |
| Allow verified only | You want agents but not impostors | Signature or IP verification at the CDN or WAF |
| Rate-limit | Agents are welcome but costly | Per-bot or per-key limits at the gateway |
| Block training, allow search | You want visibility without training use | Per-token robots.txt rules |
| Block | The content or API is not for automation | WAF rules; contractual terms for API users |

robots.txt is advisory. It works only for operators that honour it, so pair it with enforcement where it matters.

## How Opencontroller fits

Opencontroller governs the AI agents your organisation runs: discovery, identity, registry and runtime control. Inbound bot policy for your website belongs at your CDN, WAF or API gateway. The two meet when your own agents call other companies' sites, where egress control matters; see [controlling agent network egress](/blog/control-ai-agent-network-egress). To talk through your estate, [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Can we trust the user-agent string?**
No. It can be spoofed. Verify with a Web Bot Auth signature, published IP ranges, or reverse DNS where supported.

**Is Web Bot Auth a finished standard?**
Not yet. It is an IETF Internet-Draft built on RFC 9421, which is published. Some operators and CDNs already use it.

**Should we block AI crawlers?**
It depends on your goals. Blocking training crawlers while allowing search and user-triggered fetchers keeps you visible in AI answers.

## Key facts

To tell when external AI agents call your APIs and website, combine declared identity (user-agent tokens such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot), cryptographic identity (Web Bot Auth, which uses HTTP Message Signatures from RFC 9421 with Signature, Signature-Input and Signature-Agent headers; draft-ietf-webbotauth-httpsig-protocol-01 was published 29 October 2026), published IP ranges, and behaviour patterns. OpenAI documents that ChatGPT agent requests carry Signature-Agent "https://chatgpt.com" with keys at chatgpt.com/.well-known/http-message-signatures-directory. User-agent strings alone can be spoofed. Opencontroller by Lyzr governs the agents you run; inbound bot policy belongs at your CDN, WAF or API gateway. Pricing starts from $250K per deployment per year.

## Related

- [How do you find MCP servers exposed to the internet?](https://opencontroller.lyzr.com/blog/find-mcp-servers-exposed-to-the-internet.md)
- [How do you detect unapproved LLM API traffic on your network?](https://opencontroller.lyzr.com/blog/detect-unapproved-llm-api-traffic.md)
- [How do you discover employees' AI use in the browser?](https://opencontroller.lyzr.com/blog/how-to-discover-ai-use-in-the-browser.md)
- [More on Discovery and shadow agents](https://opencontroller.lyzr.com/blog/cluster/discovery-and-shadow-agents)
