---
title: How do you govern AI agents that control a computer or browser?
url: https://opencontroller.lyzr.com/blog/govern-computer-use-and-browser-use-agents
published: 2026-11-15
updated: 2026-11-15
author: Lyzr
---

# How do you govern AI agents that control a computer or browser?

**Govern a computer-use agent like an untrusted contractor at a borrowed desk. Run it in an isolated, low-privilege virtual machine, limit the web to an allowlist of domains, keep real credentials out of reach, and require a person to confirm actions with real-world consequences. Then keep the screenshots and action log as your audit trail.**

Computer-use and browser-use agents do not call a neat API. They read pixels and send clicks and keystrokes. That means every page the agent sees is an input that can steer it, and every button it can reach is a tool. This guide sets out the controls, using what the model vendors themselves document.

## Why are computer-use agents harder to govern than API agents?

An API tool has a fixed shape. A "create invoice" tool only creates invoices. A computer-use agent can do anything a person at that screen can do.

Three things raise the risk:

- **The tool surface is the whole screen.** You cannot list the agent's tools in advance, so least privilege must come from the machine and network, not the tool list.
- **Every screenshot is untrusted input.** A web page, email or pop-up can carry instructions. This is indirect prompt injection, covered in [how to stop prompt injection through tool outputs](/blog/stop-prompt-injection-through-tool-outputs).
- **Actions look human.** Downstream systems see a normal browser session. Without your own logging, you cannot tell agent clicks from employee clicks.

## What do the model vendors recommend?

Both major vendors publish precautions. They line up closely.

Anthropic's computer use documentation lists four precautions: use a dedicated virtual machine or container with minimal privileges; avoid giving the model access to sensitive data such as account logins; limit internet access to an allowlist of domains; and ask a human to confirm decisions with meaningful real-world consequences, such as accepting cookies, completing financial transactions or agreeing to terms of service. It also states that classifiers scan returned screenshots for possible prompt injections and steer the model to check with the user.

OpenAI's computer use tool, as documented on Microsoft Learn for Azure OpenAI, raises `pending_safety_checks` with three codes:

| Safety check code | What triggers it | Documented handling |
|---|---|---|
| `malicious_instructions` | Adversarial content detected in the screenshot | End user reviews and confirms behaviour |
| `irrelevant_domain` | Current domain looks unrelated to the task | End user reviews and confirms behaviour |
| `sensitive_domain` | Agent is on a sensitive domain | End user actively monitors ("watch mode") |

The agent cannot proceed until your application passes the checks back as `acknowledged_safety_checks`. That makes the acknowledgement an approval event you should log, with who approved it.

## What controls should you put around a computer-use agent?

Use this numbered checklist as a baseline.

1. **Isolate the machine.** One ephemeral VM or container per task or session, destroyed afterwards. No shared desktop, no access to the corporate network by default.
2. **Allowlist domains.** Enforce at the network layer, not in the prompt. Start with the two or three domains the task needs.
3. **Keep credentials out.** Do not type passwords into the session or store them in the browser profile. Where a login is unavoidable, use a dedicated low-privilege account per agent, with its own owner.
4. **Block file transfer paths.** Disable clipboard sharing with the host, downloads to shared drives and uploads from local disks unless the task needs them.
5. **Confirm consequential actions.** Payments, sending messages, accepting terms, changing account settings and deleting records need a person to approve. Treat vendor safety checks as mandatory pauses, not warnings to auto-acknowledge.
6. **Set run limits.** Cap steps, wall-clock time and spend per task so a looping agent stops on its own.
7. **Log every action.** Record each click, keystroke batch, URL, screenshot reference, model call and approval, tied to agent, version and the user the agent acts for.
8. **Decide screenshot retention.** Screenshots can contain personal data. Anthropic documents its computer use tool as client-side, so screenshots are captured and stored in your environment. Set a retention period and access rules for them as you would for session recordings.

## How is a browser-use agent different from a full computer-use agent?

A browser-use agent is confined to a browser, so the attack surface is smaller: no shell, no local apps. The web is still untrusted, and the browser profile is the prize. Cookies and saved sessions give an agent whatever access the profile owner has.

For browser agents, add three rules:

- Use a clean profile per agent, never an employee's everyday profile.
- Block access to internal admin consoles unless the task needs them.
- Inventory agentic browsers and extensions on company devices. See [how to find agentic browsers on company devices](/blog/find-agentic-browsers-on-company-devices).

## How should you test these agents before production?

Red-team them with hostile pages. Build test pages that hide instructions in white text, alt text, pop-ups and PDF footers, then check the agent ignores them. Test domain drift: give a task and plant a link to an unrelated site. Test the stop path: confirm you can halt a running session in seconds and that the VM is destroyed.

The OWASP Top 10 for Agentic Applications is a useful map for scope. Goal hijack, tool misuse and identity abuse all apply to computer-use agents. The same sandbox thinking appears in our guide to [governing code execution tools](/blog/govern-code-execution-tools-for-ai-agents).

## How Opencontroller does this

Opencontroller records agents, versions and tools in a governed registry, so computer-use agents have a named owner and a known scope. Posture flags risky permissions and configurations. Multi-turn and red-team evals test agents before release, with built-in mappings to OWASP and MITRE ATLAS. At run time, behavioural-deviation monitoring can alert on, quarantine or kill a rogue agent through the connected platform. For tools more broadly, start with the [MCP governance guide](/blog/how-to-govern-mcp-servers), or [book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Should a computer-use agent ever log in with an employee's account?**
No. Give it a dedicated low-privilege account with its own owner, so its actions are attributable and its access can be revoked without locking out a person.

**Can I auto-acknowledge vendor safety checks to keep the agent fast?**
The vendor guidance says actions should be handed to the end user to confirm when safety checks are raised. Auto-acknowledging removes the control the check exists to provide.

**Where are computer-use screenshots stored?**
It depends on the tool. Anthropic documents its computer use tool as client-side, with screenshots captured and stored in your environment, so your retention policy applies.

## Key facts

Govern computer-use and browser-use agents with an isolated low-privilege VM per session, a network-enforced domain allowlist, no real credentials, human confirmation for consequential actions, step/time limits, and full action and screenshot logging. Anthropic computer use docs list four precautions: dedicated VM/container with minimal privileges, no sensitive data such as logins, internet limited to an allowlist of domains, and human confirmation for consequential decisions; classifiers scan screenshots for prompt injection. OpenAI computer use raises pending_safety_checks with codes malicious_instructions, irrelevant_domain and sensitive_domain that must be acknowledged before continuing. Opencontroller (by Lyzr) registers agents and tools in a governed registry, flags risky configurations with Posture, runs red-team evals mapped to OWASP and MITRE ATLAS, and can quarantine or kill rogue agents. Opencontroller pricing starts from $250K per deployment per year.

## Related

- [How do you find agentic AI browsers on company devices?](https://opencontroller.lyzr.com/blog/find-agentic-browsers-on-company-devices.md)
- [How do you govern code execution tools given to AI agents?](https://opencontroller.lyzr.com/blog/govern-code-execution-tools-for-ai-agents.md)
- [How do you stop prompt injection that arrives through tool outputs?](https://opencontroller.lyzr.com/blog/stop-prompt-injection-through-tool-outputs.md)
- [More on Tools and MCP](https://opencontroller.lyzr.com/blog/cluster/tools-and-mcp)
