Computer Use Agents in the Enterprise: When API Tools Are Safer

Computer use agents can now operate a screen well enough to be useful. They are still the riskiest way to connect an agent to a system.

My rule is simple: use an API when one exists. Use a screen only when it doesn’t.

This post covers what computer use agents do, how reliable they are with dated numbers, the credential and data risks, how to log screen actions, and where they are a safe fit.

What computer use agents do

A computer use agent looks at screenshots and sends mouse and keyboard actions. It moves the cursor, clicks buttons, types text, and scrolls, the same way a person would. Anthropic introduced the idea in October 2024 and called it experimental, “at times cumbersome and error-prone.” OpenAI’s Operator followed in January 2025 as an early research preview that browsed websites on a user’s behalf.

The appeal is clear. A lot of business software has no API. Legacy portals, vendor sites, and old desktop tools only have a screen. An agent that can use a screen can reach all of it, without a custom integration.

That reach is also the problem. A screen is built for humans, so the agent sees everything a human sees, and anything on the page can try to instruct it.

If you are comparing this to older screen automation, see my post on moving from RPA to AI agents.

Reliability in practice

Can agents use a computer yet?

Yes, for some work. The best way to see it is through OSWorld, a benchmark of real computer tasks.

• The original OSWorld paper, submitted in April 2024, tested 369 tasks. Humans completed over 72% of them. The best model managed 12.24%, mostly failing on GUI grounding (finding the right element) and operational knowledge.
• In October 2024, Anthropic reported 14.9% on the screenshot-only category, and 22.0% with more steps.
• In August 2026, a16z reported on OSWorld-Verified: the best model scored about 42% in early 2025 and 85% in June 2026. Human testers scored about 72% on the same tasks.

Those numbers are dated on purpose. The field moved from 12% to 85% in about two years. Any figure I print will age. Check the current leaderboard before you decide.

But read the 85% carefully. As the a16z authors put it, an 85% score still means 15 of 100 tasks failed. Benchmark tasks are clean and self-contained. Your workflow has expired sessions, pop-ups, slow pages, and layout changes.

Where they fail

The a16z write-up gives two patterns that match what I’d expect in enterprise work:

1. You can’t cross-check the output. Say an agent reads payment terms off a screen. A wrong value looks plausible and passes a visual check. Nobody notices.
2. Nothing signals success or failure at run time. Their example is an insurance portal submission that goes through, but a policy number problem later stalls the claim. The agent never learns about it.

They also say agents are brittle when the work drifts off the runbook, and strong on standardized, repeatable tasks.

Why errors pile up

A screen task is many small steps. Small error rates compound. Here is an illustration, not a measurement. If an agent gets each step right 98% of the time, a 20-step task finishes cleanly about 67% of the time (0.98 to the 20th power). Real per-step rates vary. The point is that long chains of clicks lose reliability fast, and API calls don’t have this problem because one call does the whole job.

Credential and data risks

A screen agent needs access to log in, and it sees what is on screen. That creates four risks.

1. Prompt injection from the page. Anything the agent reads can contain instructions. Anthropic’s computer use documentation warns that instructions on web pages or in images can override your instructions and cause mistakes. It tells builders to isolate the model from sensitive data and actions. It also says classifiers help, but the precautions still matter. OpenAI says the same about its browser agent: prompt injection is unlikely to ever be fully “solved”. Its advice includes limiting logged-in access and carefully reviewing confirmation requests. My post on prompt injection in AI agents explains the attack.

2. Credentials on screen. If the agent types a password, the password is in a screenshot, and the screenshot goes to a model and to your logs. Operator handles this with a takeover mode where the user types sensitive information such as logins or payment details. For an unattended enterprise agent, nobody is there to take over. Use a dedicated account for the agent, per the agent identity guide, and attach the session from outside the model’s view where you can.

3. Over-broad access. A logged-in browser can reach everything that user can reach. OWASP’s Excessive Agency entry names excessive functionality, permissions, and autonomy as root causes of agent harm. A screen agent with a manager’s login has all three. Apply least privilege to the account, not just the prompt.

4. Data in screenshots. Screens show more than the task needs: other customers, balances, personal data. Every screenshot is data you sent to a model provider and stored. Check where your provider retains it before you start.

Gartner took a hard line on one version of this. In December 2025 it published research titled Cybersecurity Must Block AI Browsers for Now. Its summary says CISOs should block agentic browsers until enterprise-ready versions reach general availability. That’s about consumer-style AI browsers, not a purpose-built agent in a locked-down VM. But it tells you where cautious security teams stand.

Audit trail for screen actions

With an API call you log a request and a response. With a screen you log a movie. If you can’t replay what happened, you can’t defend it.

How do I log screen actions?

For each step, record:

Field	Why
Run ID and step number	Order of events
Agent identity and version	Who acted
Screenshot before the action	What the agent saw
Action taken	Click, type, scroll, with target and coordinates
Typed text (masked for secrets)	What was entered
Screenshot after the action	What changed
URL or window title	Where it happened
Model’s stated reason	Why it chose this
Human approval, if any	Who said yes
Timestamp (UTC)	When

Four rules for this log:

1. Mask secrets before storing. Never keep typed passwords or tokens in the log.
2. Handle screenshots as data. They may hold personal or financial data. Set retention, access control, and redaction.
3. Write to a store the agent can’t edit.
4. Add a success check outside the screen. After the run, confirm the result in a second system or by a read-only API call. The a16z example shows why “the click worked” is not proof.

The full field list for any agent is in my post on what to log for an AI agent audit trail.

API-first rule

Here is the rule I’d write into policy:

Use an API, database connection, file transfer, or MCP tool when one exists. Use screen control only when none of those can do the job.

Why API first? An API call is typed, narrow, and logged by design. It returns structured data you can validate. It has a defined permission scope. A screen action is open-ended: the agent can click anything the account can reach.

Use this table to decide.

Situation	Choose	Reason
System has a documented API	API	Typed, scoped, and logged
System offers an MCP server or connector	MCP tool, with review	See the MCP security guide
No API, but it can export a file or send a report	File export or scheduled report	Structured and read-only
No API, can be built cheaply by IT	Build a thin API	Pays back fast on repeated work
No API, legacy UI, stable layout, low risk	Computer use, in a sandbox	Screen is the only way
No API, and the action moves money or is irreversible	Computer use only with a human approval step	Confirm each consequential action
Public web page you don’t control	Avoid, or read-only in isolation	Injection risk is highest here

If the work repeats, a thin API in front of a legacy system is often cheaper than a screen agent over time. You build it once. It does not break when a button moves.

Safe uses

Computer use is a good fit when the task is stable, low stakes, and checkable. These are the patterns I’d approve:

• Legacy data entry with a second check. The agent keys data into an old system with no API. A read-back compares what was entered to the source.
• Read-only lookups. Pulling a status or a number from a vendor portal into your own system.
• Test automation. Driving your own app’s UI in a staging environment.
• Bridging while an integration is built. A temporary screen agent, with a firm end date.
• Internal tools with no external content. Pages you control carry less injection risk.

Run each of these inside a sandbox: a separate VM or container, a dedicated low-privilege account, an allowlist of sites, and no stored personal credentials. My post on the AI agent sandbox covers the setup. Confirm consequential actions before they run. Anthropic’s docs note that a batch of actions can finish a multistep action in one turn, so the human check has to come before each block runs.

I’d hold off on these:

• Anything that touches payments, payroll, or regulated records without approval.
• Open web browsing with a logged-in session that has real privileges.
• Any task where you can’t verify the result somewhere else.
• Customer-facing actions that can’t be undone.

Checklist

Before you approve a computer use agent, answer yes to all:

• I checked for an API, file export, MCP tool, or thin integration, and none fits.
• The agent runs in an isolated VM or container.
• It uses a dedicated account with least privilege, not a person’s login.
• Secrets are injected outside the model’s view or typed by a human.
• Sites are on an allowlist.
• The agent reads no untrusted pages, or does so read-only and isolated.
• Consequential actions need human approval before they run.
• Every step logs before and after screenshots, the action, and the agent identity.
• Screenshots have retention, access control, and redaction rules.
• A separate check confirms the result in another system.
• There is an owner, a kill switch, and an end date or review date.
• I tested on at least 30 real cases, including bad layouts and timeouts.

Bottom Line

Computer use agents are real and improving fast. Reliability still depends on your workflow, and a screen gives the agent far more reach than it needs.

Default to APIs. Allow screens only where nothing else works, inside a sandbox, with a limited account, approvals, and full logging.

Next step: list the systems your team wants to automate. Mark which have an API. The ones without are your short list for computer use.

FAQ

Can agents use a computer yet? Yes, for many tasks. On the OSWorld-Verified benchmark, a16z reported the best model scoring about 85% in June 2026, against about 42% in early 2025 and roughly 72% for humans. Still, a score of 85% means 15 in 100 tasks fail, and real workflows are messier than benchmarks.

What are the risks of browser agents? The main risks are prompt injection from page content, credentials appearing in screenshots, accounts with too much access, and sensitive data captured in screenshots. OpenAI says prompt injection is unlikely to ever be fully solved, so isolation and approvals matter.

When should I use an API instead? Use an API whenever one exists. APIs return structured data, have defined permissions, and log cleanly. Use a screen only when no API, file export, or MCP tool can do the job, or when building a thin API is not worth the cost.

How do I log screen actions? Log each step with before and after screenshots, the action, the target, the agent identity, the model’s reason, any human approval, and a timestamp. Mask typed secrets, protect screenshots like any sensitive data, and verify the final result in a separate system.

What is a safe use? Stable, low-stakes, checkable work: legacy data entry with a read-back check, read-only lookups, test automation in staging, and short-term bridges while an integration is built. Run it in a sandbox with a limited account.

Should I block AI browsers? Gartner advised in December 2025 that security leaders block agentic AI browsers until enterprise-ready versions are generally available. A purpose-built agent in an isolated, locked-down environment is a different case, but your security team should decide with that guidance in view.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect