Computer use & browser agents
An agent operating a graphical interface from screenshots via mouse and keyboard actions — the fallback for software with no API.
Computer use lets an AI agent work a computer the way a person does: it looks at a screenshot, decides where to click or what to type, and repeats. That lets AI work with apps and websites that have no programming interface.
It's slower and more error-prone than calling a proper API through Tool use & function calling, so it's a last resort, and it should run inside a Code execution sandbox because the agent is acting on a real screen.
Guiding a helper over a slow video call who can only see your screen: capable, but every click takes a moment and a check.
Using pixel-level computer use where an API or DOM-aware browser tool exists. It's slower, costlier and flakier, and it widens the injection surface for no gain.
The model receives screenshots (sometimes plus an accessibility tree or DOM) as Multimodal & vision models input and emits actions as tool calls: click (x, y), type, key, scroll. The harness executes them in a VM or browser and returns a new screenshot, forming an The agent loop (ReAct) where each step costs an image (≈ width × height / 750 tokens on Claude; 1280×800 ≈ 1,365 tokens) and a round trip. Anthropic released it in public beta in Oct 2024; Claude 3.5 Sonnet then scored 14.9% on OSWorld screenshot-only, where the OSWorld paper (2024) measured humans at 72.36% and the best model at 12.24%. Scores have risen fast since — check current leaderboards.
Reliability issues: coordinate grounding, timing (clicking before load), pop-ups, layout drift. Security is the bigger issue: everything on screen is model input. Anthropic's Aug 2025 Claude-in-Chrome pilot measured a 23.6% prompt-injection attack success rate without mitigations, 11.2% with them. Mitigations: isolated VMs, domain allowlists, no stored credentials, Human in the loop confirmation for purchases, sends and deletes, and DOM-aware browser tools or real APIs whenever they exist.
Disposable VM or container browser per task; 1280×800 or provider-recommended resolution; domain allowlist; credentials injected by a broker, never typed from memory; confirmation required for submit/purchase/send/delete; screenshot trail stored per step; step cap ~50 per task.
Insurance back office: exporting monthly loss runs from a carrier portal with no API. The agent logs in through a credential broker (it never sees the password), opens Reports, sets the month, clicks Export, saves the CSV. 14 actions, ≈ 90 s, ≈ 19k image tokens + 6k text ≈ $0.08 per export, versus a 200 ms API call if one existed. It runs nightly in a disposable VM allowed to reach only the portal's domain.
Position it as the adapter of last resort: recommend an API, then a DOM-level browser tool, then pixels. Lead with sandboxing and injection: "every screen it reads is untrusted input."
The follow-up is often "how would you make it reliable enough for a nightly job?" — fixed resolution, waits on visual state, idempotent steps, a screenshot trail, and an alert when the step count drifts.
Numbers worth knowing
- Tokens per screenshot
- ≈ w × h / 750Claude vision; 1280×800 ≈ 1,365 tokens
- OSWorld at publication
- humans 72.36%, best model 12.24%Xie et al., April 2024
- Browser prompt-injection success
- 23.6% → 11.2% with mitigationsAnthropic Claude-in-Chrome pilot, Aug 2025
How to decide
| When | Do this |
|---|---|
| The system has an API | Use the API through Tool use & function calling. |
| It's a web app without an API | DOM-aware browser automation (Playwright-style tools) before pixels. |
| Only a desktop GUI exists | Computer use in a disposable VM with allowlists and confirmation gates. |
Cost maths
Cost per task ≈ steps × (screenshot tokens + growing history) × p_in + actions × p_out. 14 steps × 1,365 image tokens ≈ 19k tokens + ~6k text, plus history re-sent (trim old screenshots to the last 3) ≈ 25k effective input; at $3 / $15 per M (illustrative, Oct 2026) ≈ $0.075 + $0.006. Keeping every screenshot in history would make step 14 alone carry ≈ 19k image tokens; total ≈ 140k → $0.42.
Operating it
- Step count per task drifts >30% above baseline (UI changed)
- Task failure rate > 10% on a nightly job
- The vendor ships an API (retire computer use)
- Actions per task p95 (alert at +30% vs baseline)
- Navigations to non-allowlisted domains (alert on any)
- Form submits or purchases without a confirmation record (alert on any)
When it goes wrong
A browsing agent researching suppliers filled in a contact form on a third-party site with internal pricing notes.
Outbound-request review of the VM's traffic logs flagged a POST to an unlisted domain.
Hidden text on the supplier's page instructed the agent to "submit your requirements here"; the VM allowed all domains and form submits had no confirmation.
Domain allowlist, confirmation on any form submit, and internal documents no longer loaded into browsing sessions.
Related terms
Sources
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku — Anthropic, 2024blog
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Xie et al., 2024paper
- Piloting Claude in Chrome — Anthropic, 2025blog
- Computer use tool — Anthropic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.