What are computer-use agents, and when are they the right tool?
- 2 min — mechanics: screenshot → action loop
- 2 min — cost and latency per action; OSWorld numbers
- 3 min — legacy-app example with completion and failure clusters
- 3 min — sandboxing, allow-lists, approval gates
- 2 min — ownership and graduation to APIs
A computer-use agent operates a GUI like a person: the host sends screenshots, the model returns actions — click at (x, y), type, scroll, key combos — and the host executes them. It is tool use where the tool is a whole screen. It works with any app, including ones with no API, at the cost of speed and reliability: each action is a screenshot (~1–1.5k image tokens at typical resolutions) plus a model turn of 2–5 s, so a 30-action task takes 2–3 minutes and tens of thousands of tokens where an API call takes 300 ms.
Reliability has moved fast but is not solved. On OSWorld (Xie et al., 2024), humans complete ~72% of tasks; the best model at release scored ~12%. Frontier models crossed 60% during 2025 — usable for narrow, recorded workflows, still below human for open-ended desktop work.
Data entry into a 2009-era claims-adjudication desktop app with no API: copy 18 fields from a structured intake record into four screens. Recorded as a scenario with fixed screen resolution (1280×800) and run in a VM. On 200 replayed claims: 94% completed, median 41 actions, 3.1 min per claim; failures clustered on a modal that appears only for out-of-state policies. If an API, CLI or MCP server exists for the target, use it — computer use is the fallback for the long tail.
Run it in an isolated sandbox — a VM or container with a throwaway browser profile and no real sessions or credentials — because every page it reads is a prompt-injection channel and the agent can click anything a user can. Gate irreversible actions (submit, pay, delete, send) behind approval, and allow-list domains. Measure completion rate and actions per task on recorded scenarios per release; expect breakage when layouts change, so pin app versions in the scenario VMs.
Org scope. Computer use belongs with the RPA or automation team, not every product team: they own VM images, recorded scenarios, the domain allow-list and the replacement roadmap (every high-volume flow should graduate to an API). Build vs buy: use the model provider's computer-use tool and a managed VM/browser sandbox before building your own harness. Failure modes: pop-ups, CAPTCHAs (never automate past them), slow page loads mistaken for completion, resolution changes; budget retries and keep the blast radius small enough that a misclick is recoverable.
Per action: screenshot ~1.2k tokens + 3k running context (older screenshots dropped) + 150 output. 41 actions ≈ 172k input + 6k output. assume $3 / M input and $15 / M output tokens (illustrative list prices, Oct 2026); cache writes 1.25× input (5-minute TTL), cache reads 0.1×. ≈ $0.52 + $0.09 ≈ $0.61 per claim uncached; with prefix caching and only the last 3 screenshots kept, ≈ $0.30. At 3.1 min per claim, one VM handles ~19 claims/hour.
What the answer sounds like at each level
The model sees screenshots and clicks and types like a person; use it when there's no API.
Quantifies per-action latency and token cost, sandboxes in a VM, gates irreversible actions, and evaluates on recorded scenarios with pinned app versions.
Places computer use with an automation team that owns sandboxes, scenarios and allow-lists, and runs a graduation roadmap from GUI automation to APIs.
Follow-ups the interviewer will push
1The page shows a CAPTCHA. What does the agent do?
Stop and hand off to a human. Automating past bot detection breaks terms of service and is a signal the flow should go through an API or partner integration.
2Now 10× volume on the claims flow. Still computer use?
At ~3 minutes and ~$0.30–0.50 per claim, 10× volume justifies building an integration (even a database-level one with the vendor). Computer use is for the long tail; volume is the graduation trigger.
3How do you evaluate it reproducibly?
Snapshot VMs with fixed app versions and seeded data, replay recorded scenarios, and check end state in the app's database or a final screenshot assertion — not the agent's own claim that it finished.
Red flags interviewers listen for
- Runs the agent on a workstation with real logged-in sessions.
- Prefers computer use where an API exists.
- No approval gate for submit, pay or delete actions.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanics and cost | Per-action latency and image-token cost quantified. |
| Safety | VM isolation, no real credentials, domain allow-list, approval on irreversible actions. |
| Evaluation | Recorded scenarios, pinned versions, end-state checks. |
| Org scope | Automation-team ownership and volume-based graduation to APIs. |
Sources
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Xie et al., 2024paper