Computer use is the most misunderstood feature in Claude's toolkit. The demos look like magic — Claude reads the screen, clicks buttons, types in forms, navigates websites. People's first reaction is usually either "this will replace UI automation tools" or "this will break on anything real." Both are wrong, but the second is closer to the truth in the short term.
What computer use actually is: a set of tools Claude can call that interact with a virtual display — take a screenshot, move the mouse, click, type. Claude looks at the screenshot, reasons about what's on screen, decides what action to take, and repeats. It's the observe-think-act loop applied to a desktop environment instead of a filesystem.
Understanding that loop makes the failure modes predictable. It also makes the genuinely useful applications clear. This is what I found when I spent time with it.
Quick summary
- Computer use gives Claude three tools:
screenshot,mouse_move/click, andtype. - Claude navigates by reading screenshots, not by parsing DOM or accessibility trees.
- It's slower and less reliable than API-based automation for anything you can reach via an API.
- It's genuinely useful for legacy UIs, apps with no API, and filling gaps that structured tools can't reach.
- Running computer use requires a sandboxed desktop environment — don't let it run against your real desktop.
What computer use actually gives you
The feature ships as three tool definitions you pass to the API:
const tools: Anthropic.Tool[] = [
{ type: "computer_20250124", name: "computer", display_width_px: 1280, display_height_px: 800 },
{ type: "text_editor_20250124", name: "str_replace_editor" },
{ type: "bash_20250124", name: "bash" },
];
The computer tool has three actions Claude can call:
screenshot— capture the current display statemouse_move/left_click/right_click/double_click— position and clicktype— type text into the focused element
That's the entire surface. Claude does everything by composing these primitives. Clicking a button requires: take a screenshot, locate the button visually, calculate its coordinates, call left_click with those coordinates.
The str_replace_editor and bash tools give it text editing and shell access — those are the parts that work most reliably because they don't depend on visual accuracy.
A minimal working setup
Computer use requires a virtual display. Don't run it against your local machine's desktop — one bad coordinate and it clicks something you didn't expect. The standard setup is a Docker container with a virtual framebuffer:
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
xvfb \
x11vnc \
xdotool \
python3 \
python3-pip \
firefox \
&& rm -rf /var/lib/apt/lists/*
RUN pip3 install anthropic
# Start Xvfb on display :1
ENV DISPLAY=:1
RUN Xvfb :1 -screen 0 1280x800x24 &
CMD ["bash"]
For local development, Anthropic's reference implementation (available on GitHub as anthropic-quickstarts) bundles this setup with a web-based VNC viewer so you can watch Claude work in a browser.
Once you have the display running, here's the basic API call pattern:
import Anthropic from "@anthropic-ai/sdk";
import { execSync } from "child_process";
import fs from "fs";
const client = new Anthropic();
async function takeScreenshot(): Promise<string> {
// Capture the display and convert to base64
execSync("scrot /tmp/screenshot.png");
const buffer = fs.readFileSync("/tmp/screenshot.png");
return buffer.toString("base64");
}
async function runComputerUseTask(task: string) {
const messages: Anthropic.MessageParam[] = [];
// Start with a screenshot so Claude can see the current state
const initialScreenshot = await takeScreenshot();
messages.push({
role: "user",
content: [
{
type: "image",
source: { type: "base64", media_type: "image/png", data: initialScreenshot },
},
{ type: "text", text: task },
],
});
const MAX_ITERATIONS = 50;
let i = 0;
while (i++ < MAX_ITERATIONS) {
const response = await client.messages.create({
model: "claude-opus-5-20260101", // Computer use requires a capable model
max_tokens: 4096,
tools: [
{
type: "computer_20250124",
name: "computer",
display_width_px: 1280,
display_height_px: 800,
},
],
messages,
});
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason === "end_turn") {
return response.content.find((b) => b.type === "text")?.text ?? "";
}
// Execute any computer actions and feed back screenshots
const results: Anthropic.ToolResultBlockParam[] = [];
for (const block of response.content) {
if (block.type === "tool_use" && block.name === "computer") {
const action = (block.input as { action: string }).action;
await executeComputerAction(block.input);
// After each action, take a screenshot so Claude can see the result
const screenshot = await takeScreenshot();
results.push({
type: "tool_result",
tool_use_id: block.id,
content: [
{
type: "image",
source: { type: "base64", media_type: "image/png", data: screenshot },
},
],
});
}
}
messages.push({ role: "user", content: results });
}
}
The key pattern: after every action, take a screenshot and include it as the tool result. Claude uses that to verify the action had the intended effect (did the button actually click? did the form submit? did the page load?) and decide what to do next.
Where it works well
Legacy web apps with no public API. If you're automating a government portal, an enterprise dashboard from 2008, or any system that has no machine-readable interface, computer use is sometimes the only option. It handles pagination, form submission, login flows, and data extraction from tables — all the things you'd use Selenium for, but without needing selectors that break when the page changes.
App workflows that cross application boundaries. Copy data from one desktop app to another, take a screenshot and annotate it, read a PDF and populate a web form. These multi-application workflows are awkward to automate with traditional tools; computer use handles them naturally because it's operating at the display layer.
Testing visual behavior. Verifying that a UI element appears in the right place, that a modal opens correctly, that a tooltip shows on hover — these are the tests that are tedious to write and easy for Claude to run visually.
One-off data extraction. If you need to scrape something once from a paginated UI that doesn't offer an export, running a computer use agent for 20 minutes is faster than building a custom scraper.
Where it struggles
Precision clicking. Claude calculates coordinates from the screenshot and passes them as pixel values. On dense UIs with small click targets — toolbar icons, calendar date cells, dropdown menu items — it misses more than you'd expect. A three-pixel error can click the wrong option.
Dynamic content. If a page reloads, redirects, or changes layout after an action, Claude needs a new screenshot to understand the new state. If the action that triggers the change (a form submit that shows a loading spinner) takes a few seconds, timing matters. Claude doesn't inherently know to wait.
Speed. Each iteration is: take screenshot → send to API → get response → execute action → repeat. At 2-5 seconds per iteration, a 20-step workflow takes a minute or two. API-based automation or deterministic scripts are much faster for anything with a machine-readable interface.
Reliability on novel layouts. Claude is good at recognizing common UI patterns — login forms, navigation menus, data tables. It's less reliable on custom components and unusual layouts. If the UI looks like something Claude hasn't seen in training, expect more retries.
The right framing: a fallback, not a replacement
Computer use makes the most sense as a fallback layer — the thing you reach for when structured automation options don't exist. The priority order for automation tasks:
- Direct API — fastest, most reliable, deterministic
- Playwright or Selenium with explicit selectors — fast, reliable within the scope of stable selectors
- Computer use — flexible, works without an API or stable selectors, slower and less reliable
Most automation tasks belong in category 1 or 2. Computer use earns its place in category 3 — for legacy UIs, cross-app workflows, and one-off tasks where building a proper integration isn't worth it.
Running it safely
A few things worth getting right before you run computer use against anything:
Always use a sandboxed environment. A Docker container with a virtual display is the standard. Never point computer use at your real desktop — it can move your actual mouse and type into your actual apps.
Rate-limit and timeout. Computer use tasks can run a long time if not constrained. Set a max iteration count (50 is reasonable for most tasks) and a timeout per-iteration. Kill the process if it hasn't finished in the expected time.
Log every screenshot. When something goes wrong (and it will), you need to see what Claude saw at each step. Save screenshots to disk during the run. Reviewing them after the fact is how you diagnose "why did it click there?"
Don't give it credentials for production systems in demos. Use test accounts. Computer use makes real web requests and submits real forms. A task that goes sideways with production credentials can cause real damage.
What's improving
The main limitations — coordinate precision, timing sensitivity, and reliability on unusual layouts — are all active areas of development. The model gets better at visual reasoning with each release, and tooling for sandboxed execution environments is maturing. The direction is toward something more like "software QA automation that works without selectors" than "general purpose desktop control."
For now, the useful mental model is: if you can do it with an API, do it with an API. If you can do it with Playwright selectors, use Playwright. If neither is available and the task is worth automating at all, computer use is the option left.
[INTERNAL-LINK: agentic loop fundamentals → building your first AI agent with agentic loops] [INTERNAL-LINK: VS Code and terminal integration → integrating Claude into your development workflow]