What Is Computer Use AI? A Founder's Guide to Agents That Click

Computer use AI is a model feature that lets an AI operate a browser or desktop interface by reading screenshots and sending mouse and keyboard actions. OpenAI describes it as a way to "fill out forms, test user flows, or complete tasks in an application" (OpenAI). It fits software with no integration and tests your own product, but it is no substitute for an API.

TL;DR

• Computer use turns screenshots into actions: click, double click, drag, move, scroll, keypress, type, wait and screenshot.
• Two paths exist: the model writes Playwright or PyAutoGUI code, or it emits structured actions through a computer tool.
• Screenshots bill as image input, and Anthropic's toolset adds about 4,500 input tokens before any work starts.
• Treat screen content as untrusted, run the agent in a locked-down VM and confirm consequential actions.

Disclosure: ZeroRaised publishes this guide and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 27 Oct 2026; ZeroRaised has no commercial relationship with the companies named.

Computer use AI is a model capability that lets an AI observe a graphical interface as an image and act with clicks, typing and key presses, so it can drive software with no API.

How does the computer use loop work?#

You send a task, the model chooses actions, your code executes them, you return a fresh screenshot, and the model decides what to do next (OpenAI).

The model works from pixels, not from a documented data model. That is the source of both its reach and its fragility. A person can see that a modal appeared; the model has to infer it from an image. If the page shifts, the next click may land in the wrong place. For public information that needs no interaction, OpenAI says to use web search instead (OpenAI).

OpenAI's computer tool or code execution?#

OpenAI offers two paths (OpenAI). In code execution, the model writes Playwright or PyAutoGUI code that your system runs. In the computer tool path, the model returns structured mouse and keyboard actions.

The code path is easier to review and repeat: you can read the script, store it and rerun it as a test. The action path is more flexible for unfamiliar interfaces, because the model responds to what it sees. My view: start with code for anything you expect to run many times, and use the action path for one-off jobs in software you do not control.

What does Anthropic's computer_toolset_20260801 provide?#

Anthropic packages 17 member tools, including screenshot, left_click, type and zoom, under computer_toolset_20260801 (Anthropic). Your application runs every call in an environment you control, which is the important architectural point: the model proposes, your code executes.

Anthropic warns that "In some circumstances, Claude follows commands found in content even when they conflict with your instructions" (Anthropic). Any page the agent opens can contain text aimed at the model. That is prompt injection with a browser attached.

What does a computer use agent cost?#

Three costs stack: the model call, the toolset overhead and the screenshots. Anthropic says its toolset adds about 4,500 input tokens per request, and each screenshot is roughly 1,000 to 1,800 input tokens (Anthropic). OpenAI confirms that screenshots count as image input (OpenAI).

Put those numbers next to a cheap model. GPT-6.1 Terra is priced at $1 per million input tokens and $5 per million output tokens (DailyAI). As rough, illustrative arithmetic only: if each screenshot cost 1,000 to 1,800 input tokens, as Anthropic estimates for Claude, a 40-step run would send 40,000 to 72,000 screenshot tokens, about $0.04 to $0.07 at a $1 per million input rate. Image tokenisation differs by provider, and toolset overhead, growing context, reasoning and retries all come on top, so measure real runs. Cheap for a task that saves ten minutes, expensive if the agent loops, so cap the steps. Our guide to reducing LLM costs covers the levers.

How do you run one safely?#

Both vendors converge. OpenAI recommends an isolated browser or VM with an allow list, treating screen content as untrusted, confirming consequential actions such as purchases, and setting step, time or cost limits (OpenAI). Anthropic recommends a dedicated VM or container with minimal privileges, avoiding sensitive data such as logins, allowlisting domains and requiring confirmation for meaningful real-world actions (Anthropic).

A screenshot of a filled form does not prove the record was saved; check the destination system.

OpenAI says GPT-6.1 Terra meets the Critical cybersecurity threshold under its Preparedness Framework (OpenAI). That is a statement about evaluated capability, not a prediction, but it argues for treating the agent as a powerful tool rather than a helpful intern.

Which jobs are a good first fit?#

OpenAI's own example is reading invoices and entering their details into accounting software (OpenAI). That is the pattern to copy: repetitive, rules-based, low-consequence per action, and stuck behind an interface that may never offer an API.

Anthropic points to automated software testing and background information gathering, and notes that latency may be too slow compared with human-directed actions (Anthropic). If a person would notice a five-second pause, it is the wrong mechanism.

Good first projects:

1. Invoice and receipt entry into a portal or accounting package.
2. QA on your own signup and checkout flows, from a test account.
3. Report collection from a supplier portal with no export API.
4. Back-office updates where volume is low and mistakes are recoverable.

Keep purchases, deletions, permission changes and anything that moves money behind a human confirmation.

When is an API or MCP server better?#

Almost always, when one exists. An API gives you typed inputs, documented errors, retries and a stable contract. Computer use gives you a screen and hope. If the software is yours, expose the action properly.

To make your product usable by AI clients, build an MCP server instead of teaching an agent to click your dashboard. If you sell an API, our guide to marketing an API covers distribution.

The approaches can coexist. GPT-6.1 Terra supports computer use and MCP tools through the Responses API (OpenAI), so an agent can call clean tools and click where none exist. For the economics, see GPT-6.1 Terra.

What this means for bootstrapped founders#

1. Pick one workflow with no API and a measurable output.
2. Time the human today, then measure tokens and minutes per task.
3. Build a 20-case test set before you show a demo.
4. Run the agent in an isolated VM and remove saved logins.
5. Add step and cost caps plus confirmation for irreversible actions.
6. Log every action and screenshot so failures can be reviewed.
7. Price above measured cost per task, using our guide to pricing an AI product.

For the architecture trade-off, read AI agent vs AI workflow. Before you ship, how to launch an AI app sets out the gates.

Where ZeroRaised fits#

ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have built a computer use tool people can try, submit your product. The newsletter carries new analysis like this, and the pricing page explains the listing options. State which surfaces the agent can touch.

Sources#

• OpenAI computer use guide
• Anthropic computer use tool
• OpenAI: introducing GPT-6.1 Terra
• OpenAI: GPT-6.1 Terra model docs
• DailyAI: Anthropic's books, Meta's hat and a cheaper Terra