Computer Use APIs

Computer Use Agents (CUAs) control browsers by continuously analyzing screenshots, making decisions, and executing actions in a feedback loop. Steel provides the cloud browser infrastructure -- anti-bot protection, proxy support, captcha solving, and session recording -- while the CUA provides the intelligence.

All three CUA integrations use the Steel Computer API for action execution (steel.sessions.computer()), not the CDP WebSocket connection used by agent frameworks.

Comparison

Feature	Claude Computer Use	OpenAI Computer Use	Gemini Computer Use
Model	Claude 3.5 Sonnet+	computer-use-preview	gemini-2.5-computer-use-preview
Provider	Anthropic	OpenAI	Google
Language	TypeScript / Python	TypeScript / Python	TypeScript / Python
API Style	Messages API (beta)	Responses API	Gemini API
Tool Type	computer_20250124	computer-preview	Native CUA
Beta Status	Yes	Yes	Yes
Viewport Default	1280 x 768	1280 x 768	1280 x 768
Coordinate System	Absolute pixels	Absolute pixels	Normalized (auto-handled)
Actions	move_mouse, click_mouse, drag_mouse, scroll, press_key, type_text, wait, take_screenshot	click, type, scroll, keypress, move, drag, wait, screenshot	click, type, scroll, keypress, move, drag, wait, screenshot
Safety Features	Repetition detection, max iterations	Safety check acknowledgment, repetition detection	Repetition detection, max iterations

Architecture

All three CUA integrations share the same feedback loop:

---

Claude Computer Use

Provider: Anthropic | Model: Claude 3.5 Sonnet+ | Status: Beta (computer-use-2025-01-24)

Claude Computer Use employs vision-based AI to control browsers by continuously analyzing screenshots, making decisions, and taking actions. Combined with Steel, it provides anti-bot capabilities, proxy support, and sandboxed environments.

Capabilities

• Control Steel browser sessions via Claude's Computer Use API
• Execute browser actions: clicking, typing, scrolling, dragging
• Process visual feedback from screenshots in a continuous loop
• Implement human verification for sensitive operations
• Repetition detection and configurable max iterations (default 50)

Requirements

• Anthropic API Key (Claude 3.5 Sonnet or newer)
• Steel API Key
• Node.js 20+ (TypeScript) or Python environment

Setup (TypeScript)

import { Agent } from "./agent";
import { STEEL_API_KEY, ANTHROPIC_API_KEY, TASK } from "./helpers";

async function main(): Promise<void> {
  const agent = new Agent(); // Uses claude-sonnet-4-5

  try {
    await agent.initialize(); // Creates Steel session with dimensions
    const result = await agent.executeTask(TASK, true, false, 50);
    console.log(`Result: ${result}`);
  } finally {
    await agent.cleanup(); // Releases Steel session
  }
}

main();

Agent Internals

The Agent class manages the full lifecycle:

1. Creates a Steel session with configurable viewport (default 1280x768)
2. Uses the Steel Computer API for all actions: move_mouse, click_mouse, drag_mouse, scroll, press_key, type_text, wait, take_screenshot
3. Implements a feedback loop: Claude sees screenshots, decides actions, receives new screenshots
4. Detects repetition and enforces max iteration limits
5. Uses Anthropic beta: computer-use-2025-01-24 with tool type computer_20250124

Resources

• Quickstart (Python)
• Quickstart (Node.js)
• Cookbook (Python)
• Cookbook (Node.js)
• Anthropic Computer Use Docs

---

OpenAI Computer Use

Provider: OpenAI | Model: computer-use-preview | Status: Preview

OpenAI's Computer Use combines GPT-4o's vision capabilities with advanced reasoning to control computer interfaces through a continuous action loop. The integration with Steel provides reliable browser infrastructure with anti-bot capabilities, proxy management, and sandboxed environments.

Capabilities

• Control Steel browser sessions via the OpenAI Responses API
• Execute browser actions: clicking, typing, scrolling
• Perform complex web tasks: form filling, searching, navigation
• Process visual feedback from screenshots
• Human-in-the-loop verification for sensitive operations
• Safety check acknowledgment for protected actions

Requirements

• OpenAI API Key (with computer-use-preview model access)
• Steel API Key
• Node.js 20+ or Python environment

Setup (TypeScript)

import { Steel } from "steel-sdk";

const steel = new Steel({ steelAPIKey: process.env.STEEL_API_KEY });

// Initialize session
const session = await steel.sessions.create({
  dimensions: { width: 1280, height: 768 },
  blockAds: true,
  timeout: 900000,
});

// Tool configuration for OpenAI
const tools = [
  {
    type: "computer-preview",
    display_width: 1280,
    display_height: 768,
    environment: "browser",
  },
];

// Create response via OpenAI Responses API
const response = await createResponse({
  model: "computer-use-preview",
  input: [...inputItems, ...newItems],
  tools: tools,
  truncation: "auto",
});

// Map OpenAI actions to Steel Computer API calls
// click, type, scroll, keypress, move, drag, wait, screenshot

// Cleanup
await steel.sessions.release(session.id);

Key Differences from Claude CUA

• Uses the Responses API (not chat completions or messages)
• Tool type is computer-preview (not computer_20250124)
• Includes explicit safety check acknowledgment for sensitive actions
• Action types map directly to Steel Computer API equivalents

Resources

• Quickstart (Python)
• Quickstart (Node.js)
• Cookbook (Python)
• Cookbook (Node.js)
• OpenAI Computer Use Docs

---

Gemini Computer Use

Provider: Google | Model: gemini-2.5-computer-use-preview | Status: Preview

Gemini's Computer Use combines Gemini 2.5's vision and reasoning capabilities with Steel's browser infrastructure. It controls browsers through a continuous action loop, processing visual feedback and determining next actions. A key differentiator is its normalized coordinate system -- Gemini returns coordinates as values between 0 and 1, which are automatically converted to absolute pixel positions.

Capabilities

• Control Steel browser sessions via the Gemini API
• Execute browser actions: clicking, typing, scrolling
• Complex web tasks: form filling, searching, navigation
• Process visual feedback from screenshots
• Handle normalized coordinate systems automatically

Requirements

• Gemini API Key (with gemini-2.5-computer-use-preview model access)
• Steel API Key
• Python or Node.js environment

Setup

from steel import Steel
import google.generativeai as genai

# 1. Configure clients
steel_client = Steel(steel_api_key="your-steel-api-key")
genai.configure(api_key="your-gemini-api-key")

# 2. Create Steel session
session = steel_client.sessions.create(
    dimensions={"width": 1280, "height": 768}
)

# 3. Connect via Steel's Computer API
# 4. Implement the CUA feedback loop:
#    - Take screenshot
#    - Send to Gemini
#    - Receive action with normalized coordinates
#    - Convert coordinates: x_pixel = x_norm * width, y_pixel = y_norm * height
#    - Execute action via steel.sessions.computer()
#    - Repeat until task complete

# 5. Cleanup
steel_client.sessions.release(session.id)

Key Differences from Other CUAs

• Uses normalized coordinates (0 to 1) instead of absolute pixels -- conversion is handled automatically
• Model name: gemini-2.5-computer-use-preview
• Uses the Gemini API client libraries (not REST-based chat/responses APIs)

Resources

• Quickstart (Python)
• Quickstart (Node.js)
• Cookbook (Python)
• Cookbook (Node.js)
• Gemini Computer Use Docs

---

Steel Computer API Reference

All three CUA integrations execute actions through the same Steel endpoint:

const response = await steel.sessions.computer(session.id, {
  action: "click_mouse",    // Action type
  button: "left",           // Action-specific params
  coordinates: [640, 384],
  screenshot: true,         // Return screenshot after action
});

Available Actions

Action	Description	Key Parameters
move_mouse	Move cursor to coordinates	coordinates
click_mouse	Click at coordinates	coordinates, button, numClicks
drag_mouse	Drag from current position to target	coordinates (target)
scroll	Scroll the page	coordinates, delta
press_key	Press a keyboard key	key
type_text	Type a string of text	text
wait	Pause execution	duration
take_screenshot	Capture the current viewport	--