Usage

Basic run (default: gpt-4o)

operate

On first run you will be prompted to enter your OpenAI API key. After that, enter your objective at the prompt:

Objective: Open a browser, go to google.com, and search for "self-operating computer"

The framework screenshots your screen, sends the image to the model, and then executes the mouse/keyboard actions the model returns. It loops until the objective is complete or it determines it cannot proceed.

Select a different model

# OpenAI gpt-4o with OCR overlay
operate -m gpt-4-with-ocr

# OpenAI o1
operate -m o1-with-ocr

# OpenAI gpt-4.1
operate -m gpt-4.1-with-ocr

# Anthropic Claude 3
operate -m claude-3

# Google Gemini Pro Vision
operate -m gemini-pro-vision

# Alibaba Qwen-VL
operate -m qwen-vl

# Local LLaVA (requires Ollama running)
operate -m llava

Voice input

operate --voice

Dictate your objective instead of typing it. Requires the audio extras install (see Install).

Set-of-Mark prompting

operate -m gpt-4-with-som

Overlays numbered markers on interactive UI elements before sending the screenshot to the model, improving click accuracy on dense interfaces.

Minimal end-to-end example

# 1. Install
pip install self-operating-computer

# 2. Set your key
export OPENAI_API_KEY=sk-...

# 3. Launch
operate
# > Objective: Take a screenshot of the desktop and save it as test.png

The model will observe the screen, open a terminal or use keyboard shortcuts, run a screenshot command, and confirm completion. Each step is printed to stdout as the model reasons through it.

How it works internally

1. SOC captures a screenshot of the current display.
2. The screenshot (and optionally OCR-extracted text) is sent to the chosen multimodal model with a system prompt describing available actions.
3. The model returns a JSON action (click at coordinates, type text, scroll, etc.).
4. SOC executes the action using platform mouse/keyboard APIs.
5. Steps 1-4 repeat until the model signals the objective is complete.