Test

Status: PARTIAL (Hetzner server - 2026-07-01)

Self Operating Computer Framework enables multimodal models to control the host machine via mouse, keyboard, and screen capture. Running this against the host OS is unsafe without full isolation.

Hetzner (server)

Verified: 2026-07-01

Verify command:

ssh -o BatchMode=yes -o ConnectTimeout=15 agent-runtime-hetzner-1 \
  'source ~/browser-tools/venv-soc/bin/activate && xvfb-run -a operate --help 2>&1 | head -2'

Real output (exit code 0):

NOTE: You must install tkinter on Linux to use MouseInfo. Run the following: sudo apt-get install python3-tk python3-dev
usage: operate [-h] [-m MODEL] [--voice] [--verbose] [--prompt PROMPT]

Verdict: PARTIAL. The operate CLI is installed, importable, and returns a valid help page under xvfb-run (exit 0). The one caveat is that tkinter is not installed, which prevents the MouseInfo GUI helper from loading. This does not block headless/agentic use of operate itself -- it only affects the interactive mouse-info helper. Install python3-tk python3-dev to clear the warning if MouseInfo is needed.

Environment: ~/browser-tools/venv-soc on agent-runtime-hetzner-1, user merlino.

---

Why this was not tested on the local host

The framework takes direct control of the mouse and keyboard and reads the screen. Executing it against a live host machine risks unintended actions on open windows, files, and applications. The test was not executed locally.

What is required before local testing

• An isolated environment: a macOS or Linux VM with no access to host files or network shares
• macOS Screen Recording permission granted in System Settings > Privacy and Security
• macOS Accessibility permission granted in System Settings > Privacy and Security
• An OpenAI API key (or compatible multimodal model provider)
• Install: pip install self-operating-computer

Smoke test procedure (inside an isolated VM only)

Step 1: Confirm install

pip show self-operating-computer

Expected output:

Name: self-operating-computer
Version: 1.x.x
Summary: A framework to enable multimodal models to operate a computer.
Home-page: https://github.com/OthersideAI/self-operating-computer
Author: OthersideAI

Step 2: Confirm the package is importable

python -c "import operate; print('operate package importable')"

Expected output:

operate package importable

Step 3: Live run (inside VM only)

operate

After entering your API key at the prompt, enter a minimal objective such as Move the mouse to the center of the screen. The framework should print a step-by-step reasoning trace and execute the action.

Signs of a healthy run:

• Steps are prefixed with OPERATE and show coordinates or keystrokes
• The model confirms completion with a message such as Objective complete

Common failure modes

Symptom	Likely cause	Fix
ModuleNotFoundError: No module named 'operate'	Install did not complete	Re-run pip install self-operating-computer
openai.AuthenticationError	API key missing or invalid	Set OPENAI_API_KEY env var
Screenshots return black frames	macOS Screen Recording not granted	Grant in System Settings > Privacy and Security
Mouse actions do nothing	macOS Accessibility not granted	Grant in System Settings > Privacy and Security
NOTE: You must install tkinter... warning	tkinter not installed	sudo apt-get install python3-tk python3-dev (non-blocking for headless use)

Do not use until

A dedicated VM is provisioned, OS-level permissions are granted inside that VM, and no host desktop or sensitive files are accessible from within it.