Self-Operating Computer Framework

Category: OSS Agent (Browser Automation / Computer Use)

Official repo: github.com/OthersideAI/self-operating-computer

License / Pricing: Open source (MIT). Free to run locally. Requires API credits from the model provider you choose. For OpenAI gpt-4o (the default), your account must have spent at least $5 in API credits to access the model.

---

What It Does

The Self-Operating Computer Framework (SOC) lets multimodal AI models operate a computer the same way a human does: by viewing the screen and executing mouse and keyboard actions. You give it a plain-language objective; the model sees your screen, decides what to click or type, and works toward the goal autonomously.

Key capabilities:

• Screen observation - the model receives screenshots of your current display and uses them to reason about next steps.
• Mouse and keyboard control - SOC translates model decisions into real pointer moves, clicks, scrolls, and keystrokes.
• Multimodel support - works with OpenAI gpt-4o, o1, gpt-4.1, Anthropic Claude 3, Google Gemini Pro Vision, Alibaba Qwen-VL, and local LLaVA via Ollama.
• OCR - optical character recognition mode available for models that benefit from extracted text alongside screenshots.
• Voice input - pass --voice to dictate your objective instead of typing it.
• Cross-platform - macOS, Windows, and Linux (X server required on Linux).

Safety and Permissions

SOC has full control over your computer's mouse and keyboard. On macOS you must explicitly grant Terminal (or your shell) permission for Screen Recording and Accessibility in System Settings > Privacy and Security before the framework can function.

Do not run unattended against sensitive applications or with stored credentials visible on screen. The model sees everything on the display.

Template / Boilerplate

No starter template is provided. Installation is a single pip command; see Install.