Computer Use Architecture Deep Dive

A deep dive into the Computer Use implementation: from MCP tool definitions to Python Bridge, from 9-layer security gates to feature flag bypasses.

Patch Environment · Layered Architecture · MCP Tool Layer · Security Gates · Python Bridge · Interaction Loop · Source File Index

Computer Use Architecture Overview

1. Patch Environment Overview

The original Claude Code's Computer Use feature (internal codename Chicago) depends on three components that are not publicly available:

Component	Purpose	Availability
`@ant/computer-use-swift`	Screenshots, display enumeration	Anthropic private npm package
`@ant/computer-use-input`	Mouse/keyboard simulation	Anthropic private npm package
GrowthBook remote config	Feature flags, kill switch	Anthropic internal service

Our approach: preserve the original MCP tool definitions and security mechanisms, only replace the execution layer and feature flag controls.

Patch Environment Comparison

What We Changed

Original Claude Code                     AutoABM (Patched)
────────────────────                     ─────────────────────────
@ant/computer-use-swift  ──replaced──→   Python Bridge (mac_helper.py)
@ant/computer-use-input  ──replaced──→   pyautogui + pyobjc
GrowthBook feature flags ──bypassed──→   gates.ts hardcoded return true
Subscription check (Max/Pro) ──bypassed──→ getChicagoEnabled() = true
Build macro CHICAGO_MCP  ──replaced──→   true
isDefaultDisabledBuiltin ──modified──→   returns false

What We Kept Intact

MCP tool definitions (24 tools with their schema and parameter validation)
9-layer security gates (TCC permissions, app allowlist, permission tiers, pixel validation, etc.)
App classification system (191 bundle IDs categorized with permission mappings)
Session context management (global lock, screenshot cache, state synchronization)
Keyboard shortcut blocklist (system-level dangerous operation interception)

Feature Flag Bypass Details

The original code uses three layers of gating to restrict Computer Use access:

typescript

// Original code (simplified)
function getChicagoEnabled(): boolean {
  // Layer 1: GrowthBook remote config
  const config = getDynamicConfig('tengu_malort_pedway')
  // Layer 2: Subscription check
  const hasSubscription = hasRequiredSubscription() // Max/Pro
  // Layer 3: Build-time macro
  return feature('CHICAGO_MCP') && config.enabled && hasSubscription
}

Our modification:

typescript

// gates.ts — our change
export function getChicagoEnabled(): boolean {
  return true  // ← all three gate layers bypassed
}

Note: Sub-gates (pixelValidation, mouseAnimation, etc.) still retain the original logic and can be controlled via configuration.

2. Layered Architecture

Computer Use uses a 6-layer architecture with clear responsibilities and boundaries:

┌─────────────────────────────────────────────────────────────┐
│  Layer 1 — MCP Tool Interface                               │
│  tools.ts: 24 tool schemas + parameter validation           │
│  buildComputerUseTools() → MCP Tool Definition              │
├─────────────────────────────────────────────────────────────┤
│  Layer 2 — Tool Dispatch & Security Control                 │
│  toolCalls.ts: handleToolCall() + 9 security gates          │
│  deniedApps.ts: 191 app classifications + permission tiers  │
├─────────────────────────────────────────────────────────────┤
│  Layer 3 — MCP Server Binding                               │
│  mcpServer.ts: session context + global lock + screenshot   │
│  bindSessionContext() → per-call overrides                  │
├─────────────────────────────────────────────────────────────┤
│  Layer 4 — CLI Integration                                  │
│  wrapper.tsx: permission dialogs + state read/write         │
│  setup.ts: MCP config initialization                        │
│  gates.ts: feature flags (bypassed)                         │
├─────────────────────────────────────────────────────────────┤
│  Layer 5 — Python Bridge IPC                        [PATCH] │
│  pythonBridge.ts: venv mgmt + JSON RPC + error handling     │
│  callPythonHelper<T>(command, payload) → T                  │
├─────────────────────────────────────────────────────────────┤
│  Layer 6 — Python Runtime Execution                 [PATCH] │
│  mac_helper.py: pyautogui + mss + pyobjc                    │
│  660 lines of Python implementing all system interactions   │
└─────────────────────────────────────────────────────────────┘

Layers marked [PATCH] are our replaced/new code. All other layers are preserved from the original Claude Code.

Why This Layering?

Layer	Source	Reusability
Layer 1-2	`vendor/computer-use-mcp/`	Platform-agnostic, reusable for Electron, Web hosts
Layer 3	`vendor/computer-use-mcp/`	Platform-agnostic, standard MCP protocol
Layer 4	`utils/computerUse/`	CLI-specific, bound to app state
Layer 5-6	`utils/computerUse/` + `runtime/`	macOS-specific, Python implementation

3. MCP Tool Layer

24 Tools Overview

Computer Use exposes 24 tools to the model via MCP (Model Context Protocol):

Category	Tools	Permission Tier
Permission	`request_access`, `list_granted_applications`	None required
Screenshot	`screenshot`, `zoom`	read
Mouse Click	`left_click`, `double_click`, `triple_click`	click
Mouse Advanced	`right_click`, `middle_click`, `left_click_drag`	full
Mouse Move	`mouse_move`, `cursor_position`, `scroll`	click
Mouse Low-level	`left_mouse_down`, `left_mouse_up`	full
Keyboard	`type`, `key`, `hold_key`	full
Application	`open_application`, `switch_display`	full
Clipboard	`read_clipboard`, `write_clipboard`	full
Batch	`computer_batch`	Inherits from sub-operations
Wait	`wait`	None required

Coordinate System

The model interacts with the screen through two coordinate modes:

pixels mode (default):
  Model sees screenshot size (1176 x 784)
  Model outputs coordinate [588, 392]
  scaleCoord() conversion:
    x_logical = (588 * displayWidth / 1176) + originX
    y_logical = (392 * displayHeight / 784) + originY

normalized_0_100 mode:
  Model outputs coordinate [50, 50] (percentage)
  scaleCoord() conversion:
    x_logical = (50 / 100) * displayWidth + originX
    y_logical = (50 / 100) * displayHeight + originY

Screenshot dimensions are calculated by imageResize.ts to ensure:

Long edge <= 1568 pixels
Token budget <= 1568 (vision encoder at 28px/token)
Aspect ratio preserved

App Classification System

deniedApps.ts precisely classifies 191 applications:

Browsers (55 bundle IDs) -> tier read

Safari, Chrome, Firefox, Arc, Edge, Opera, Brave, Vivaldi...
Reason: browser operations should use Chrome MCP, not blind clicking

Terminals (102 bundle IDs) -> tier click

Terminal, iTerm2, VS Code, Cursor, JetBrains IDEs, Xcode...
Reason: terminal operations should use Bash Tool, limited to click only

Trading (34 bundle IDs) -> tier read

Webull, Fidelity, Interactive Brokers, Binance, Kraken...
Reason: financial operations are extremely high-risk, screenshot only

Completely Blocked (policy deny list):

Netflix, Spotify, Apple Music, Kindle...
Reason: copyright compliance, rejected without permission dialog

4. Security Gate System

Every input action (click, keyboard, drag) must pass through 9 security gates before execution:

Security Gates Flow

Gate Details

Gate 1: Kill Switch

typescript

if (adapter.isDisabled()) return errorResult("Computer Use is disabled")

Reads getChicagoEnabled() — always returns true in patched version.

Gate 2: TCC Permission Check

typescript

await adapter.ensureOsPermissions()
// → Python: check_permissions
//   → Accessibility: osascript "tell System Events..."
//   → Screen Recording: CGDisplayCaptureDisplay()

Reports error if macOS Accessibility or Screen Recording permissions are missing.

Gate 3: Global Mutex Lock

typescript

await tryAcquireComputerUseLock(sessionId)
// File lock: ~/.claude/computer-use.lock
// JSON: { sessionId, pid, acquiredAt }

Ensures only one Claude session can control the computer at a time. Supports stale PID recovery.

Gate 4: Hide Non-Allowlisted Apps

typescript

await executor.prepareForAction(allowlistBundleIds)
// Hide all app windows not in the allowlist
// Ensures screenshots only contain authorized apps

Gate 5: Frontmost App Check

typescript

const frontmost = await executor.getFrontmostApp()
if (!allowlist.includes(frontmost.bundleId)) {
  return errorResult("Application not authorized")
}

Even after passing the allowlist, verifies the current foreground app is authorized.

Gate 6: Permission Tier Check

Three-tier permission model:

Tier	Allowed Operations	Prohibited Operations
`read`	Screenshot viewing	Any input action
`click`	Left click, scroll	Right-click, drag, keyboard input
`full`	All operations	No restrictions

typescript

function tierSatisfies(tier: CuAppPermTier, required: ActionKind): boolean {
  const order = { read: 0, click: 1, full: 2 }
  return order[tier] >= order[required]
}

Anti-subversion: If permissions are insufficient, the response includes a TIER_ANTI_SUBVERSION hint to prevent the model from bypassing restrictions via AppleScript or System Events.

Gate 7: Clipboard Guard

Threat model:

1. Agent calls write_clipboard("rm -rf /")
2. Switches to Terminal (click-tier allows clicking)
3. Model clicks Terminal's paste button
4. Malicious command executed

Protection:

When click-tier app becomes frontmost:
  → Save current clipboard content (stash)
  → Clear clipboard
  → Re-clear after each operation

When non-click-tier app becomes frontmost:
  → Restore original clipboard content

Gate 8: Pixel Validation (Staleness Guard)

Last screenshot              Current actual screen
┌────────────┐              ┌────────────┐
│   Button A  │              │   Dialog    │  ← UI has changed
│  [756,342]  │              │  Confirm?   │
└────────────┘              └────────────┘

Validation: sample 9x9 pixel grid at [756,342]
  → Compare last screenshot vs live screenshot pixels
  → Different → reject click + prompt to re-screenshot
  → Same → allow click

Note: In patched version, pixelValidation is off by default (hostAdapter.cropRawPatch() returns null).

Gate 9: System Shortcut Interception

keyBlocklist.ts blocks dangerous shortcuts:

Shortcut	Dangerous Action
`Cmd+Q`	Quit application
`Shift+Cmd+Q`	Log out
`Option+Cmd+Esc`	Force quit dialog
`Cmd+Tab`	App switcher
`Cmd+Space`	Spotlight
`Ctrl+Cmd+Q`	Lock screen

5. Python Bridge Mechanism

Python Bridge Communication

Architecture Design

Original Claude Code uses compiled Swift native modules (.node NAPI plugins) to directly call macOS APIs. We replaced this layer with Python subprocess + JSON RPC:

TypeScript (Bun Runtime)                    Python (venv)
┌────────────────────┐                   ┌────────────────────┐
│  executor.ts       │                   │  mac_helper.py     │
│                    │   execFile()      │                    │
│  callPythonHelper  │ ──────────────→   │  main()            │
│  ('click',         │   command +       │    ├─ parse argv   │
│   {x:756,y:342})   │   --payload JSON  │    ├─ dispatch()   │
│                    │                   │    └─ click()      │
│  ← JSON.parse ──── │ ←──────────────   │       pyautogui    │
│  {ok:true,         │   stdout JSON     │                    │
│   result:true}     │                   │  json_output(...)  │
└────────────────────┘                   └────────────────────┘

Bootstrap Flow

First call to callPythonHelper() automatically completes environment setup:

ensureBootstrapped()
  │
  ├─ Check if .runtime/venv/bin/python3 exists
  │   └─ Missing → python3 -m venv .runtime/venv/
  │
  ├─ Check if pip is available
  │   └─ Missing → python3 -m ensurepip --upgrade
  │
  ├─ Compute SHA256 of runtime/requirements.txt
  │   └─ Compare with .runtime/requirements.sha256
  │       └─ Different → pip install -r requirements.txt
  │                      Write new SHA256 hash
  │       └─ Same → skip installation
  │
  └─ Ready, return venv Python path

Dependencies (runtime/requirements.txt):

Library	Purpose
`mss`	High-performance screen capture
`Pillow`	JPEG encoding and image processing
`pyautogui`	Mouse click, keyboard input
`pyobjc-core`	macOS Objective-C bridge
`pyobjc-framework-Cocoa`	NSWorkspace (app management), NSPasteboard (clipboard)
`pyobjc-framework-Quartz`	CGDisplay (monitors), CGWindow (window list)

Command Mapping

mac_helper.py (660 lines) implements the following commands:

Command	Python Implementation	Return Value
`screenshot`	`mss.grab()` + `PIL.Image` JPEG encoding	`{base64, width, height, displayWidth, displayHeight}`
`zoom`	`mss.grab(region)` region capture	`{base64, width, height}`
`click`	`pyautogui.moveTo()` + `pyautogui.click()`	`true`
`key`	`pyautogui.hotkey()` / `pyautogui.press()`	`true`
`type`	`pyautogui.write(interval=0.008)`	`true`
`drag`	`pyautogui.dragTo(duration=0.2)`	`true`
`scroll`	`pyautogui.scroll()` / `pyautogui.hscroll()`	`true`
`hold_key`	`pyautogui.keyDown()` + `sleep` + `pyautogui.keyUp()`	`true`
`frontmost_app`	`NSWorkspace.frontmostApplication()`	`{bundleId, displayName}`
`list_displays`	`CGGetActiveDisplayList()` + `CGDisplayBounds()`	`[DisplayGeometry...]`
`open_app`	`NSWorkspace.launchApplicationAtURL_options_`	`void`
`read_clipboard`	`NSPasteboard.stringForType_()`	`string`
`write_clipboard`	`NSPasteboard.setString_forType_()`	`void`
`check_permissions`	`osascript` + `CGDisplayCaptureDisplay`	`{accessibility, screenRecording}`

Error Handling

python

# mac_helper.py unified error handling
def main():
    try:
        result = dispatch(command, payload)
        json_output({"ok": True, "result": result})
    except Exception as e:
        error_output({"ok": False, "error": {"message": str(e)}})

TypeScript side:

typescript

// pythonBridge.ts
const parsed = JSON.parse(stdout)
if (!parsed.ok) {
  throw new Error(parsed.error.message)  // → MCP tool error
}
return parsed.result

6. Screenshot-Analyze-Act Loop

A complete Computer Use interaction consists of multiple screenshot-analyze-act cycles:

Typical Interaction Flow

User: "Open NetEase Music and search for a song"

┌─ Cycle 1: Discover and open application ─────────────────┐
│                                                           │
│  Step 1: request_access                                   │
│    → Permission dialog, user authorizes allowed apps      │
│    → Set allowedApps, grantFlags                          │
│                                                           │
│  Step 2: screenshot                                       │
│    → Full screen capture → JPEG encode → base64           │
│    → Cache screenshot dimensions (lastScreenshotDims)     │
│    → Return to model                                      │
│                                                           │
│  Step 3: Model analyzes screenshot                        │
│    → "NetEase Music not on desktop, need to open it"      │
│    → Decides to call open_application                     │
│                                                           │
│  Step 4: open_application("com.netease.163music")         │
│    → Gates 1-9 all pass                                   │
│    → Python: NSWorkspace.launchApplicationAtURL_()        │
│                                                           │
└───────────────────────────────────────────────────────────┘

┌─ Cycle 2: Locate search box ─────────────────────────────┐
│                                                           │
│  Step 5: screenshot                                       │
│    → Full screen (app now open)                           │
│    → Update lastScreenshotDims                            │
│                                                           │
│  Step 6: Model analyzes screenshot                        │
│    → Vision identifies search box at (756, 342)           │
│    → Decides to click search box                          │
│                                                           │
│  Step 7: left_click({coordinate: [756, 342]})             │
│    → Gate 4: Hide non-allowlisted apps                    │
│    → Gate 5: Frontmost is NetEase Music ✓                 │
│    → Gate 6: tier=full >= click ✓                         │
│    → scaleCoord(756, 342) → screen coordinates            │
│    → Python: pyautogui.click(x_logical, y_logical)        │
│                                                           │
└───────────────────────────────────────────────────────────┘

┌─ Cycle 3: Type search query ─────────────────────────────┐
│                                                           │
│  Step 8: type({text: "my favorite song"})                 │
│    → Gate 6: tier=full >= keyboard ✓                      │
│    → Python: pyautogui.write("...", interval=0.008)       │
│                                                           │
│  Step 9: screenshot                                       │
│    → Confirm search results appeared                      │
│                                                           │
│  Step 10: left_click({coordinate: [...]})                 │
│    → Click target song                                    │
│                                                           │
└───────────────────────────────────────────────────────────┘

Coordinate Conversion

What the model sees vs actual screen are different coordinate spaces:

Physical screen (2560 x 1600, Retina 2x)
  ├─ Logical size: 1280 x 800
  └─ Physical pixels: 2560 x 1600

After imageResize:
  ├─ Scaled size: 1176 x 735 (≤1568px budget)
  └─ This is the screenshot size the model "sees"

Model outputs coordinate: [588, 368] (in image space)

scaleCoord conversion:
  x_logical = (588 / 1176) * 1280 + originX = 640
  y_logical = (368 / 735) * 800 + originY = 400

Python executes:
  pyautogui.moveTo(640, 400)  ← logical coords (macOS handles Retina)

Screenshot Cache & State Sync

bindSessionContext closure
  │
  ├─ lastScreenshot (in-memory)
  │   ├─ base64: JPEG data (for pixel validation)
  │   ├─ width/height: model-visible dimensions
  │   └─ displayWidth/displayHeight/originX/originY: display geometry
  │
  └─ AppState.computerUseMcpState (persisted)
      ├─ allowedApps: AppGrant[]        — authorized app list
      ├─ grantFlags: {...}              — clipboard/system shortcut permissions
      ├─ selectedDisplayId?: number     — selected display
      ├─ lastScreenshotDims?: {...}     — screenshot geometry (survives restart)
      └─ hiddenDuringTurn?: Set<string> — apps hidden this turn

7. Key Source File Index

vendor/computer-use-mcp/ (Original Code Layer)

File	Lines	Responsibility
`types.ts`	622	Permission model, session context, all type definitions
`tools.ts`	707	24 MCP tool schemas and parameter validation
`toolCalls.ts`	1600+	Core: tool dispatch, 9 security gates, permission flow
`deniedApps.ts`	554	191 app classifications (browser/terminal/trading) and permission mappings
`sentinelApps.ts`	44	Sensitive app warning labels (shell/filesystem/system_settings)
`mcpServer.ts`	314	MCP server factory, session context binding, global lock
`pixelCompare.ts`	172	Click target pixel validation (staleness guard)
`imageResize.ts`	109	Screenshot dimension calculation (API image transcoder)
`keyBlocklist.ts`	154	System shortcut interception (Cmd+Q, Cmd+Tab, etc.)
`executor.ts`	101	ComputerExecutor interface definition
`subGates.ts`	20	Feature flag sub-gate presets

utils/computerUse/ (CLI Adaptation Layer)

File	Lines	Responsibility	Patched?
`executor.ts`	231	ComputerExecutor Python bridge implementation	Yes, rewritten
`pythonBridge.ts`	111	Python subprocess management, venv bootstrap, JSON RPC	Yes, new
`hostAdapter.ts`	54	HostAdapter implementation (permission checks, flag reading)	Partially
`gates.ts`	51	GrowthBook feature flags (`getChicagoEnabled` bypass)	Yes, modified
`wrapper.tsx`	300+	Session context construction, permission dialogs, lock management	Unchanged
`setup.ts`	54	MCP config initialization	Unchanged
`computerUseLock.ts`	216	Global file lock (`~/.claude/computer-use.lock`)	Unchanged
`common.ts`	62	Constants (server name, bundle ID)	Unchanged
`cleanup.ts`	—	Turn-end cleanup (app restore, clipboard restore)	Unchanged
`toolRendering.tsx`	—	Tool result UI rendering	Unchanged

runtime/ (Python Runtime)

File	Lines	Responsibility	Patched?
`mac_helper.py`	660	All system interactions in Python	Yes, new
`requirements.txt`	6	Python dependency declarations	Yes, new

8. Design Trade-offs

Why Python Bridge?

Dimension	Native Swift (.node)	Python Bridge
Performance	~0ms (in-process call)	~50-100ms (subprocess startup)
Readability	Not readable after compilation	660 lines of clear Python
Modifiability	Requires Swift build environment	Edit .py files directly
Dependencies	Specific Bun version NAPI	Any Python 3.8+
Cross-platform	macOS only	pyautogui/mss are natively cross-platform
User experience	Imperceptible	Imperceptible (model thinking takes 2-5s)

Conclusion: The 50-100ms extra latency is completely negligible in Computer Use scenarios — the model typically takes 2-5 seconds for screenshot analysis and decision-making, so users won't notice the additional 100ms for underlying operations.

Approaches We Tried But Abandoned

Approach 1: Extract native .node modules

Successfully extracted computer-use-swift.node (ARM64 424KB) from the Claude Code binary
Synchronous methods worked, but Swift async method continuations never resumed
Root cause: .node files compiled for Claude Code's built-in Bun, incompatible with user's Bun version

Approach 2: Empty stub packages

Code compiled but all operations threw errors — no actual execution capability

Computer Use Guide — Usage, quick start, environment variables
Source Fixes — Detailed records of other fixes and patches

Computer Use Architecture Deep Dive ​

1. Patch Environment Overview ​

What We Changed ​

What We Kept Intact ​

Feature Flag Bypass Details ​

2. Layered Architecture ​

Why This Layering? ​

3. MCP Tool Layer ​

24 Tools Overview ​

Coordinate System ​

App Classification System ​

4. Security Gate System ​

Gate Details ​

Gate 1: Kill Switch ​

Gate 2: TCC Permission Check ​

Gate 3: Global Mutex Lock ​

Gate 4: Hide Non-Allowlisted Apps ​

Gate 5: Frontmost App Check ​

Gate 6: Permission Tier Check ​

Gate 7: Clipboard Guard ​

Gate 8: Pixel Validation (Staleness Guard) ​

Gate 9: System Shortcut Interception ​

5. Python Bridge Mechanism ​

Architecture Design ​

Bootstrap Flow ​

Command Mapping ​

Error Handling ​

6. Screenshot-Analyze-Act Loop ​

Typical Interaction Flow ​

Coordinate Conversion ​

Screenshot Cache & State Sync ​

7. Key Source File Index ​

vendor/computer-use-mcp/ (Original Code Layer) ​

utils/computerUse/ (CLI Adaptation Layer) ​

runtime/ (Python Runtime) ​

8. Design Trade-offs ​

Why Python Bridge? ​

Approaches We Tried But Abandoned ​

Related Documentation ​

Computer Use Architecture Deep Dive

1. Patch Environment Overview

What We Changed

What We Kept Intact

Feature Flag Bypass Details

2. Layered Architecture

Why This Layering?

3. MCP Tool Layer

24 Tools Overview

Coordinate System

App Classification System

4. Security Gate System

Gate Details

Gate 1: Kill Switch

Gate 2: TCC Permission Check

Gate 3: Global Mutex Lock

Gate 4: Hide Non-Allowlisted Apps

Gate 5: Frontmost App Check

Gate 6: Permission Tier Check

Gate 7: Clipboard Guard

Gate 8: Pixel Validation (Staleness Guard)

Gate 9: System Shortcut Interception

5. Python Bridge Mechanism

Architecture Design

Bootstrap Flow

Command Mapping

Error Handling

6. Screenshot-Analyze-Act Loop

Typical Interaction Flow

Coordinate Conversion

Screenshot Cache & State Sync

7. Key Source File Index

vendor/computer-use-mcp/ (Original Code Layer)

utils/computerUse/ (CLI Adaptation Layer)

runtime/ (Python Runtime)

8. Design Trade-offs

Why Python Bridge?

Approaches We Tried But Abandoned

Related Documentation