app-screen-mcp

Control simulators, read accessibility UI trees, capture screenshots,
and run grounded agent actions through one
Model Context Protocol server.

Why this exists

Most mobile automation breaks when scripts act without observing current screen state.

app-screen-mcp fixes that by combining:

• Accessibility structure from idb ui describe-all
• Real pixels from Simulator screenshots
• Deterministic interactions (tap, type_text, swipe, hardware buttons)

This gives agents a closed loop: observe, reason, act, verify.

What you get

• Simulator discovery and boot control
• App launch and termination by bundle ID
• Full normalized accessibility tree
• JPEG screenshots with size/quality controls
• Image-hash suppression to skip unchanged screenshots
• Semantic actions by text or accessibility ID
• Relative-coordinate taps for resolution-independent flows
• One-call screen summary (UI tree + optional screenshot)

Architecture

MCP Client / Agent
        |
        v
   app-screen-mcp
        |
        +--> xcrun simctl (devices, app lifecycle, screenshots)
        |
        +--> idb (UI tree, tap, swipe, type, hardware buttons)
        |
        v
   iOS Simulator

Prerequisites

• macOS with Xcode + iOS Simulator
• Node.js 18+
• idb tooling

Manual install for idb:

brew tap facebook/fb
brew install idb-companion
pip3 install fb-idb

Install

Option 1: one-step installer (recommended)

bash <(curl -fsSL https://raw.githubusercontent.com/xmuweili/app-screen-mcp/main/install.sh)

The script checks/installs:

• Xcode Command Line Tools
• Homebrew
• Node.js (18+)
• idb-companion
• fb-idb
• app-screen-mcp (global npm package)

Option 2: global npm install

npm install -g app-screen-mcp

Option 3: from source

git clone https://github.com/xmuweili/app-screen-mcp.git
cd app-screen-mcp
npm install
npm run build

Configure your MCP client

Claude Desktop

Config file: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Cursor / VS Code MCP

Common config paths:

• .cursor/mcp.json
• .vscode/mcp.json

Use the key your client expects: mcpServers or mcp.servers.

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Restart your MCP client after updating config.

Tool reference (15 tools)

Quick usage patterns

1) Observe without image (cheap)

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": false,
    "compact_tree": true
  }
}

2) Add image when visual confirmation is required

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "max_dim": 720,
    "quality": 55
  }
}

3) Skip unchanged screenshots

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "only_if_changed": true,
    "previous_image_hash": "<last_hash>"
  }
}

4) Stable interaction flow

1. get_screen_summary(include_image=false)
2. find_elements("Sign In")
3. tap_id("login_email") or tap_text("Email")
4. type