MCP AutoGUI Multinode MCP Server

English|中文

About

An MCP and HTTP server wrapper for PyAutoGUI, enabling LLMs to control your mouse and keyboard.

Architecture

The service supports two deployment architectures:

LLM -> MCP -> TOOL (Remote Tool Service)

This architecture separates the MCP server from the tool service, allowing the MCP server to connect to a remote tool service via HTTP.

graph LR
    LLM[LLM Client] -->|MCP Protocol| MCP[MCP Servermain.pyClient-based Tools]
    MCP -->|HTTP APIwith API Key| TOOLA[Tool Servicetool.pyHTTP API Server]
    MCP -->|HTTP APIwith API Key| TOOLB[Tool Servicetool.pyHTTP API Server]

    TOOLA -->|PyAutoGUI| COMPUTERA[Computer Control]
    TOOLB -->|PyAutoGUI| COMPUTERB[Computer Control]
    style LLM fill:#e1f5ff
    style MCP fill:#fff4e1
    style TOOLA fill:#ffe1f5
    style COMPUTERA fill:#e1ffe1
    style COMPUTERB fill:#e1ffe1

Characteristics:

• MCP server uses client-based tools (register_computer_tools_with_client)
• MCP server forwards requests to remote tool service via HTTP
• Tool service performs actual computer control operations
• Suitable for distributed deployments where MCP server and tool service run on different machines
• Requires endpoint parameter in MCP tool calls

Architecture 2: LLM -> MCP (Direct Tools)

This architecture uses direct tools where the MCP server directly performs computer control operations.

graph LR
    LLM[LLM Client] -->|MCP Protocolstdio/http| MCP[MCP Servermcp_local.pyDirect Tools]
    MCP -->|PyAutoGUI| COMPUTER[Computer Control]
    
    style LLM fill:#e1f5ff
    style MCP fill:#fff4e1
    style COMPUTER fill:#e1ffe1

Characteristics:

• MCP server uses direct tools (register_computer_tools)
• MCP server directly executes computer control operations
• No separate tool service required
• Suitable for local deployments where everything runs on the same machine
• No endpoint parameter needed in MCP tool calls

Features

• 🚀 Dual Protocol Support: HTTP REST API and MCP (Model Context Protocol)
• 🔐 API Key Authentication: Optional API key authentication for service-to-service communication
• 🌐 Multiple MCP Transports: Support for both HTTP and stdio (Standard Input/Output) transport modes
• 🖱️ Mouse Control: Move, click, drag, scroll operations
• ⌨️ Keyboard Control: Press keys, type text, key combinations
• 📸 Screenshot: Capture screen and get base64-encoded images
• 📊 Screen Info: Get cursor position and screen resolution
• ⚙️ Configuration Management: Pydantic Settings with environment variable support
• 📝 Auto Documentation: Swagger UI for HTTP API
• 🔧 Flexible Deployment: Run HTTP server or MCP server independently
• 📋 Request Tracing: Request ID middleware for request tracking
• 📝 Structured Logging: Loguru-based logging with request ID integration
• 🔌 Remote MCP Support: Optional HTTP client for remote tool server integration

Quick Start

Prerequisites

• Python >= 3.12
• uv package manager (recommended)

Installation

1. Clone the repository:

git clone https://github.com/stonehill-2345/mcp-autogui-multinode.git
cd mcp-autogui-multinode

2. Install dependencies based on your deployment scenario:

Local Full Development

For local development with all features (GUI control + testing):

uv sync --group gui --group dev

Deploy MCP Server Only

For deploying MCP server that connects to remote tool service (no GUI dependencies needed):

uv sync --no-group gui

Deploy Tool Service Only

For deploying HTTP tool service that performs actual computer control (requires GUI):

uv sync --group gui

Running the Service

The service supports two independent servers:

1. Run Tool Service (HTTP API)

Starts the HTTP API server for computer control:

uv run python tool.py

2. Run MCP Server

Starts the MCP server that can connect to remote tool services. The server supports two transport modes:

HTTP Transport Mode:

uv run python mcp_local.py http

stdio Transport Mode (default):

uv run python mcp_local.py stdio

After starting, you can access:

• HTTP API Documentation: http://localhost:8000/docs
• Health Check: http://localhost:8000/health
• MCP Endpoint: http://localhost:8001/mcp (if using HTTP transport)

API Endpoints

Base Endpoints

• GET / - Root path, returns API information
• GET /health - Health check endpoint

Computer Control Endpoints

All computer control actions are available at:

• POST /api/computer/{action} - Execute a computer control action