Install Screen Agent

Pick your client, copy the command, done.

claude mcp add screen-agent -- python "C:\\path\\to\\screen_agent\\mcp_server.py"

Environment Variables

Set these before running Screen Agent.

VariableDescriptionRequired

API_KEYAPI key for cloud-based visual recognition services (if used).No

Once configured, Screen Agent gives your AI agent access to:

screen_get_layoutBind to a window and retrieve its layout information.

screen_clickClick on a specific screen element using OCR, UIA, or color matching.

targetmodecontrol_typefeatures

screen_input_textInput text into the active or specified element.

screen_scrollPerform a scroll action on the screen.

screen_hotkeyExecute a keyboard shortcut.

screen_captureCapture a screenshot and identify UI elements.

screen_waitPause execution for a specified duration.

screen_exploreAutomatically explore the current interface.

screen_detect_uiDetect the position of UI elements.

screen_scan_ui_elementsScan the screen and generate icon features.

screen_ask_user_locateRequest assistance from the user to locate an element.

screen_learn_successRecord a successful operation to the learning system.

screen_query_knowledgeQuery the learned knowledge base for previous operations.

After setup, try these prompts with your AI agent:

→Find the 'Settings' button on the current window and click it using OCR.

→Scan the current UI and identify all available buttons.

→Input the text 'Hello World' into the active text field.

→Query the knowledge base to see if we have learned how to handle this specific application window.

→Perform a scroll action to reach the bottom of the current page.

Prerequisites & system requirements

An MCP-compatible client (Claude Code, Cursor, Windsurf, Claude Desktop, or Codex)
Python 3.8+ with pip installed

Conare · memory for coding agents

Save the docs, env vars, and workflow around Screen Agent in Conare so Claude Code, Codex, and Cursor remember it next time.