← Back to Robot Resources Scraper
Install Robot Resources Scraper
Pick your client, copy the command, done.
1
Add it to Claude Code
claude mcp add robot-resources-scraper -- npx -y @robot-resources/scraper-mcpAvailable Tools (2)
Once configured, Robot Resources Scraper gives your AI agent access to:
scraper_compress_urlCompress a single web page into markdown with 70-90% fewer tokens.urlmodetimeoutmaxRetriesscraper_crawl_urlCrawl multiple pages from a starting URL using BFS link discovery.urlmaxPagesmaxDepthmodeincludeexcludetimeoutTry It Out
After setup, try these prompts with your AI agent:
→Compress https://docs.example.com/getting-started into markdown for me.
→Crawl the documentation at https://docs.example.com with a maximum of 20 pages.
→Scrape the article at https://blog.example.com using the stealth mode to avoid bot detection.
→Crawl https://example.com/docs up to a depth of 3 pages.
Prerequisites & system requirements
- An MCP-compatible client (Claude Code, Cursor, Windsurf, Claude Desktop, or Codex)
- Node.js 18+ with npm/npx installed
Alternative installation methods
Global NPM
npm install -g @robot-resources/scraper-mcpKeep this setup from going cold
Save the docs, env vars, and workflow around Robot Resources Scraper in Conare so Claude Code, Codex, and Cursor remember it next time.