Web scraper and token compressor that converts HTML to clean markdown
@robot-resources/scraper-mcp
MCP server for Scraper — context compression for AI agents.
What is Robot Resources?
Human Resources, but for your AI agents.
Robot Resources gives AI agents two superpowers:
- Router — Routes each LLM call to the cheapest capable model. 60-90% cost savings across OpenAI, Anthropic, and Google.
- Scraper — Compresses web pages to clean markdown. 70-80% fewer tokens per page.
Both run locally. Your API keys never leave your machine. Free, unlimited, no tiers.
Install the full suite
npx robot-resources
One command sets up everything. Learn more at robotresources.ai
About this MCP server
This package gives AI agents two tools to compress web content into token-efficient markdown via the Model Context Protocol: single-page compression and multi-page BFS crawling.
Installation
npx @robot-resources/scraper-mcp
Or install globally:
npm install -g @robot-resources/scraper-mcp
Claude Desktop Configuration
Add to your claude_desktop_config.json:
{
"mcpServers": {
"scraper": {
"command": "npx",
"args": ["-y", "@robot-resources/scraper-mcp"]
}
}
}
Tools
`scraper_compress_url`
Compress a single web page into markdown with 70-90% fewer tokens.
Parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
url |
string | yes | — | URL to compress |
mode |
string | no | 'auto' |
'fast', 'stealth', 'render', or 'auto' |
timeout |
number | no | 10000 |
Fetch timeout in milliseconds |
maxRetries |
number | no | 3 |
Max retry attempts (0-10) |
Example prompt: "Compress https://docs.example.com/getting-started"
`scraper_crawl_url`
Crawl multiple pages from a starting URL using BFS link discovery.
Parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
url |
string | yes | — | Starting URL to crawl |
maxPages |
number | no | 10 |
Max pages to crawl (1-100) |
maxDepth |
number | no | 2 |
Max link depth (0-5) |
mode |
string | no | 'auto' |
'fast', 'stealth', 'render', or 'auto' |
include |
string[] | no | — | URL patterns to include (glob) |
exclude |
string[] | no | — | URL patterns to exclude (glob) |
timeout |
number | no | 10000 |
Per-page timeout in milliseconds |
Example prompt: "Crawl the docs at https://docs.example.com with max 20 pages"
Fetch Modes
| Mode | How | Use when |
|---|---|---|
'fast' |
Plain HTTP | Default sites, APIs, docs |
'stealth' |
TLS fingerprint impersonation | Anti-bot protected sites |
'render' |
Headless browser (Playwright) | JS-rendered SPAs |
'auto' |
Fast → stealth fallback on 403/challenge | Unknown sites (default) |
Stealth requires impit and render requires playwright as peer dependencies of @robot-resources/scraper.
Requirements
- Node.js 18+
Related
- @robot-resources/scraper - Core compression library
- @robot-resources/router-mcp - MCP server for LLM cost optimization
- Robot Resources - Human Resources, but for your AI agents
License
MIT
Tools (2)
scraper_compress_urlCompress a single web page into markdown with 70-90% fewer tokens.scraper_crawl_urlCrawl multiple pages from a starting URL using BFS link discovery.Configuration
{"mcpServers": {"scraper": {"command": "npx", "args": ["-y", "@robot-resources/scraper-mcp"]}}}