SEO Crawler MCP - Website Crawler & SEO Analyzer for LLMs
Crawl and analyse your website for errors and issues that probably affect your site's SEO
Quick Navigation
Installation | CLI mode | How to use | What gets detected | Data storage | Performance | Tools reference | Available queries
I wanted to build on my experience working with the MCP protocol SDK to see just how far we can extend an AI assistant's capabilities. I decided that I'd quite like to build a crawler to check my site's "technical SEO" health and came across Crawlee - which seemed like the ideal library to base the crawl component of my MCP.
What's interesting is that MCP usually indicates a server connection of some sort. This is not so with SEO Crawler MCP. The MCP protocol is probably more powerful than I realised - this is a self-contained application wrapped in the MCP SDK that handles everything locally:
- Smart request scheduling and queue management
- Automatic retry logic and error handling
- Respectful crawling with configurable delays
- Memory-efficient streaming for large sites
- Better-SQLite3 embedded database storing every crawled page's HTML, metadata, headers, link relationships, and site structure
- Custom SQL analysis engine with 25+ specialised queries detecting content issues, technical SEO problems, security vulnerabilities, and optimisation opportunities
Claude (or your AI assistant of choice) can orchestrate this entire stack through simple function calls. The crawl runs asynchronously, stores everything in SQLite, and then Claude can query that data through natural language - "analyse this crawl for seo opportunities" or "report on internal broken links" - and the MCP server translates that into sophisticated SQL analysis.
You can also run crawls directly from the terminal - perfect for large sites or background processing. The CLI mode lets you run a crawl, get the output directory, and then hand that over to Claude for AI-powered analysis via the MCP tools.
Credits
The core crawling architecture is inspired by the logic and patterns from the LibreCrawl project. We've adapted their proven crawling methodology for use within the MCP protocol whilst adding comprehensive SEO analysis capabilities.
Installation
For Beginners
If you're new to MCP servers, I'd recommend reading these first:
I'd also suggest installing Desktop Commander first - it's useful for working with the crawl output files. See the Desktop Commander setup guide for details.
Quick Install (NPX)
Add this to your Claude Desktop config file:
Windows: C:\Users\[YourName]\AppData\Roaming\Claude\claude_desktop_config.json
Mac: ~/Library/Application Support/Claude/claude_desktop_config.json
{
"mcpServers": {
"seo-crawler-mcp": {
"command": "npx",
"args": ["-y", "@houtini/seo-crawler-mcp"],
"env": {
"OUTPUT_DIR": "C:\\seo-audits"
}
}
}
}
Restart Claude Desktop. Four tools will be available:
seo-crawler-mcp:run_seo_auditseo-crawler-mcp:analyze_seoseo-crawler-mcp:query_seo_dataseo-crawler-mcp:list_seo_queries
Claude Code (CLI)
Claude Code
Tools 4
run_seo_auditInitiates a crawl of a specified website to gather SEO data.analyze_seoPerforms an analysis on the crawled data to identify SEO issues.query_seo_dataExecutes custom SQL queries against the crawled site data.list_seo_queriesLists all available pre-defined SEO analysis queries.Environment Variables
OUTPUT_DIRrequiredThe directory path where crawled site data and SQLite databases are stored.