Index Any Documentation. Search Locally. Stay Private.
MCP Web Docs
Index Any Documentation. Search Locally. Stay Private.
A self-hosted Model Context Protocol (MCP) server that crawls, indexes, and searches documentation from any website. Unlike remote MCP servers limited to GitHub repos or pre-indexed libraries, web-docs gives you full control over what gets indexed — including private documentation behind authentication.
Features • Installation • Quick Start • Tools • Tips • Troubleshooting • Contributing
❌ The Problem
AI assistants struggle with documentation:
- ❌ Remote MCP servers only work with GitHub or pre-indexed libraries
- ❌ Private docs behind authentication can't be accessed
- ❌ Outdated indexes don't reflect your team's latest documentation
- ❌ No control over what gets indexed or when
✅ The Solution
MCP Web Docs crawls and indexes documentation from ANY website locally:
- ✅ Any website - Docusaurus, Storybook, GitBook, custom sites, internal wikis
- ✅ Private docs - Interactive browser login for authenticated sites
- ✅ Always fresh - Re-index anytime with one command
- ✅ Your data, your machine - No API keys, no cloud, full privacy
✨ Features
- 🌐 Universal Crawler - Works with any documentation site, not just GitHub
- 🔍 Hybrid Search - Combines full-text search (FTS) with semantic vector search
- 📂 Collections - Group related docs into named collections for project-based organization
- 🏷️ Tags & Categories - Organize docs with tags and filter searches by project, team, or category
- 📦 Version Support - Index multiple versions of the same package (e.g., React 18 and 19)
- 🔐 Authentication Support - Crawl private/protected docs with interactive browser login (auto-detects your default browser)
- 📊 Smart Extraction - Automatically extracts code blocks, props tables, and structured content
- ⚡ Local Embeddings - Uses FastEmbed for fast, private embedding generation (no API keys)
- 🗄️ Persistent Storage - LanceDB for vectors, SQLite for metadata
- 🔄 Real-time Progress - Track indexing status with progress updates
🚀 Installation
Prerequisites
- Node.js >= 22.19.0
Option 1: Install from NPM (Recommended)
npm install -g @cosmocoder/mcp-web-docs
Option 2: Run with npx
No installation required - just configure your MCP client to use npx (see below).
Option 3: Build from Source
# Clone the repository
git clone https://github.com/cosmocoder/mcp-web-docs.git
cd mcp-web-docs
# Install dependencies (automatically installs Playwright browsers)
npm install
# Build
npm run build
Configure Your MCP Client
Cursor
Add to your Cursor MCP settings (~/.cursor/mcp.json):
Using npx (no install required):
{
"mcpServers": {
"web-docs": {
"command": "npx",
"args": ["-y", "@cosmocoder/mcp-web-docs"]
}
}
}
Using global install:
{
"mcpServers": {
"web-docs": {
"command": "mcp-web-docs"
}
}
}
Using local build:
{
"mcpServers": {
"web-docs": {
"command": "node",
"args": ["/path/to/mcp-web-docs/build/index.js"]
}
}
}
Claude Desktop
Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):
Using npx:
{
"mcpServers": {
"web-docs": {
"command": "npx",
"args": ["-y", "@cosmocoder/mcp-web-docs"]
}
}
}
Using global install:
{
"mcpServers": {
"web-docs": {
"command": "mcp-web-docs"
}
}
}
VS Code
Add to .vscode/mcp.json in your workspace:
Using npx:
{
"servers": {
"web-docs": {
"command": "npx",
"args": ["-y", "@cosmocoder/mcp-web-docs"]
}
}
}
Using global install:
{
"servers": {
"web-docs": {
"command": "mcp-web-docs"
}
}
}
Windsurf
Add to ~/.codeium/windsurf/mcp_config.json:
Using npx:
{
"mcpServers": {
"web-docs": {
"command"
Configuration
{"mcpServers": {"web-docs": {"command": "npx", "args": ["-y", "@cosmocoder/mcp-web-docs"]}}}