MCP Library updated 16 min read

Fetch MCP Server: Web Content Retrieval for AI

Enable AI to fetch and read web content. Learn how to retrieve documentation, articles, and API responses using the Fetch MCP Server.

Rajesh Praharaj

· Updated

Contents 12 sections
  1. TL;DR - Fetch MCP Quick Start
  2. When to Use Fetch vs Playwright
  3. Installation & Configuration
  4. Available Tools
  5. Practical Workflows
  6. Working with Different Content Types
  7. Combining with Other MCPs
  8. Troubleshooting
  9. Best Practices
  10. Related MCP Servers
  11. Summary
  12. Frequently Asked Questions

TL;DR - Fetch MCP Quick Start

Give AI assistants the ability to retrieve and parse live web content using the official mcp-server-fetch package (uvx mcp-server-fetch or npx -y @modelcontextprotocol/server-fetch). Whether you are prompting Claude Opus 4.6, GPT-5.4, or Gemini 3.1 Pro (compare tool-calling speeds on our LLM Benchmarks hub), Fetch MCP retrieves static web pages, strips DOM boilerplate, and converts HTML into clean, token-efficient Markdown. For a foundational overview of the protocol, see the MCP Introduction guide.

Key Takeaways

  • Chunked Pagination via max_length & start_index: The fetch tool accepts url, max_length (default 5,000 characters, roughly 1,250 tokens), start_index (default 0 character offset), and raw (boolean), allowing LLMs to paginate through long technical specifications in controlled chunks without overflowing a 128K context window.
  • 10x Faster Than Headless Browsers: Static HTTP retrieval and HTML-to-Markdown conversion completes in 150ms to 600ms using ~35MB RAM, compared to 2,500ms to 6,000ms and 400MB+ RAM for headless Chromium instances in Playwright MCP.
  • Built-in SSRF Protection & robots.txt Compliance: Automatically blocks private/loopback IP ranges and cloud metadata endpoints (169.254.169.254) to prevent Server-Side Request Forgery (SSRF), and obeys robots.txt rules unless explicitly overridden via --ignore-robots-txt (modelcontextprotocol/servers).
  • Cross-Pillar Developer Workflow: Validate your claude_desktop_config.json and API JSON responses in our JSON Formatter & Validator, test HTML/Markdown conversions with our Markdown to HTML Converter, and craft research prompts with our AI Prompt Generator.
Verified Configuration: Running mcp-server-fetch via uvx or npx (Click to Expand)

The official Fetch MCP reference server is maintained in Python (uvx mcp-server-fetch) and also available via Docker (mcp/fetch) or Node wrappers. Here is the recommended uvx configuration with a custom User-Agent:

{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": [
        "mcp-server-fetch",
        "--user-agent=MyEngineeringBot/1.0 (+https://example.com)"
      ]
    }
  }
}

Quick Setup:

{
  "mcpServers": {
    "fetch": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-fetch"]
    }
  }
}

What you can do:

  • Fetch: Retrieve any URL content
  • Convert: HTML to markdown automatically
  • Documentation: Read technical docs
  • APIs: Make HTTP requests
  • Content: Read articles and pages

Example conversation:

You: Fetch the Astro docs homepage and summarize what Astro is

Claude: I'll fetch the documentation...

        **Astro Overview** (from docs.astro.build)
        
        Astro is a web framework for building content-focused 
        websites. Key features:
        - Zero JavaScript by default
        - Component Islands architecture
        - Supports React, Vue, Svelte, etc.
        - Built-in optimizations
        ...

Simple and fast - For static content. Use Playwright for dynamic sites.

Universal: Works with Claude, ChatGPT, GitHub Copilot, Gemini, and all MCP-compatible AI tools.


When to Use Fetch vs Playwright

Choosing between Fetch MCP (mcp-server-fetch), Playwright MCP (@playwright/mcp), and Firecrawl MCP (firecrawl-mcp) comes down to whether the target URL renders its content server-side (SSR / static HTML / JSON REST API) or requires client-side JavaScript execution (SPA hydration, login flows, or interactive DOM clicking). Because Fetch MCP operates as a lightweight HTTP client (httpx / readabilipy) without launching a headless Chromium browser, it executes 8x to 10x faster and uses 90% less RAM for reading documentation sites, changelogs, RSS feeds, and JSON endpoints:

Use Fetch MCP When…Use Playwright MCP When…
Reading static HTML pages & SSR docsSite requires client-side JavaScript rendering (SPAs)
Fetching technical documentation & RFCsNeed to fill forms, log in, or click buttons
Making REST API calls & inspecting JSONNeed visual full-page screenshots or PDF capture
Reading robots.txt, sitemap.xml, or RSS feedsInteractive end-to-end UI testing & browser automation
Ultra-fast (150ms–600ms), low memory (~35MB RAM)Full Chromium browser (2.5s–6s, 400MB+ RAM)

Decision Matrix

Target Content ConditionRecommended MCP ServerWhy It Wins
Single static webpage, Markdown doc, or JSON endpointFetch MCP (mcp-server-fetch)Instant HTTP GET + clean Markdown extraction with chunked start_index pagination
Multi-page documentation site or recursive domain crawlFirecrawl MCP (firecrawl-mcp)Crawls entire URL trees (firecrawl_crawl) and handles anti-bot proxies automatically
Dynamic React/SPA dashboard or authenticated web appPlaywright MCP (@playwright/mcp)Executes live browser JavaScript, maintains session cookies, and interacts with accessibility trees

Installation & Configuration

Prerequisites

  • Python 3.10+ with uv (uvx) or Node.js v18+ (npx)
  • MCP-compatible client (Claude Desktop, ChatGPT, Cursor, Copilot, Gemini, etc.)

Basic Setup

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": ["mcp-server-fetch"]
    }
  }
}

With Custom User-Agent

Some sites require a specific user agent:

{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": [
        "mcp-server-fetch",
        "--user-agent=Mozilla/5.0 (compatible; MyCrawler/1.0)"
      ]
    }
  }
}

Verify Installation

Restart your AI client and test:

You: Fetch example.com and show me what's there

Claude: Fetched example.com:

        # Example Domain
        
        This domain is for use in illustrative examples 
        in documents. You may use this domain in literature 
        without prior coordination or asking for permission.
        
        [More information...](https://www.iana.org/domains/example)

Available Tools

The official mcp-server-fetch package exposes a single, purpose-built fetch tool designed to retrieve web URLs safely while protecting your LLM’s context window from multi-megabyte HTML payloads. By default, fetch parses the HTML DOM, strips navigation bars, scripts, and styling noise, converts the main article body into Markdown, and truncates the response at max_length: 5000 characters. Whenever a document exceeds 5,000 characters, the tool output includes the remaining character count so the AI can call fetch again with start_index: 5000 to read the next section seamlessly:

Primary Tool

ToolDescriptionExample Prompt
fetchRetrieve URL content as clean Markdown or raw text”Fetch the React docs homepage”

Request Parameters (inputSchema)

ParameterType & DefaultDescription & Use Case
urlstring (Required)Target http:// or https:// URL to retrieve
max_lengthinteger (Default: 5000)Maximum number of characters returned in a single call to prevent context bloat
start_indexinteger (Default: 0)Character offset for chunked pagination through long documents (0, 5000, 10000)
rawboolean (Default: false)Set to true to return raw HTML/JSON/XML source without Markdown conversion

Response Format

Fetch MCP automatically:

  • Converts HTML to markdown
  • Preserves JSON as-is
  • Handles plain text
  • Extracts main content when possible

Practical Workflows

Workflow 1: Documentation Research

Read and summarize technical docs. For more on working with AI assistants for research, see the AI Search Engines guide.

You: Fetch the Next.js App Router documentation and explain 
     how routing works

Claude: Fetching Next.js App Router docs...

        **Next.js App Router - Routing**
        
        The App Router uses a file-system based router:
        
        **Key Concepts:**
        
        1. **Folder Structure = Routes**
           - `app/page.tsx` → `/`
           - `app/about/page.tsx` → `/about`
           - `app/blog/[slug]/page.tsx` → `/blog/:slug`
        
        2. **Special Files:**
           - `page.tsx` - Route UI
           - `layout.tsx` - Shared layout
           - `loading.tsx` - Loading UI
           - `error.tsx` - Error boundary
        
        3. **Route Groups:**
           - `(folder)` - Group without affecting URL
           
        4. **Dynamic Routes:**
           - `[param]` - Dynamic segment
           - `[...slug]` - Catch-all
           - `[[...slug]]` - Optional catch-all

Workflow 2: API Documentation

Fetch and understand APIs:

You: Fetch the GitHub REST API docs for creating issues 
     and show me the endpoint details

Claude: Fetching GitHub API documentation...

        **Create an Issue - GitHub REST API**
        
        **Endpoint:**

POST /repos/:owner/:repo/issues


**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| title | string | Yes | Issue title |
| body | string | No | Issue content |
| labels | array | No | Label names |
| assignees | array | No | User logins |
| milestone | integer | No | Milestone number |

**Example Request:**
```json
{
  "title": "Found a bug",
  "body": "Description here",
  "labels": ["bug"],
  "assignees": ["octocat"]
}

Response: 201 Created with issue object


### Workflow 3: Competitive Research

Gather information from websites:

```text
You: Fetch the pricing pages from these tools and compare them:
     - notion.so/pricing
     - airtable.com/pricing
     - coda.io/pricing

Claude: Fetching pricing pages...

        **Pricing Comparison**
        
        | Feature | Notion | Airtable | Coda |
        |---------|--------|----------|------|
        | Free tier | ✅ | ✅ | ✅ |
        | Personal Pro | $10/mo | $20/mo | $10/mo |
        | Team | $18/user/mo | $20/user/mo | $30/user/mo |
        | Storage | Unlimited | 5GB (Pro) | Unlimited |
        
        **Key Differences:**
        - Notion: Best for docs and wikis
        - Airtable: Best for structured data
        - Coda: Best for mixed documents + data

Workflow 4: News and Content Aggregation

Read articles and summarize. For more productivity tips, see the AI for Everyday Productivity guide.

You: Fetch the Hacker News front page and summarize the 
     top 5 stories

Claude: Fetching Hacker News...

        **Top 5 Stories on Hacker News**
        
        1. **"PostgreSQL 17 Released"** (342 points)
           New features: JSON improvements, performance gains
           
        2. **"Why SQLite Is Great for Edge Computing"** (287 pts)
           Discussion on using SQLite in serverless environments
           
        3. **"Building a Compiler in Rust"** (256 points)
           Tutorial series on compiler construction
           
        4. **"The Hidden Costs of Cloud Computing"** (234 pts)
           Analysis of true cloud infrastructure costs
           
        5. **"Show HN: AI Code Review Tool"** (201 points)
           Open-source tool for automated PR reviews

Workflow 5: Local Development

Fetch from local services:

You: Fetch http://localhost:3000/api/health and check 
     if my service is running

Claude: Fetching local API...

        **Health Check Response:**
        
        ```json
        {
          "status": "healthy",
          "version": "2.1.0",
          "uptime": "2 days, 3 hours",
          "database": "connected",
          "cache": "connected"
        }

✅ All systems operational!


---

## Example Prompts

### Documentation

```text
# Official docs
"Fetch the React Hooks documentation"
"Get the Python requests library tutorial"
"Read the Docker Compose file reference"

# API references
"Fetch the Stripe API docs for creating payments"
"Get the OpenAI API rate limits page"
"Read the AWS S3 API reference"

# Tutorials
"Fetch a TypeScript generics tutorial"
"Get the Tailwind CSS installation guide"
"Read the PostgreSQL indexing best practices"

Web Content

# Articles
"Fetch this blog post and summarize it"
"Read the Medium article at this URL"
"Get the content from this tech blog"

# Company pages
"Fetch the about page from company.com"
"Read the careers page for TechCorp"
"Get the product features from startup.io"

# Comparisons
"Fetch pricing from these 3 competitors"
"Read the feature comparison at URL"
"Get release notes from these projects"

APIs and Services

# API endpoints
"Fetch http://localhost:3000/api/users"
"Get the JSON from this public API"
"Read the response from my test endpoint"

# Status pages
"Fetch the GitHub status page"
"Check the AWS service health"
"Read the Cloudflare system status"

# Data feeds
"Fetch the latest posts from this RSS feed"
"Get the JSON data from this endpoint"
"Read the XML sitemap from this site"

Working with Different Content Types

Fetch MCP automatically inspects the HTTP Content-Type response header and adapts its parsing pipeline accordingly. When fetching text/html pages, it runs content-extraction heuristics to strip <script>, <style>, <nav>, and <footer> tags before converting headings, tables, and code blocks into clean Markdown, typically reducing raw HTML token volume by 65% to 80%. When fetching application/json, application/xml, or text/plain endpoints (such as robots.txt, llms.txt, or OpenAPI specs), or when raw: true is passed, it preserves the exact payload structure so your AI assistant can validate schemas or parse configuration syntax without distortion.

HTML (Converted to Markdown)

You: Fetch https://example.com/about

Claude: **About Us**
        
        We are a company that...
        
        [Content converted from HTML to markdown]

JSON (Preserved)

You: Fetch https://api.example.com/data

Claude: ```json
        {
          "users": 1234,
          "status": "active",
          "data": [...]
        }

### Plain Text

```text
You: Fetch a robots.txt file

Claude: ```
        User-agent: *
        Disallow: /private/
        Allow: /public/
        Sitemap: https://example.com/sitemap.xml

---

## Advanced Configuration

In enterprise networks or specialized documentation research workflows, you can customize `mcp-server-fetch` via CLI arguments (`--user-agent`, `--ignore-robots-txt`, `--proxy-url`) and chunked pagination prompts. Configuring an explicit corporate proxy (`--proxy-url`) or custom `User-Agent` string ensures outbound documentation requests pass through enterprise egress firewalls cleanly, while instructing the LLM to iterate over `start_index` offsets (`0`, `5000`, `10000`) lets you ingest 50-page RFCs or API specifications methodically without hitting single-call truncation limits.

### Custom Headers & CLI Flags for Enterprise Access

For APIs requiring authentication:

```text
You: Fetch this API with the bearer token xyz123

Claude: I'll fetch with authorization header...

        [Makes request with Authorization: Bearer xyz123]

Handling Timeouts

For slow sites:

You: Fetch this page with a 30-second timeout

Claude: Fetching with extended timeout...

Multiple URLs

Batch fetching:

You: Fetch these three pages and compare them:
     1. docs.react.dev
     2. vuejs.org
     3. svelte.dev

Claude: Fetching documentation sites...

        **Framework Comparison:**
        
        | Aspect | React | Vue | Svelte |
        |--------|-------|-----|--------|
        | Paradigm | Component | Component | Compiler |
        | Learning Curve | Medium | Lower | Lower |
        | Size | Larger | Medium | Smallest |
        ...

Combining with Other MCPs

Fetch MCP serves as the real-time external knowledge ingestion layer when combined with local and workspace MCP servers. Because LLM training cutoffs leave models unaware of newly released SDK versions, breaking API changelog deprecations, or live competitor pricing, chaining Fetch MCP into Filesystem MCP lets your AI assistant pull the latest upstream documentation and immediately generate a local Markdown reference guide or update your project’s code. Pairing Fetch MCP with Memory MCP stores distilled architectural takeaways in your persistent knowledge graph, while Slack MCP broadcasts release-note summaries to your engineering channels. For more on file organization and multi-step automation, see the AI-Powered Workflows guide.

Fetch + Filesystem

Save fetched content locally:

You: Fetch the API documentation and save it as a markdown file

Claude: [Uses Fetch to get documentation]
        [Uses Filesystem to save file]
        
        Saved API documentation to:
        /docs/external-api-reference.md
        
        - 2,340 words
        - Includes 15 endpoint descriptions
        - Formatted as markdown

Fetch + Memory

Remember useful resources:

You: Fetch this documentation and remember the key concepts

Claude: [Uses Fetch to read docs]
        [Uses Memory to store concepts]
        
        Fetched and memorized:
        - Authentication patterns
        - Rate limiting rules
        - Error code meanings
        
        I'll remember these for future reference!

Fetch + Slack

Share findings with team:

You: Fetch the latest changelog and post a summary to #dev

Claude: [Uses Fetch to read changelog]
        [Uses Slack to post summary]
        
        Posted to #dev:
        
        📢 **Latest Release Notes**
        - Feature: New dashboard widgets
        - Fix: Memory leak in worker process
        - Improvement: 30% faster API responses

Troubleshooting

When mcp-server-fetch fails to retrieve a target URL or returns truncated/empty text, the issue typically stems from one of four root causes: strict robots.txt disallow rules on the target domain, client-side JavaScript SPA rendering (where the initial HTML shell contains only an empty <div id="root"></div>), WAF/Cloudflare bot challenges (HTTP 403), or document truncation at the default 5,000-character boundary. Use the diagnostic tables below to pinpoint the failure mode and switch to start_index pagination, a custom --user-agent, or Playwright/Firecrawl MCP when appropriate.

Issue: “Unable to fetch URL”

Symptoms: Request fails

Solutions:

CauseSolution
URL unreachableCheck URL is correct and site is up
HTTPS requiredUse https:// not http://
Firewall blockingCheck network/proxy settings (--proxy-url)
Site blocking botsPass a custom --user-agent or check robots.txt rules

Issue: “Timeout”

Symptoms: Request takes too long

Solutions:

  • Increase timeout setting
  • Check if site is slow
  • Try alternative URL/mirror
  • Site may be blocking automated requests

Issue: “Content not readable”

Symptoms: Returns garbled or empty content

Solutions:

CauseSolution
JavaScript-renderedUse Playwright MCP instead
Login requiredCan’t fetch without auth
Anti-bot protectionUse Firecrawl MCP or Playwright MCP
Binary contentFetch returns text only

Best Practices

Operating mcp-server-fetch responsibly in production engineering workflows requires balancing network etiquette (robots.txt compliance and rate limiting) with context-window efficiency (max_length chunking and local caching). Rather than repeatedly fetching the same 20-page API documentation URL across every turn of a coding session, instruct your AI assistant to fetch the specification once and save a curated summary to a local .md file via Filesystem MCP or into Memory MCP. This reduces external HTTP latency to 0ms on follow-up questions and prevents unnecessary load on upstream documentation servers. For more on responsible AI tool usage, see the Understanding AI Safety, Ethics, and Limitations guide.

Respectful Fetching

✓ Do✗ Don’t
Check robots.txt and site termsIgnore site scraping policies
Add delays between batch requestsFire dozens of parallel requests
Cache repeated fetches locallyFetch the same static URL repeatedly
Use for legitimate documentation/researchScrape personal or gated user data

Efficient Usage

PracticeWhy
Be specific”Fetch the pricing page” not “fetch the whole site”
Cache locallySave content if you’ll reference it again
Use appropriate toolPlaywright for dynamic content
Check freshnessSites change - refetch if needed

ServerComplements Fetch By…
Playwright MCPJavaScript-heavy sites
Firecrawl MCPDeep crawling
Filesystem MCPSaving fetched content
Memory MCPRemembering key info

Summary

The Fetch MCP Server enables simple web content retrieval:

  • ✓ HTTP requests to any URL
  • ✓ Auto-conversion from HTML to markdown
  • ✓ JSON support for APIs
  • ✓ Fast and lightweight - no browser needed
  • ✓ Great for docs - perfect for technical documentation
  • ✓ Universal support - Claude, ChatGPT, Copilot, Gemini

Key Protocol & Security Features:

  • Streamable HTTP and stdio transport support (JSON-RPC 2.0)
  • Part of Agentic AI Foundation (Linux Foundation)
  • Built-in SSRF protection against private/loopback IP ranges
  • Chunked pagination via max_length and start_index

Best use cases:

  • Reading documentation
  • Fetching API responses
  • Gathering web content
  • Competitive research
  • Checking local services

When to use Playwright instead:

  • JavaScript-rendered content
  • Interactive automation
  • Form submission
  • Screenshot capture

Next: Learn about Firecrawl MCP Server → for deep website crawling.


Questions about Fetch MCP? Check the source code on GitHub or the MCP specification.

Frequently Asked Questions

What's the difference between Fetch MCP and Playwright MCP?

Fetch MCP makes simple HTTP requests and returns content (like curl). Playwright MCP runs a full browser with JavaScript execution and interaction. Use Fetch for static content, Playwright for dynamic or interactive sites.

Can Fetch MCP access sites that require login?

Not directly. Fetch MCP doesn't handle authentication flows. For authenticated content, you'd need to pass cookies or tokens in headers, or use Playwright MCP for login flows.

What format does Fetch MCP return?

Fetch MCP converts HTML to markdown for easier AI processing. It can also return raw HTML, JSON, or plain text, communicating over MCP's JSON-RPC 2.0 transport.

Are there rate limits?

Fetch MCP itself has no limits, but websites may block rapid requests. Be respectful of target sites - add delays between requests and check robots.txt.

Can I fetch from localhost?

Yes! Fetch MCP can access local services and APIs, making it great for development and testing workflows.

Does Fetch MCP execute JavaScript?

No, it's a simple HTTP client. For JavaScript-rendered content, use Playwright MCP which runs a full browser.

Can I set custom headers?

Yes, you can pass custom headers including User-Agent, Authorization, and others for API access or specific requirements.

Is web scraping with Fetch MCP legal?

It depends on the site's terms of service. Always check robots.txt, respect rate limits, and only scrape content you're allowed to access.

Written by Rajesh Praharaj

Product Leader & AI Builder

Product leader and AI builder focused on practical LLM systems, Model Context Protocol (MCP) integrations, agentic workflows, and free browser-based tools for product & engineering teams.

Was this page helpful?

Let us know if you found what you were looking for.