Different websites need different approaches to scrape properly. Some are simple and open, others need JavaScript rendering, and some need specialized services. Most people just pick one powerful tool and use it for everything.
But what if the system could start with the simplest, fastest option and automatically get smarter only when it needs to? What if it could try free local tools first, and only use paid services when absolutely necessary?
That's what we built. The progressive escalation is pretty elegant.
The Four Tiers (Each One Smarter Than The Last) The system tries four approaches in order:
It tries each one in order, and stops the second something works. No wasted resources, no overkill.
Tier 1: WebFetch - Start Simple For about 60-70% of websites, you don't need anything fancy. Claude Code has this built-in WebFetch tool that handles basic scraping well.
What it does:
typescript
// WebFetch tool (simplified)WebFetch({ url: "https://example.com", prompt: "Extract all content from this page and convert to markdown"})
1
2
3
4
5
It's not just fetching HTML. It has AI-powered content extraction that understands page structure and converts it to clean markdown. Typically takes 2-5 seconds.
🤖 WebFetch uses AI to understand page structure - it's not just raw HTML scraping.When it fails:
Some sites need proper browser headers to work correctly. That's when we escalate to Tier 2.
Tier 2: cURL with Complete Browser Headers When WebFetch isn't enough, we use cURL with complete Chrome browser headers. Every header that a real browser sends, we send too.
The full command:
bash
curl -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" \ -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8" \ -H "Accept-Language: en-US,en;q=0.9" \ -H "Accept-Encoding: gzip, deflate, br" \ -H "DNT: 1" \ -H "Connection: keep-alive" \ -H "Upgrade-Insecure-Requests: 1" \ -H "Sec-Fetch-Dest: document" \ -H "Sec-Fetch-Mode: navigate" \ -H "Sec-Fetch-Site: none" \ -H "Sec-Fetch-User: ?1" \ -H "Cache-Control: max-age=0" \ --compressed \ "https://target-site.com"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Why this works:
-L: Follow redirects (real browsers do this automatically)-A (User-Agent): Identifies as Chrome 120 on macOSAccept headers: Tells the server what content types we handleSec-Fetch-* headers: Chrome's security headers that indicate request context:Sec-Fetch-Dest: document - We're fetching a webpageSec-Fetch-Mode: navigate - This is a navigation requestSec-Fetch-Site: none - Direct navigationSec-Fetch-User: ?1 - User-initiated request--compressed: Handle gzip/br compression like real browsersThese headers match exactly what Chrome sends, which means sites that need proper browser context work properly.
This gets us another 20-30% of sites that Tier 1 couldn't handle.
Tier 3: Browser Automation - Full JavaScript Execution When even perfect headers aren't enough (because the site needs actual JavaScript execution), we use Playwright.
What Playwright provides:
typescript
import { chromium } from 'playwright';const browser = await chromium.launch();const page = await browser.newPage();await page.goto('https://dynamic-site.com');await page.waitForLoadState('networkidle');const content = await page.content();
1
2
3
4
5
6
7
8
9
This is an actual Chrome browser running - not pretending, actually executing:
Perfect for:
The downside? Takes 10-20 seconds because we're running an actual browser. But when you need it, nothing else works.
This tier catches another 10-15% of sites that the first two couldn't handle.
⏱️ When you see "empty content" from Tier 1 or Tier 2, it's usually a JavaScript-heavy site. Skip straight to Tier 3 on retry to save time.Tier 4: Bright Data MCP - Professional Infrastructure Sometimes you need specialized infrastructure. That's when we use Bright Data.
The Bright Data MCP Tools We Use Our implementation uses four MCP tools that connect to Bright Data's Web Scraper API and SERP API:
1. scrape_as_markdown - Single URL scraping:
typescript
mcp__Brightdata__scrape_as_markdown({ url: "https://complex-site.com"})
1
2
3
Returns the page content in clean markdown using Bright Data's Web Scraper API.
2. scrape_batch - Multiple URLs at once (up to 10):
typescript
mcp__Brightdata__scrape_batch({ urls: [ "https://site1.com", "https://site2.com", "https://site3.com" ]})
1
2
3
4
5
6
7
All scraped in parallel using bulk request handling.
3. search_engine - Scrape Google, Bing, or Yandex results:
typescript
mcp__Brightdata__search_engine({ query: "AI web scraping tools", engine: "google"})
1
2
3
4
Gets structured search results with URLs, titles, descriptions via SERP API.
4. search_engine_batch - Multiple searches at once:
typescript
mcp__Brightdata__search_engine_batch({ queries: [ { query: "AI tools", engine: "google" }, { query: "web scraping", engine: "bing" }, { query: "automation", engine: "yandex" } ]})
1
2
3
4
5
6
7
📚 These MCP tools are just the tip of the iceberg. Bright Data offers many more capabilities including Scraping Browser, Web Unlocker, Custom Scrapers, Serverless Functions, and pre-built datasets. Our four-tier system could easily expand to use these specialized tools for different use cases.What Makes Bright Data Special Infrastructure Features:
Data Collection:
Coverage:
Success rate: 95%+ for publicly available data. Only fails if the site is completely down or content requires authentication.
⚠️ Important: This system is designed for publicly available data only. We're not bypassing authentication controls - if content requires login, you need proper credentials. This is about accessing public information reliably, not circumventing access controls.Installation - Setting Up Bright Data MCP To use Tier 4 (Bright Data MCP), you need to install and configure the Bright Data MCP server. This section walks you through the complete setup process.
Step 1: Get Your Bright Data API Key First, you need a Bright Data account and API token:
Full documentation: Bright Data API Documentation
Step 2: Configure the MCP Server Add the Bright Data MCP server to your Claude Code MCP configuration file (.claude/.mcp.json or ~/.claude/.mcp.json):
json
{ "mcpServers": { "brightdata": { "command": "bunx", "args": [ "-y", "@brightdata/mcp" ], "env": { "API_TOKEN": "your_bright_data_api_token_here" } } }}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Configuration details:
command: "bunx" - Uses bunx to run the MCP server (no installation needed)args: ["-y", "@brightdata/mcp"] - Automatically installs and runs the latest versionenv.API_TOKEN - Your Bright Data API token from Step 1Step 3: Restart Claude Code After adding the MCP server configuration, restart Claude Code to load the Bright Data MCP server.
The server will be automatically downloaded and started on next launch.
Step 4: Verify Installation Once Claude Code restarts, the Bright Data MCP tools will be available:
mcp__Brightdata__scrape_as_markdown - Single URL scrapingmcp__Brightdata__scrape_batch - Multiple URLs (up to 10)mcp__Brightdata__search_engine - Google, Bing, Yandex search resultsmcp__Brightdata__search_engine_batch - Multiple searchesYou can verify by asking Claude Code to list available MCP tools, or just try using the scraping system - Tier 4 will automatically work when needed.
🔧 The MCP server uses bunx, which means it's always running the latest version without manual updates. No installation or maintenance required.The Progressive Escalation Flow The complete flow:
START ↓Try Tier 1 (WebFetch) - Fast and free ↓Did it work? → YES → Return content ✓ ↓No, needs more ↓Try Tier 2 (cURL + Chrome headers) - Still free ↓Did it work? → YES → Return content ✓ ↓No, needs JavaScript ↓Try Tier 3 (Browser Automation) - Still free, just slower ↓Did it work? → YES → Return content ✓ ↓No, needs specialized infrastructure ↓Try Tier 4 (Bright Data) - Costs money but works ↓Did it work? → YES → Return content ✓ ↓(Extremely rare - site is probably down)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Why this works well:
Real-World Performance Timing for each tier:
Worst case: ~40 seconds if we have to try all four Best case: ~3 seconds if Tier 1 works Average: ~10 seconds (usually succeeds by Tier 2 or 3)
Cost breakdown:
So we only pay when we really need it.
Real Examples Example 1: Simple Blog Post
Request: "Scrape https://some-blog.com/article"
What happened:
Most sites work like this.
Example 2: Modern React App
Request: "Scrape https://modern-spa.com"
What happened:
Example 3: Site with Complex Requirements
Request: "Scrape https://complex-site.com"
What happened:
Still succeeded - that's the whole point.
Real-World Use Cases Beyond simple web scraping, this system enables some interesting practical applications:
Use Case 1: Japanese eCommerce Research
You're researching product trends on Japanese Amazon (amazon.co.jp) to understand what's popular in the Japanese market.
The challenge: You need accurate data from a Japanese IP address to see region-specific products and pricing. Using your regular connection shows different results.
How the four-tier system handles it:
Result: Accurate market intelligence showing what Japanese consumers actually see and buy.
Use Case 2: Cybersecurity Defense Investigation
You're on a security team investigating attacker infrastructure - malicious domains, phishing sites, command-and-control servers.
The challenge: You need to analyze these sites without revealing your organization's IP address. Direct connection could alert attackers or burn your investigation.
How the four-tier system handles it:
Result: Anonymous investigation capability that doesn't tip off adversaries you're analyzing their operations.
🔒 For the cybersecurity use case: This is defensive security research on publicly accessible malicious infrastructure. You're not attacking anything, just safely observing what's already public. The anonymity protects your investigation, not malicious intent.Use Case 3: Bypassing Over-Eager Reverse Proxies
You're trying to access a perfectly legitimate public website, but Cloudflare or another reverse proxy is blocking you for no good reason.
The challenge: Aggressive reverse proxy settings flag your datacenter IP, your VPN, or even your regular home connection as "suspicious." You get CAPTCHAs, rate limits, or outright blocks trying to access publicly available content.
How the four-tier system handles it:
Result: Access to public content that over-eager security settings were blocking unnecessarily.
Anthropic's WebFetch Tool Since we use this for Tier 1, here's what Anthropic's WebFetch documentation says about it:
WebFetch - Fetches content from a specified URL and processes it using an AI model. Takes a URL and a prompt as input. Fetches the URL content, converts HTML to markdown. Processes the content with the prompt using a small, fast model. Returns the model's response about the content.
Features:
💡 WebFetch is what makes Claude Code great at web scraping out of the box - no setup required, just works.Smart Optimizations You can skip tiers when you know what you're dealing with:
Skip to Tier 3 if:
Skip to Tier 4 if:
Error handling helps:
The Conclusion Web scraping doesn't have to be an all-or-nothing approach. By building a progressive system like this, you get:
✅ Efficiency - Use simple tools for simple sites ✅ Reliability - Multiple fallbacks when things fail ✅ Cost optimization - Only pay for advanced features when needed ✅ Automatic operation - No manual intervention required ✅ High success rate - Handles everything from simple blogs to complex sites
🔮 The four-tier pattern works for any progressive capability system - start simple, escalate intelligently, only use advanced resources when necessary.The four-tier system works - from straightforward static sites to sophisticated web applications, it automatically finds the right tool for each job.
Building this was pretty satisfying. Watching it automatically escalate through the tiers until it succeeds is genuinely useful.
Pricing Reality Check One important thing about this system - the cost is extremely reasonable for typical usage.
My week-to-week usage: Pennies to maybe a couple dollars per week at most.
Real-world costs over 3 weeks with tons of queries: Total of $0.31, averaging $0.01 per day.
Example weekly costs:
When costs scale up:
Even then, Bright Data's pricing is competitive in the professional scraping market. For detailed pricing and comparison: Bright Data Pricing
The progressive escalation system means you only pay for the advanced infrastructure when you actually need it - which is the whole point.
💵 Typical usage: pennies per week. The free tiers do most of the work.Available as a Public Skill This entire system is available as a public skill in the PAI (Personal AI) repository.
The PAI project exists because advanced AI capabilities shouldn't be locked behind corporate walls or expensive consulting engagements. This is about democratizing AI access - giving everyone the tools to build their own AI systems, not just large companies.
We're building toward Human 3.0: a world where every person has access to AI that amplifies their capabilities, automates their workflows, and helps them accomplish more. That future only works if these tools are freely available and openly shared.
The four-tier web scraping system, along with dozens of other skills and workflows, lives at github.com/danielmiessler/PAI - completely free, fully documented, ready to use.
Drafting...Notes1. Complete implementation available at ~/.claude/skills/brightdata/ with automatic routing and error handling.
2. This system is designed exclusively for scraping publicly available data - not bypassing authentication or accessing restricted content.
3. Questions or improvements? Contact Daniel at daniel@danielmiessler.com or @danielmiessler on X/Twitter.
4. AIL Tier Level 5 (Highest AI Involvement) - Daniel's idea completely implemented by Kai.
5. 🤖 AIL 1: Daniel wrote this post. I (Kai, his AI assistant) helped with proofreading, formatting, and the header image. Learn more about AIL.