Best AI Agent Skills for Web Scraping (2026)

You need a CSV of 400 competitor product pages, name, price, stock status, and you need it before lunch. Writing a bespoke scraper with Playwright, rotating proxies, and a parser that breaks the moment someone ships a CSS change is the old way to do that. An AI agent skill plugged into Claude Code can do the fetch, the parsing, and the schema mapping in one pass, and re-do it next week when the site moves a div.

That's the appeal, and also the trap. The web-scraping corner of the Claude skill and MCP ecosystem is crowded, the licenses range from MIT to AGPL-3.0 copyleft, and a couple of the popular picks need a paid API key before they do anything useful. This guide ranks eight of them by what they actually do, how you install them, and how much you should trust the code before it touches your machine.

The eight at a glance

Two things to read off that table before going further. The license column matters if anything you build ships to paying customers, more on the AGPL question below. And "trust" here means community-reviewed and verified in our directory, not independently security-audited. Read the repo before you install. Every one links to its source.

Three shapes of scraper

These tools don't compete head-to-head, because they solve different problems. Sort them into three buckets and the choice gets easier.

Full-scrape engines — Firecrawl and Scrapling, point at a URL or a domain and pull the whole thing: rendered HTML, markdown, structured fields. Use these when you have specific pages and you want everything on them. Firecrawl will crawl an entire site map; Scrapling focuses on surviving the anti-bot defenses that kill naive scrapers.

Search-and-extract APIs — Exa and Tavily, flip the order. You don't know the URLs yet. You give them a query, they find the relevant pages across the open web and hand back clean extracted content. This is the right shape for market research, competitive monitoring, and anything where "find me sources about X" comes before "scrape this page." Both need an API key, both have free tiers.

Marketplace actors — Apify, sit a level up. Instead of a generic crawler, Apify runs purpose-built "actors" maintained by other developers: an Instagram profile scraper, a Google Maps reviews extractor, an Amazon product actor. You're renting someone else's maintained scraper for a platform that fights scrapers hard. Convenient, but the good actors are paid, and you're trusting a third-party actor's code on top of Apify's.

A data scientist building a one-off dataset wants a full-scrape engine. A researcher mapping a topic wants search-and-extract. Someone pulling social signals at scale wants Apify. Match the tool to the question, not the hype.

The full-scrape engines

Firecrawl

The most-starred of the group by a wide margin (~132k stars) and the one most people mean when they say "the scraping skill." It crawls, scrapes single pages, and extracts structured data with a schema, all built to feed an LLM downstream. If you want one tool that covers fetch-through-to-clean-JSON, start here.

claude skill add firecrawl/firecrawl

Repo: github.com/firecrawl/firecrawl. Verified, community-reviewed, AGPL-3.0 — that license is the catch, covered below.

Firecrawl MCP

Same engine, packaged as an MCP server instead of a skill, under a friendlier MIT license. The trade-off: it talks to Firecrawl's hosted API, so you need a Firecrawl key, and the free tier has limits. Pick this version if you're wiring Firecrawl into an MCP-based agent setup and you're fine with a managed backend doing the heavy lifting.

claude mcp add firecrawl -- npx -y firecrawl-mcp

Repo: github.com/firecrawl/firecrawl-mcp-server. Verified, freemium, MIT, ~7k stars.

Scrapling

The one I reach for when a site is actively hostile. Scrapling is built around adaptive selectors, when a site re-orders its DOM or renames classes, a scraper that hard-codes paths breaks, and Scrapling re-finds the elements anyway. It's also stealthy by design, which matters on sites running bot detection. BSD-3-Clause, so no copyleft headache for commercial use.

claude mcp add scrapling -- npx -y D4Vinci/Scrapling

Repo: github.com/D4Vinci/Scrapling. Verified, community-reviewed, ~63k stars.

llm-scraper

Lighter than Firecrawl and pointed at one job: take a page, hand it a schema, get typed structured data back. The LLM does the parsing, so you skip writing selectors entirely, you describe the shape you want and let the model fill it. Good for messy pages where the structure varies row to row. MIT-licensed.

claude skill add mishushakov/llm-scraper

Repo: github.com/mishushakov/llm-scraper. Verified, community-reviewed, ~7k stars. Pair it with a data-analysis pipeline and the extracted JSON drops straight into your notebook.

The search-and-extract APIs

Exa MCP

Exa runs neural search, semantic, not keyword, across the web, then crawls and extracts the pages it finds. When your starting point is a question rather than a list of URLs, this is the cleaner path. It needs an Exa API key; the free tier is enough to evaluate it.

claude mcp add exa-mcp-server -- npx -y exa-labs/exa-mcp-server

Repo: github.com/exa-labs/exa-mcp-server. Verified, community-reviewed, MIT. Useful enough that it shows up in our kit for researchers.

Tavily MCP

Tavily covers similar ground, real-time search plus extract, map, and crawl, and it was built with AI agents as the first customer rather than an afterthought. In practice Exa and Tavily are close substitutes; try both free tiers on your actual queries and keep the one that returns cleaner sources for your domain. Needs a Tavily key.

claude mcp add tavily-mcp -- npx -y tavily-ai/tavily-mcp

Repo: github.com/tavily-ai/tavily-mcp. Verified, community-reviewed, MIT.

The marketplace and the catalog

Apify MCP

Apify's pitch is that you don't write the scraper, you run an actor someone already maintains. The catalog covers the hard targets: social profiles, maps, marketplaces, sites that ban generic crawlers within minutes. The MCP server lets your agent call those actors directly. The good actors cost money per run, and you're layering trust in a third-party actor on top of Apify itself, so read what an actor does before you run it.

claude mcp add apify-mcp-server -- npx -y apify/apify-mcp-server

Repo: github.com/apify/apify-mcp-server. Verified, community-reviewed, MIT.

scraping-apis-for-devs

Not a scraper itself, a curated catalog of scraping APIs, handy when none of the above fits and you want to see what else exists. Treat it as a reference, not a dependency.

claude skill add cporter202/scraping-apis-for-devs

Repo: github.com/cporter202/scraping-apis-for-devs. Verified, community-reviewed, ~4k stars. One flag: its license is Unknown in our records. No license file means you have no granted rights to reuse the contents in a product, fine to read, risky to copy from. Confirm with the maintainer before you build on it.

The AGPL question, and where the legal lines sit

License first, because it's the one that bites quietly. Firecrawl's skill form is AGPL-3.0. AGPL is strong copyleft with a network clause: if you modify the code and let users reach it over a network, a SaaS, an internal web tool, an API your customers hit, you owe those users the complete corresponding source under AGPL too. For a personal dataset or internal one-off, that's a non-issue. For a commercial product where the scraper runs server-side behind your endpoint, it's a real obligation that can force your own code open. If that's your situation, use the MIT-licensed Firecrawl MCP against the hosted API, or pick BSD-licensed Scrapling instead. Talk to whoever owns licensing risk before shipping, this is a description of how AGPL works, not legal advice for your case.

The other lines are about the scraping itself. This section is analytical, not a green light. Public data is not the same as freely usable data, and "I could fetch it" is not "I'm allowed to." A few principles that keep most projects out of trouble:

Respect robots.txt and the site's Terms of Service, a ToS that forbids automated access is a contract you're choosing to accept by using the site. Rate-limit your requests so you're not degrading someone's server; a scraper that hammers a small site can look like an attack. Don't collect personal data without a lawful basis, GDPR and similar regimes apply to scraped data the same as any other. Cache and identify your bot honestly rather than spoofing a human browser to dodge detection. None of the tools here enforce these for you; they'll scrape whatever you point them at, which is exactly why the responsibility lands on you. When the data is sensitive or the target is large and litigious, get advice specific to your situation before you run anything.

How to choose

Start from the question, not the tool. If you have URLs and want everything on them, a full-scrape engine. Firecrawl for breadth, Scrapling for hostile sites. If you have a question and need to find sources, Exa or Tavily. If your target is a social platform or a site that bans crawlers fast, Apify's actors. If you only need typed JSON out of varied pages, llm-scraper is the smallest thing that works.

Then filter on license and cost. Shipping commercially? Avoid the AGPL skill form, skip the Unknown-license catalog as a dependency, and budget for the API keys the search-and-extract and marketplace tools require. Building for yourself? Any of them is fair game, install, point it at one page, and check the output before you trust it on 400.

For where these fit in a larger workflow, see our guides on building a data-analysis pipeline with Claude skills and how to research a market with Claude skills, plus the full kits for data scientists and researchers.

FAQ

What's the difference between a scraping skill and a scraping MCP server? A skill is packaged code Claude loads and runs in your environment, you own the execution. An MCP server is a running process (often talking to a hosted API) that your agent calls as a tool. Firecrawl ships as both: the skill runs the engine yourself under AGPL, the MCP calls Firecrawl's managed API under MIT. Skills give you more control; MCP servers often need an API key but offload the hard infrastructure.

Which one is best for sites with strong bot detection? Scrapling is built for it, adaptive selectors plus stealth, for direct scraping. For platforms that fight back hardest (Instagram, Google Maps, large marketplaces), Apify's purpose-built actors usually beat a generic crawler, at a per-run cost.

Do I need to pay to use any of these? Firecrawl's skill, Scrapling, and llm-scraper run free with your own compute. Exa, Tavily, and the Firecrawl MCP need API keys with free tiers you'll outgrow at volume. Apify's better actors are paid per run. Start on free tiers and watch where the limits hit your real workload.

Is it legal to scrape a website with these tools? It depends on the site, the data, and your jurisdiction. Respecting robots.txt and the Terms of Service, rate-limiting, and avoiding personal data without a lawful basis keep most projects safe. The tools don't enforce any of that, you do. For sensitive data or large targets, get advice specific to your case.

Can I use the AGPL Firecrawl skill in a commercial product? You can, but AGPL's network clause may require you to release your own source if users reach the modified code over a network. For closed commercial work, the MIT Firecrawl MCP or BSD-licensed Scrapling sidesteps that. Confirm with whoever handles your licensing before shipping.