Firecrawl vs Crawl4AI vs Jina Reader vs Browser Use — Getting Web Data Into Your LLM (2026)

Every retrieval pipeline has an unglamorous first step: getting web pages into a shape a language model can actually use. Before embeddings, before the vector database, before the agent loop, something has to fetch a URL, render the JavaScript, strip the chrome, and hand back clean markdown or JSON. In 2026 four open-source projects dominate that step, and they have almost nothing in common architecturally: Firecrawl is a hosted API you can also self-host, Crawl4AI is an async Python library you run yourself, Jina Reader is literally a URL prefix, and Browser Use is an LLM agent that drives a real browser the way a human would.

This comparison is built from each product's own docs, LICENSE files, pricing pages and blogs, all accessed 21 July 2026 — plus one executed fact: we have already self-hosted Firecrawl v2.11.0 end-to-end in Docker, so its self-host claims are verified hands-on, not quoted. Where a number couldn't be verified from a primary source, we say "not published" rather than guessing. The output of all four usually lands in a vector store; our Qdrant vs Weaviate vs Milvus vs Pinecone comparison covers that memory layer.

Firecrawl — the API that absorbs the hard parts

Firecrawl pitches itself as "The Web Data API for AI." It's a single HTTP API that returns markdown, HTML, or schema-shaped JSON from its Scrape, Crawl, Map and Search endpoints, plus newer ones for interaction and agent control. Its pitch is that the cloud side handles proxies, anti-bot measures, JavaScript rendering and dynamic content so your code is a single POST request. The /search endpoint searches the web and returns full page content for results — only Jina's s.jina.ai comes close.

The license is the sharpest edge here: the core is AGPL-3.0, with the SDKs and some UI components under MIT — the README states this split explicitly. Latest release is v2.11.0 (19 June 2026) on a roughly monthly cadence, and that is exactly the version we ran in our executed self-host guide: docker compose build, docker compose up, API on localhost:3002, and with USE_DB_AUTHENTICATION=false it runs entirely key-free — the build took about ten minutes on our machine. Self-hosting doesn't include everything. Per Firecrawl's docs, the self-hosted version omits their Fire-engine IP/bot-detection layer — the gap we hit hands-on in our executed run — plus the cloud-only /agent and /browser endpoints. AI-dependent features like /extract work self-hosted only if you wire in your own LLM. Company backing, from their own blog: a $14.5M Series A led by Nexus Venture Partners (August 2025). There is an official MCP server, firecrawl-mcp-server, under MIT.

Crawl4AI — the library with nothing withheld

Crawl4AI is the purist option: an Apache-2.0 async Python library (AsyncWebCrawler) driving real browsers via Playwright, aimed squarely at producing "AI-ready" markdown for RAG pipelines. Its docs lead with "No forced API keys, no paywalls", and that holds up: there is no hosted product to buy. A "Crawl4AI Cloud API" is announced as closed beta, with no pricing or date published. It is community-run, maintained by unclecode (Hossein Tohidi), funded by sponsors rather than venture capital, and it ships fast: v0.9.2 landed 15 July 2026, the tenth release in about eight months.

Install is two commands per the docs (we did not execute this one):

pip install crawl4ai
crawl4ai-setup

The crawl4ai-setup step installs browser dependencies for both regular and "undetected" modes and runs OS-level checks. There is also a Docker server (unclecode/crawl4ai, port 11235, at least 4 GB RAM) exposing REST endpoints — /crawl, /md, /screenshot, /pdf, an async job queue — plus a playground and monitoring dashboard. Structured extraction supports CSS, XPath or LLM-based schemas. And the MCP server is built into that Docker server: point Claude Code at http://localhost:11235/mcp/sse and the crawler becomes a tool.

Jina Reader — the URL prefix

Jina Reader has the lowest integration bar here. Prefix a URL with r.jina.ai to get back LLM-friendly markdown. For search, prefix a query with s.jina.ai; it will fetch the top five results and run Reader on each. No SDK, no install. Keyless use is allowed at 20 requests per minute for Reader (Search requires a key). From the docs, the whole client is:

curl "https://r.jina.ai/https://example.com"

It reads pages via headless Chrome or curl-impersonate, handles PDFs and MS Office documents, and captions images with a vision model. The repo is Apache-2.0, with one nuance worth knowing: the ReaderLM-v2 model is CC-BY-NC 4.0 — non-commercial, not open source, per Jina's own page. There are no GitHub releases or tags at all; versioning happens on the SaaS side, and the open-source branch is periodically re-synchronized — most recently April 2026, when it gained a real docker compose self-host path that runs stateless (optionally with MinIO/S3 caching). The SaaS proxy pool and MongoDB storage layer stay hosted-only.

The company context changed in late 2025: Jina AI joined Elastic, per Elastic's own announcement (October 2025), with former Jina CEO Han Xiao now VP of AI at Elastic. The project still seems healthy; the OSS branch was re-synced post-acquisition and Reader is still a listed product. The lack of a public roadmap for Reader, however, is worth noting. Reader is also the only one of the four with no official MCP server documented.

Browser Use — when the data needs an agent

Browser Use works differently from the other three. While they are fetch-shaped tools that take a URL, Browser Use operates on tasks. In its own words: "Browser Use lets an AI agent use a web browser the same way you do — it opens pages, clicks buttons, types, and fills in forms." It converts website interfaces into structured text (DOM-based rather than vision-based, by their own framing) and lets the LLM you supply decide what to do. The library is MIT-licensed, requires Python 3.11+, and installs with:

pip install browser-use

You bring your own LLM key (Anthropic, OpenAI, Google and others are supported via .env) or a Browser Use Cloud key. Latest release is 0.13.6 (17 July 2026) — ten releases in two months, still pre-1.0, so expect API movement between minors. The open-core split is explicit in their README: the library is complete and free, while stealth browsers, captcha handling, proxy rotation and the "much more powerful agent" live in the cloud product. MCP support runs both directions — a local server via browser-use --mcp exposing navigate/click/type/extract tools, plus a hosted cloud MCP server. Backing, from their own blog: a $17M seed led by Felicis Ventures (March 2025).

Side by side

The robots.txt spread — four tools, four answers

This row in the table matters more than any performance benchmark, because the four projects have starkly different policies. Firecrawl respects robots.txt by default — its README says so plainly, alongside a reminder that "It is the sole responsibility of end users to respect websites' policies when scraping." Crawl4AI makes it opt-in: set check_robots_txt=True in CrawlerRunConfig and disallowed URLs fail with a 403, with rules cached locally for seven days — but if robots.txt can't be fetched, crawling proceeds, so the flag is politeness, not a guarantee. Jina Reader and Browser Use publish no robots.txt statement at all that we could find in their docs or READMEs — and for Browser Use that's almost structural, since an agent impersonating a human user has no natural place to consult a crawler policy.

Our position carries over from the Firecrawl self-host guide, where we demoed against our own site: whatever the tool's default, the ethic is yours. Turn the Crawl4AI flag on. Rate-limit yourself even when nothing forces you to. Point agents only at sites you own or have permission to automate. Whatever the tool's default, the ethic is yours.

Pricing: three different meters and one $0

The four tools bill on completely different bases, making direct price comparisons tricky. Firecrawl bills in flat credits — one credit per scraped page, two per ten search results — with published tiers: Free (1,000 credits/mo, crawl capped at 1 request/min), Hobby $16/mo, Standard $83/mo for 100,000 credits, Growth $333/mo for 500,000, all as billed-yearly prices. Jina Reader meters tokens processed, not pages: new users get ten million free tokens, Search costs from 10,000 tokens per request, and the USD price of token packs is not published, so a dollar-for-dollar comparison with Firecrawl cannot honestly be made. Browser Use Cloud is the most meter-shaped of all: plans at $0/$29/$299/$999 plus usage components ($0.02 per browser-hour, $5/GB proxy, per-step or per-token agent billing) — agent-shaped pricing for agent-shaped work, and by its own structure expensive per page for bulk scraping. Crawl4AI costs nothing beyond your infrastructure and, if you use LLM extraction, your model tokens.

At around 100,000 pages a month, Firecrawl's Standard tier is the only flat, published number among the four. At hobby scale, all four have a workable free path: Firecrawl's free credits, Jina's keyless tier, Crawl4AI's $0, and Browser Use's library where the meter is your own LLM key.

Self-host sovereignty

All four are self-hostable, but "self-hostable" spans a spectrum. Crawl4AI gives full parity — there is no withheld tier today. Firecrawl self-hosts well (we've done it) but with named gaps: no Fire-engine anti-bot layer, no /agent or /browser. Reader's OSS branch is real since the April 2026 re-sync but runs stateless, without the SaaS proxy pool or storage layer, and a few licensed assets are fetched separately because they're non-redistributable. Browser Use's library is complete, but stealth, captcha handling and proxies are explicitly cloud features. License terms matter at this layer too: Firecrawl's AGPL-3.0 carries network-copyleft obligations if you modify and serve it, while Apache-2.0 and MIT are permissive. For a fully self-hosted stack — say, feeding the local RAG setup in our AnythingLLM guide — Crawl4AI and Firecrawl have the most complete offline story, with different licenses attached.

How to choose

For the simplest single-URL reads, Jina Reader's URL prefix is the lowest-friction path — keyless at 20 RPM, with search grounding via s.jina.ai when you add a free key. If you want a managed API that handles proxies, JS rendering and anti-bot work for flat credits, use Firecrawl; its AGPL self-host path is also proven if you'd rather run it yourself. For library-level control in Python with zero vendor dependency and zero licence cost, Crawl4AI is the choice — your browsers, your schemas, full self-host parity. Browser Use is for a different class of problem: reach for it only when fetch-shaped tools stop working, like data behind logins, multi-step forms, or UI interactions. For static pages it is overkill by its own pricing structure: an agent spends model tokens per step to do what Reader does in one keyless GET.

FAQ

Can I use all four completely free?

Yes, with different meters. Crawl4AI is free software with no hosted tier at all. Firecrawl's free plan gives 1,000 credits a month (or self-host it key-free, as we did). Jina Reader works keyless at 20 requests/minute and gives new keyholders ten million free tokens. Browser Use's library is free — you pay only your own LLM provider.

Which ones respect robots.txt?

Firecrawl by default, per its README. Crawl4AI only if you set check_robots_txt=True — and it allows the crawl if robots.txt can't be fetched. Jina Reader and Browser Use document no robots.txt behaviour at all, so treat compliance as entirely your job there.

Is Jina Reader still maintained after the Elastic acquisition?

The signals as of July 2026 say yes: the open-source branch was re-synchronized with the SaaS code in April 2026 — after the October 2025 acquisition — gaining a Docker self-host path, and Reader remains a listed Jina product. What doesn't exist is a published Reader-specific roadmap, so factor that into long-term bets.

Which work as MCP servers with Claude?

Three of the four ship official MCP servers: Firecrawl (the MIT-licensed firecrawl-mcp-server), Crawl4AI (built into its Docker server at /mcp/sse), and Browser Use (a local --mcp mode plus a hosted cloud server). Jina Reader has no official MCP server documented — though its URL-prefix design means any HTTP-capable tool can call it anyway.

Do I need Browser Use if I already run Firecrawl?

Only for interaction-shaped jobs. Firecrawl's cloud-only Interact endpoint (2 credits per browser-minute) and Agent endpoint (5 free runs daily, then dynamic pricing) cover some of the same ground, but Browser Use is purpose-built for "log in, fill this form, extract what you see" rather than "fetch this URL".