Crawl4AI
Open-source Python crawler that turns web pages into clean, LLM-ready Markdown
Overview
Crawl4AI is an Apache-2.0 asynchronous web crawler and scraper built on Playwright that converts pages into clean Markdown and structured data for RAG pipelines, agents, and data workflows. It runs as a Python library or a self-hosted Docker server with a FastAPI endpoint.
Pricing
Pricing shown for reference only. These figures reflect RECATOOLS research as of 21 Jul 2026 and may be out of date or incomplete. This is not financial or purchasing advice — always confirm the current price on the provider’s official website before making any decision.
What you can produce with Crawl4AI
- Convert any URL into clean, noise-filtered Markdown ready to drop into a RAG pipeline or LLM context window.
- Extract structured JSON from pages using CSS selectors, XPath, or an LLM-based extraction strategy driven by your own API keys or a local Ollama model.
- Crawl JavaScript-heavy sites with full browser control — sessions, proxies, custom JS execution, lazy-load and infinite-scroll handling, and screenshots or PDF captures.
- Run adaptive site crawls that stop automatically once enough relevant content has been gathered, instead of exhausting every link.
- Batch-crawl large URL lists concurrently with memory-adaptive dispatching from async Python code.
- Self-host a Dockerised crawling server with REST endpoints, JWT auth, and a monitoring dashboard so non-Python services can request Markdown over HTTP.
ASEAN Perspective
Crawl4AI in Southeast Asia
ASEAN-region availability and pricing notes coming soon. Drop the editorial team a note via /contact/ if you can supply local context (Singapore/Malaysia/Indonesia/Thailand/Vietnam).
Crawl4AI is for turning websites into LLM-ready data without paying a scraping API. It fetches pages through a real Playwright browser, strips the noise, and emits clean Markdown or structured JSON — with CSS, XPath, or LLM-driven extraction strategies, session and proxy management, and an adaptive mode that stops crawling once it has enough relevant content. It runs as an async Python library or as a self-hosted Docker server (with FastAPI endpoints and JWT auth) that non-Python stacks can call over HTTP.
It's a good fit for Python developers who want deep control and zero licence cost, and are willing to manage the operational side — installing browsers, tuning concurrency, and watching for memory leaks on long-running deployments. Teams that just want a simple URL-in, Markdown-out endpoint should look at a managed service like Firecrawl instead.
Availability: free and open source under Apache-2.0, installed via pip (crawl4ai on PyPI, Python 3.10+, latest 0.9.2 from July 2026) or the official unclecode/crawl4ai Docker image. Development is very active, with releases landing every few weeks. A hosted Crawl4AI Cloud API is in closed beta with early-access signups but no published pricing yet.
What people say
Crawl4AI is the most visible library-shaped crawler of the LLM era because it does one job well: turning web pages into clean Markdown, for free, on your own hardware. Developers repeatedly frame it as the self-hosted answer to Firecrawl: no per-page credits, no vendor account, and structured extraction that can run against local models through Ollama instead of a paid API. Data-sovereignty teams in particular like that nothing has to leave their infrastructure.
The praise centres on output quality and depth of control. The Markdown it produces is clean enough to feed straight into RAG pipelines, and the toolkit underneath is unusually deep: CSS, XPath, and LLM-based extraction strategies, session management, proxy support, lazy-load and infinite-scroll handling, plus an adaptive crawling mode that stops when it has gathered enough relevant content rather than exhausting a site. Release cadence is genuinely fast — the 0.9.x series shipped three releases between mid-June and mid-July 2026, and the 0.8.5 changelog alone lists over sixty fixes.
The criticism is just as consistent: because it's a library and not a service, you're on the hook for managing Python, Playwright, proxies, and LLM keys. Reviewers always flag this setup cost when comparing it to managed APIs. The heaviest complaints live in the GitHub issue tracker around long-running Docker deployments — memory that creeps until the container falls over, orphaned Chrome processes chewing CPU, and batch crawls via arun_many misbehaving under load. Maintainers have been chipping at this (browser recycling and context-leak fixes landed in 0.8.5), but users running sustained crawls on small VPSes still report crashes. It's excellent value, producing best-in-class Markdown for a free tool, but production hardening is clearly still a work in progress.
Summary of public user & expert reviews, compiled by RECATOOLS.
About this listing
This entry was compiled from publicly available data including Crawl4AI's official website, press releases, documentation, and reputable third-party publications. RECATOOLS is not affiliated with Crawl4AI unless explicitly stated.
Third-party AI tools update their pricing, features, availability, and policies frequently. Information here may be outdated by the time you read this — we make reasonable efforts to keep listings current, but cannot guarantee absolute accuracy.
For the latest details, please refer to Crawl4AI directly →
Spotted something out of date? Suggest an update →
More in Code & Dev Tools
