Discover great AI tools
📢 NoticeWebsite launched:DeepToSeek AI tools directory is live. Submit great tools and get featured!
Home / Open Source Radar / RAG & Knowledge / crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

Stars87.5k
Forks9.1k
LanguagePython
LicenseApache-2.0

About this project

crawl4ai is an open-source retrieval and knowledge project on GitHub: Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it. It lives in retrieval and knowledge management, the layer that lets a model answer from your own documents instead of from training data alone. It is maintained by unclecode and published as a public repository you can read, fork and run yourself.

On GitHub the repository is tagged ai, ai-agents, crawl4ai-cloud, crawler, data-extraction, llm, markdown, mcp, open-source, playwright, python and rag, which is a fair summary of what it covers. There is normally an ingestion pipeline, an embedding and vector index, and a query layer on top. Most of the engineering effort goes into chunking, ranking and keeping the index fresh as documents change.

It is written mainly in Python and released under APACHE-2.0, a permissive license — commercial use is fine as long as copyright notices are kept. It is actively maintained: the most recent commit landed on November 17, 2026.

To try it, clone the repository and follow its README; that file is the canonical install guide. Because it runs on your own infrastructure, budget time for environment setup: dependencies, model weights and configuration usually take longer than the first successful run.

Key facts as recorded on GitHub: 87,470 stars; 9,125 forks; primary language Python; license APACHE-2.0; last commit on November 17, 2026; topics ai, ai-agents, crawl4ai-cloud, crawler, data-extraction, llm.