unclecode/crawl4ai

Selection

Select when

• Claude Code needs a self-hosted MCP surface for crawling, Markdown, screenshots, PDFs, or JavaScript execution.
• Dynamic pages require browser rendering, scrolling, sessions, cookies, or custom interaction scripts.
• A pipeline needs both heuristic Markdown and schema-based structured extraction.
• Multi-page discovery requires BFS, DFS, best-first, adaptive, or resumable crawling.
• Teams want Python library, CLI, REST, Docker, and MCP access to the same crawler.
• Sensitive workflows favor local deployment over a mandatory hosted crawling account.
• Concurrent crawling needs streaming, resource monitoring, rate limiting, and memory-aware dispatch.
• RAG ingestion needs cleaner and relevance-filtered web-to-Markdown output.

Boundaries

Avoid when

• A native Claude Code plugin or portable Agent Skill is required.
• A small static-page fetch can be handled without browser binaries or a self-hosted service.
• The deployment cannot safely process hostile URLs, HTML, PDFs, JavaScript, or user-supplied configuration.
• The organization cannot maintain current security patches and hardened Docker settings.
• Website terms, robots rules, privacy obligations, or law prohibit the intended collection.
• Authenticated browser profiles or cookies cannot be exposed to automation risk.
• Deterministic extraction accuracy is required without site-specific validation.
• The environment cannot support Python and browser dependencies or Docker resources.
• Anti-bot bypass is expected to work reliably across protected sites.
• External LLM data transfer is unacceptable but LLM-based extraction is mandatory.

Strengths

Capabilities

Risk profile

Risks and limitations

• Uncertainty: only selected first-party files were supplied, so crawler internals, MCP implementation, security controls, tests, CI results, and SBOM contents were not directly inspected.
• Uncertainty: metadata and documentation contain dates through 2026 relative to this analysis context, so release recency, adoption, and current dependency availability cannot be independently reconciled.
• Uncertainty: repository claims such as battle tested, fastest, 5–10x prefetch gains, 3x performance, and anti-bot success were not independently reproduced.
• The project remains version 0.x in the supplied README, so APIs and behavior may still change despite stated stable releases.
• The Docker guide contains inconsistent version text and examples, including a stated latest 0.9.2 while commands use 0.8.6, reducing setup confidence.
• First-party release notes disclose multiple past critical issues including RCE, SSRF, arbitrary file write, authentication bypass, XSS, denial of service, and a dependency supply-chain compromise.
• Browser automation and Docker deployment require substantial dependencies, downloaded browsers, shared memory, and operational resources.
• MCP exposes consequential tools including JavaScript execution, crawling, screenshots, and PDF generation, so client permissions and server isolation require careful control.
• Persistent browser profiles, cookies, headers, proxies, API tokens, crawled content, screenshots, and PDFs may contain sensitive information.
• LLM extraction can transmit crawled content and prompts to configured providers under their separate privacy and retention terms.
• Robots compliance is not evidenced in the supplied core documentation, and anti-bot or authenticated scraping can violate site rules or law.
• Dynamic rendering and heuristic or LLM extraction cannot guarantee completeness, fidelity, factual accuracy, or stable schemas across changing sites.

First-party notes document secure-by-default changes in v0.9.0 and fixes through v0.9.3 for RCE, SSRF, arbitrary file write, auth bypass, XSS, denial of service, hardcoded JWT secrets, unsafe deserialization, and a compromised dependency. Docker hooks were disabled by default in v0.8.0. Deploy only a current patched release, keep authentication enabled, bind privately, restrict egress and MCP clients, constrain resources, protect secrets, and treat URLs, request bodies, page content, PDFs, scripts, hooks, and checkpoints as untrusted. No independent audit was supplied.