unclecode/crawl4ai instruction manual

1. Purpose, scope, and relationship to Claude Code

Crawl4AI is an Apache-2.0 Python crawler and scraper that converts web pages into clean Markdown, structured records, HTML, screenshots, PDFs, links, media, and metadata for RAG systems, agents, and data pipelines. It can run as a Python library, the `crwl` command-line tool, or a self-hosted Docker API. Browser automation is asynchronous and Playwright-based by default; the older synchronous Selenium extra is deprecated.

**Claude Code classification:** `tool_surface` + `external_cli_library` + `infrastructure`; install mode: `runs_alongside`; setup effort: `medium`; confidence: `high`. Crawl4AI does not document a Claude Code plugin, skill, hook, or prompt injection mechanism. Instead, its Docker server exposes MCP endpoints, and Claude Code registers that separately running service as a tool provider. Claude Code can then call `md`, `html`, `screenshot`, `pdf`, `execute_js`, `crawl`, and `ask`. The Python library and CLI are also external tools usable beside an agent.

The supplied documents span the main library, Docker server, MCP, stress tests, C4A-Script tutorial, Chrome extension, example applications, marketplace prototype, SBOM, and release workflows. Some text is visibly stale or inconsistent: the root README announces 0.9.3, while the Docker guide calls 0.9.2 stable but pins examples to 0.8.6. Security defaults also vary by release. Pin versions where reproducibility matters and consult the matching release notes before exposing a server.

2. Install, verify, and perform a first crawl

For the current stable Python package, the documented setup is:

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

Pre-releases are opt-in with `pip install crawl4ai --pre`. A specific documented version example is `pip install crawl4ai==0.4.3b1`. If browser installation fails, the supplied alternatives are `playwright install`, `python -m playwright install chromium`, or:

python -m playwright install --with-deps chromium

The minimal asynchronous Python crawl opens a managed crawler, fetches one URL, and prints Markdown:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://www.nbcnews.com/business")
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

The CLI equivalents include:

crwl https://www.nbcnews.com/business -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
crwl https://www.example.com/products -q "Extract all product prices"

The first emits Markdown, the second performs breadth-first deep crawling capped at ten pages, and the third asks for LLM-assisted extraction. `crwl profiles` creates persistent browser profiles: choose **New profile**, name it, authenticate in the opened Chromium window, and reuse that profile in workflows that require saved cookies. Development installation is documented as cloning the repository and running `pip install -e .`; optional editable extras are `torch`, `transformer`, `cosine`, `sync`, and `all`.

3. Core crawling, Markdown, extraction, and browser controls

`AsyncWebCrawler` owns browser resources; `arun()` crawls one target and `arun_many()` handles batches. `BrowserConfig` controls headless mode, browser type, verbosity, profiles, persistent contexts, user data, headers, cookies, user agents, proxies, viewports, and remote CDP connections. `CrawlerRunConfig` controls caching, scripts, extraction, Markdown generation, streaming, virtual scrolling, link scoring, Shadow DOM flattening, retries, prefetching, and other per-run behavior. Inputs may be HTTP URLs, raw HTML, or local `file://` targets where the selected interface permits them; Docker API releases block `file://` for security.

Markdown output includes raw and filtered forms. `DefaultMarkdownGenerator` generates Markdown; `PruningContentFilter` heuristically removes noise, with `preserve_classes` and `preserve_tags` protecting selected elements. `BM25ContentFilter` retains query-relevant material. Links can become numbered citations and references. Custom Markdown strategies are supported.

Structured extraction has several paths. `JsonCssExtractionStrategy` applies a schema containing a repeating `baseSelector` and CSS-selected fields, including text, HTML, or attributes. XPath is also documented. `LLMExtractionStrategy` sends crawled HTML, Markdown, or fit Markdown to a configured model, optionally with a Pydantic schema and instruction. `LLMConfig` selects a provider and token and supports retry backoff settings. `LLMTableExtraction` chunks, processes, and merges large tables. LLM use can incur provider cost and transmits content under that provider’s terms.

Dynamic pages can receive `js_code`, explicit waits, lazy-load handling, full-page scrolling, iframe extraction, screenshots, and media discovery. Sessions preserve state across steps. Hooks customize browser lifecycle points; Docker hooks have had serious security fixes and are disabled by default in the cited 0.8.0 release. Only enable executable customization after reviewing the exact deployed release and trust boundary.

4. Deep, adaptive, concurrent, and resilient crawling

Deep crawling supports breadth-first, depth-first, and best-first strategies. It can prioritize links, limit pages or depth, cancel gracefully, and recover from checkpoints. `resume_state` restores JSON-serializable state; `on_state_change` receives updated state after each URL so an application can persist it. `prefetch=True` performs discovery while skipping Markdown, extraction, and media processing, returning HTML and links for a later selective pass.

Adaptive crawling attempts to gather enough query-relevant knowledge rather than exhaustively visiting everything. `AdaptiveConfig` sets a confidence threshold, maximum depth, maximum pages, and strategy. `AdaptiveCrawler.digest()` starts from a URL and query. The statistical strategy is documented as fast and suitable for terminology-heavy technical material; the embedding strategy targets conceptual queries and irrelevance detection, using either local `sentence-transformers` or an API embedding provider. Examples also cover comparing strategies, advanced tuning, custom `CrawlStrategy` implementations, and exporting/importing a knowledge base as JSONL.

Other discovery tools include `AsyncUrlSeeder`, which can discover URL candidates from sources such as sitemaps and Common Crawl, and link previews with query-based scoring. `VirtualScrollConfig` handles infinite or virtualized feeds by selecting a container, controlling scroll count and distance, and waiting between scrolls. Shadow DOM content can be flattened with `CrawlerRunConfig(flatten_shadow_dom=True)`.

For batches, `arun_many()` works with dispatchers such as `MemoryAdaptiveDispatcher`. Dispatchers manage concurrent sessions, optional rate limiting, streaming results, and memory pressure. Multi-config batches associate different `CrawlerRunConfig` objects with glob-like or callable URL matchers. Caching avoids redundant work; `CacheMode.BYPASS` forces fresh processing. Anti-bot detection can retry through an ordered proxy configuration and optional fallback fetch function, but bypass claims are not independently established and users remain responsible for site terms and law.

5. Docker server, REST API, jobs, monitoring, and configuration

The shortest supplied Docker launch uses the floating latest image:

docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

Open `http://localhost:11235/playground` to build/test configurations and `http://localhost:11235/dashboard` for monitoring. Stop it with `docker stop crawl4ai && docker rm crawl4ai`. Docker 20.10+, Compose, and at least 4 GB RAM are documented prerequisites. LLM deployments may pass a `.llm.env` through `--env-file`; never commit that file. Compose and local `docker buildx` paths support `INSTALL_TYPE` values `default`, `all`, `torch`, or `transformer`, plus AMD64 GPU builds through `ENABLE_GPU=true`.

The main synchronous endpoint is `POST /crawl`; streaming uses `POST /crawl/stream` and emits NDJSON. Non-primitive JSON configuration must use `{"type":"ClassName","params":{...}}`; dictionaries use `{"type":"dict","value":{...}}`. The Python `Crawl4aiDockerClient` accepts normal `BrowserConfig` and `CrawlerRunConfig` objects and supports streaming iteration plus `get_schema()`.

Specialized endpoints are `POST /html`, `/screenshot`, `/pdf`, and `/execute_js`. They respectively return preprocessed HTML, capture a full-page PNG, render a PDF, or run sequential JavaScript snippets and return the crawl result. JavaScript and output-path features cross important trust boundaries.

Background work uses `POST /crawl/job`, followed by `GET /crawl/job/{task_id}`. Optional webhooks can include custom headers and either metadata or full results; delivery retries five times with exponential delays. `/llm/job` provides the analogous single-URL LLM extraction flow. Health and observability endpoints include `/health`, `/metrics`, `/schema`, and monitor endpoints for health, requests, browsers, and endpoint statistics. Production guidance calls for authentication, specific trusted hosts, rate limits, appropriate timeouts, complete mounted `config.yml` files, and careful secret handling.

6. Connect Crawl4AI to Claude Code through MCP

Start the Docker service first, then register its SSE endpoint with Claude Code using the exact documented commands:

claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse
claude mcp list

The server also exposes a WebSocket endpoint at `ws://localhost:11235/mcp/ws`. Tool schemas are available at `http://localhost:11235/mcp/schema`. From the repository root, the documented connection test is:

python tests/mcp/test_mcp_socket.py

Once registered, Claude Code receives seven tools. `md` turns web content into Markdown. `html` returns preprocessed HTML suitable for extraction. `screenshot` captures a page image. `pdf` renders a page as PDF. `execute_js` runs supplied JavaScript against a page. `crawl` handles crawling, including multiple URLs. `ask` queries Crawl4AI’s library context. This is a direct tool surface, not automatic context injection: Claude Code decides when to invoke a server tool, and the separately running service performs the browser or processing operation.

Treat URLs, page content, JavaScript, output locations, cookies, profiles, and model prompts as untrusted input. The supplied release history reports prior RCE, SSRF, arbitrary-file-write, authentication-bypass, XSS, and denial-of-service vulnerabilities in Docker, hook, PDF, and playground paths. Version 0.9.3 is described as closing five coordinated-disclosure advisories and imposing PDF limits; 0.9.0 is described as secure by default. Because an older Docker guide still shows disabled security settings and old tags, verify the effective configuration of the exact image rather than assuming a pasted default. Avoid granting the service broader network, filesystem, or authenticated-browser access than required.

7. C4A-Script, Chrome extension, and included applications

C4A-Script is Crawl4AI’s visual browser-automation language. Documented commands include `GO` for navigation, `WAIT` for time or selectors, `CLICK`, `TYPE`, `SCROLL`, `IF ... THEN`, `REPEAT`, reusable `PROC` procedures, `SET` variables, and `EVAL` JavaScript. CSS selectors appear in backticks. The tutorial offers a text editor, Blockly visual editor, recording, generated JavaScript, live execution, timeline editing, and practice pages for authentication, infinite scroll, forms, popups, tables, and exports. Run the supplied tutorial with:

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai/docs/examples/c4a_script/tutorial/
pip install -r requirements.txt
python server.py

Then open `http://localhost:8000`. In Python, `CrawlerRunConfig(c4a_script="...")` applies a script during `arun()`.

The unpacked Chrome extension provides Click2Crawl schema construction, element-to-Markdown selection, an alpha action recorder, and Python/JSON code generation. Load its folder through `chrome://extensions/` with Developer mode. Select a repeating container, select and name fields, test, then export. Generated scripts contain an HTML snippet, extraction query, and functions to generate and test a schema. The extension claims client-side operation, while generated LLM code still requires provider credentials.

Included examples are separate applications, not core commands. Prospect-Wizard uses a persistent LinkedIn profile, `c4ai_discover.py` for companies/people, `c4ai_insights.py` for embeddings, organization charts, similarity graphs, and decision-maker CSV output, then `graph_view_template.html` for visualization. The website-to-API example runs `python app.py`, offers schema-based or direct LLM extraction, model CRUD, schema caching, request history, and health endpoints. The marketplace prototype runs a FastAPI backend over SQLite and serves apps, articles, categories, sponsors, search, and statistics; its admin panel is explicitly “coming soon.”

8. Testing, benchmarking, maintenance, releases, and known limits

Memory stress tools generate a local heavy test site, serve it, crawl it with `arun_many()` and `MemoryAdaptiveDispatcher`, display `CrawlerMonitor` output, and save JSON summaries plus CSV memory samples. Standard presets are `quick` (50 URLs/4 sessions), `small` (100/8), `medium` (500/16), `large` (1000/32), and `extreme` (2000/64). Examples include:

python run_benchmark.py quick
python run_benchmark.py custom --urls 300 --max-sessions 24 --chunk-size 50
python test_stress_sdk.py --urls 50 --max-sessions 8 --stream --use-rate-limiter
python benchmark_report.py --limit 5

Options control streaming, monitor mode, rate limiting, port, report generation, cleanup, server lifetime, and reuse of generated or externally served sites. The memory README labels some report behavior and parameters “assumed” and warns that `run_all.sh` may need updating, so do not treat those portions as verified implementation guarantees. Cleanup commands documented there remove `test_site`, `reports`, and `benchmark_reports`.

The SBOM directory contains a best-effort CycloneDX bill of materials; maintainers regenerate it with `./scripts/gen-sbom.sh`. Release automation validates a `vMAJOR.MINOR.PATCH` tag against `crawl4ai/__version__.py`, builds and checks Python artifacts, publishes PyPI/GitHub releases, then triggers cached multi-architecture Docker builds. A `docker-rebuild-v...` tag rebuilds Docker without republishing Python. These are maintainer workflows requiring repository secrets, not normal user installation steps.

Limitations include mutable websites, browser resource cost, proxy/cookie sensitivity, LLM cost and privacy, optional large ML dependencies, and the deprecated sync path. Respect robots directives where applicable, site terms, privacy, copyright, and access controls. The supplied evidence contains ambitious performance and anti-bot claims but no independent validation. Apache-2.0 is the repository metadata license; the README additionally requests attribution and supplies badges and citation formats.