apify/crawlee-python instruction manual

1. What Crawlee for Python is—and how it relates to Claude Code

Crawlee for Python is an Apache-2.0, Python 3.10+ library for web crawling, scraping, browser automation, and persistent result storage. The supplied package file identifies version 1.10.1. It supports ordinary HTTP-based crawlers for speed and Playwright-based crawlers when pages require client-side JavaScript or browser interaction. The documented feature set includes automatic parallel crawling based on available resources, retries, proxy rotation, sessions, URL routing, persistent request queues, datasets, key-value stores, state persistence, and error handling. Crawlers are regular asyncio Python programs, so they can be embedded in a larger application without a special launcher.

**Claude Code classification: `unclassified`.** None of the supplied first-party files documents a Claude Code plugin, skill, hook, MCP server, command integration, or other reliable Claude-specific mechanism. `AGENTS.md` merely contains `.rules.md`; that is insufficient to establish how the repository extends or integrates with Claude Code. Treat Crawlee as a general Python library, not as an evidenced Claude Code extension.

The repository metadata describes support for raw HTTP, BeautifulSoup, Parsel, and Playwright, with headful or headless browser operation and proxy rotation. The package also declares optional integrations for adaptive crawling, Pydantic AI, curl impersonation, HTTPX, OpenTelemetry, SQL backends, Stagehand, and Redis. This manual limits behavioral claims to the supplied documentation and source. The complete upstream API and additional examples are linked from the project website, but were not supplied here.

2. Installation, verification, and project creation

For the full documented feature set, install the `all` extra, install Playwright’s browser dependencies, and verify the import. These commands are copied from the root README:

python -m pip install 'crawlee[all]'
playwright install
python -c 'import crawlee; print(crawlee.__version__)'

The base package keeps dependencies smaller; optional extras add features. Relevant declared extras are `beautifulsoup`, `playwright`, `parsel`, `adaptive-crawler`, `pydantic-ai`, `cli`, `curl-impersonate`, `httpx`, `otel`, `sql_sqlite`, `sql_postgres`, `sql_mysql`, `stagehand`, and `redis`. Install the extra required by your selected crawler; for example, the README explicitly requires the `beautifulsoup` extra for `BeautifulSoupCrawler` and the `playwright` extra for `PlaywrightCrawler`.

The fastest documented project-generation route uses the Crawlee CLI. First check that `uv` is present:

uv --help

Then create a project from one of the prepared templates:

uvx 'crawlee[cli]' create my-crawler

If Crawlee with its CLI is already installed, use:

crawlee create my-crawler

The CLI’s `create` operation generates a project skeleton and lets you select a template. Generated dependencies vary by crawler choice: adaptive templates add adaptive, BeautifulSoup, and Parsel extras; Playwright templates add Playwright; Stagehand adds Stagehand; selected HTTP clients can add HTTPX or curl impersonation. A Playwright Camoufox template adds `camoufox[geoip]~=0.4.5`, and an Apify-enabled template adds `apify`. The supplied files do not document other CLI subcommands, so none should be assumed.

3. HTTP crawling with BeautifulSoupCrawler

Use `BeautifulSoupCrawler` when server-returned HTML contains the data and JavaScript execution is unnecessary. It downloads pages through an HTTP client—`ImpitHttpClient` by default—and parses HTML with BeautifulSoup. This avoids browser overhead and is documented as the higher-performance choice. Install the BeautifulSoup extra before using it.

In the supplied example, `BeautifulSoupCrawler(max_requests_per_crawl=10)` constructs a crawler and caps the run at ten requests. Removing or increasing the argument permits more discovered links to be processed. `@crawler.router.default_handler` registers the asynchronous function used for every request that has no more specific route. The handler receives a `BeautifulSoupCrawlingContext`.

Within that context, `context.request.url` is the current URL, and `context.log.info(...)` records progress. `context.soup` is the parsed document. The expression `context.soup.title.string if context.soup.title else None` returns the title text when a title element exists and otherwise stores `None`. `await context.push_data(data)` appends the dictionary to the default dataset. `await context.enqueue_links()` discovers and enqueues links from the current page. Finally, `await crawler.run(['https://crawlee.dev'])` starts from the supplied URL list. A run creates a `storage/` directory in the current working directory.

The program is asynchronous: `asyncio.run(main())` creates the event loop and runs `main`. Keep network and Crawlee calls awaited as shown. Choose this crawler for efficient HTML extraction; switch to Playwright if required content appears only after client-side JavaScript executes.

4. Browser crawling with PlaywrightCrawler

Use `PlaywrightCrawler` for JavaScript-heavy pages or work that requires a browser. It is built on Playwright and exposes the current page for browser-driven extraction and interaction. The README describes a headless browser by default, while the global configuration can change headless behavior. Install the Playwright extra and the documented Playwright browser dependencies before running it.

The example follows the same control flow as the HTTP crawler. `PlaywrightCrawler(max_requests_per_crawl=10)` creates the crawler and limits the crawl to ten requests. `@crawler.router.default_handler` registers the default asynchronous request handler, whose parameter is a `PlaywrightCrawlingContext`. `context.request.url` identifies the current request, and `context.log.info(...)` emits a progress message.

The important browser-specific object is `context.page`. In the example, `await context.page.title()` asks the browser page for its title. `await context.push_data(data)` stores the URL and title in the default dataset, and `await context.enqueue_links()` finds page links and adds them to the crawl queue. `await crawler.run(['https://crawlee.dev'])` begins the crawl from the initial URL list, while `asyncio.run(main())` executes the asynchronous entry point.

Prefer this crawler only when JavaScript execution or interaction is needed; the documentation recommends `BeautifulSoupCrawler` when browser execution is unnecessary or higher performance matters. Browser launch behavior can be affected by `default_browser_path`, `disable_browser_sandbox`, and `headless` configuration. Disabling a browser sandbox changes a browser security control, so it should not be treated as an ordinary performance switch.

5. Global Configuration reference

`Configuration` is a Pydantic settings class containing common Crawlee defaults. It can read environment variables with Crawlee-prefixed names; the source also accepts selected legacy Apify aliases. Defaults normally require no changes.

• `internal_timeout` defaults to `None` and limits internal asynchronous operations when set.
• `default_browser_path` defaults to `None`; it supplies Playwright’s `executable_path`.
• `disable_browser_sandbox` defaults to `False`; it is passed as Playwright’s `chromium_sandbox` setting.
• `headless` defaults to `True` and is passed to Playwright browser launch.
• `log_level` defaults to `INFO`; input is normalized to uppercase.
• `purge_on_start` defaults to `True` and tells storage clients whether to clear storage at startup.
• `persist_state_interval` defaults to one minute and controls `PersistState` event frequency.
• `system_info_interval` defaults to one second and controls `SystemInfo` events from `LocalEventManager`.
• `max_used_cpu_ratio` defaults to `0.95`; exceeding it marks the system overloaded.
• `max_used_memory_ratio` defaults to `0.9` and provides a memory overload threshold.
• `max_event_loop_delay` defaults to 50 milliseconds and defines the event-loop overload threshold.
• `max_client_errors` defaults to `1`; it is the allowed HTTP 429 client-error count before overload.
• `memory_mbytes` defaults to `None`; when supplied, it gives Snapshotter a maximum memory amount.
• `available_memory_ratio` defaults to `0.25`; when `memory_mbytes` is absent, it helps calculate usable memory and supports dynamic scaling.
• `storage_dir` defaults to `./storage` and sets the storage-client directory.

The ratio fields are validated as greater than zero and no greater than one. Millisecond-named environment aliases feed duration values for the corresponding interval or delay settings.

6. Configuration access, resource control, routing, and persistence

`Configuration.get_global_configuration()` retrieves the process-wide configuration. It exists mainly for backward compatibility; the source recommends `service_locator.get_configuration()` instead. Internally, the class method asks the service locator for the current object, verifies that it is an instance of the requested configuration class, and raises `TypeError` if another configuration type is registered. No source-supported standalone command is needed to use configuration.

Resource settings work with Crawlee’s automatic concurrency controls. `Snapshotter` uses CPU, memory, event-loop delay, HTTP 429, and memory-cap settings to determine whether the system is overloaded. `LocalEventManager` emits periodic system information, while the persist-state interval controls events intended to preserve crawler state during a run. The README says this persistence helps interrupted pipelines avoid restarting from scratch.

Request routing directs URLs to appropriate handlers. The examples demonstrate only `router.default_handler`, the fallback route for each request. Crawlee also documents automatic retries, session management, proxy rotation, a persistent URL queue, and robust error handling, but the supplied files do not provide configuration examples for those features; do not infer method names or commands.

Storage supports tabular data and files. The examples’ `push_data` writes records to the default dataset, while the README separately identifies datasets and key-value stores as organized storage options. The default local path is `./storage`, and a crawler run creates `storage/` in the working directory. Because `purge_on_start` defaults to true, review storage-lifecycle requirements before relying on prior local contents. The project can run anywhere and may also be deployed through the separately documented Apify SDK path.

7. Running generated projects and choosing package managers

A generated project includes usage instructions selected by its package-manager choice. For a Poetry template, install Poetry, install dependencies, and run the generated package using the commands embedded in the template:

pipx install poetry
poetry install
poetry run python -m {{cookiecutter.__package_name}}

For a pip template, the generated instructions are:

python -m pip install .
python -m {{cookiecutter.__package_name}}

For a uv template, the generated instructions are:

pipx install uv
uv sync
uv run python -m {{cookiecutter.__package_name}}

The `{{cookiecutter.__package_name}}` text is a template placeholder: in a generated project it is replaced with the actual importable package name. Likewise, the generated README title and project metadata replace `{{cookiecutter.project_name}}`. Do not paste unresolved placeholders into an unrelated project and expect them to run.

A template configured for manual dependency installation receives a `requirements.txt`; its generated README explicitly leaves dependency installation to the user and supplies no command, so this manual does not invent one. Generated projects require Python `>=3.10,<4.0`. Poetry templates use `poetry-core` as their build backend and disable package mode. Other generated project metadata declares the selected Crawlee extras and optional Apify or Camoufox dependencies.

For help, use the repository’s GitHub issues for bugs, Stack Overflow’s `apify` tag, GitHub Discussions, or the linked Discord server. The repository is open source under Apache License 2.0.

8. Contributor workflow, tests, documentation, and releases

Development requires Python 3.10+ and uses uv plus Poe the Poet. Install all development dependencies with `uv run poe install-dev`. Run the full lint, type-check, and unit-test sequence with `uv run poe check-code`. Individual documented tasks are `lint`, `format`, `type-check`, `unit-tests`, `unit-tests-cov`, `e2e-templates-tests`, `build-docs`, `run-docs`, `build`, and `clean`, each invoked as `uv run poe <task>`.

Ruff performs linting and formatting; ty performs type checks; pytest runs tests. Unit tests marked `run_alone` execute separately, while other unit tests use parallel workers. When a test is flaky, first investigate and repair its root cause. `@run_alone_on_mac` isolates a resource-sensitive macOS test; `@run_alone` isolates a test on every executor; `@pytest.mark.flaky` retries a known unresolved flaky test; and `@pytest.mark.skip` is the last resort and should be tracked in a GitHub issue.

End-to-end template tests require `apify-cli` on `PATH` and `APIFY_TEST_USER_API_TOKEN`, then use `uv run poe e2e-templates-tests`. Documentation uses Google-style docstrings, pydoc-markdown, Markdown content, and Docusaurus. Running docs locally requires Node.js 20+ and `uv run poe run-docs`. Website linting requires Node.js 22.12+, pnpm, and the supplied `pnpm lint`, `pnpm lint:fix`, or `pnpm format` commands from `website/`.

Use Conventional Commits with the documented types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, or `revert`. Beta publication from `master` and stable publication through the release workflow are automated. Manual PyPI publishing is documented only for exceptional cases and requires deliberate version editing, `uv run poe build`, and `uv publish --token YOUR_API_TOKEN`.