apify/crawlee-python installation guide

Install Crawlee for Python on macOS Tahoe (Apple silicon)

What this repository is

`apify/crawlee-python` is the public, Apache-2.0-licensed source repository for **Crawlee for Python**, a web-scraping and browser-automation library. It supports ordinary HTTP/HTML crawling as well as browser-based crawling with Playwright. The supplied project metadata identifies Python as its main language, and the package configuration marks it as operating-system independent.

Relationship to Claude Code: unclassified

The supplied first-party files do not document a Claude Code plugin, MCP server, hook, skill installation, or any other dependable Claude-specific integration. `AGENTS.md` only points to `.rules.md`; that is not enough evidence to identify how the project extends Claude Code. Therefore, its Claude Code relationship is **unclassified**. Install Crawlee as a normal Python library or use its own CLI; do not expect this procedure to add anything to Claude Code.

Before you begin

You need:

• A Mac with Apple silicon running macOS Tahoe.
• Access to the **Terminal** application.
• Python **3.10 or later**. The package configuration lists support for Python 3.10 through 3.14 and requires Python 3.10 or newer.
• Internet access so Python can download Crawlee and its dependencies. Browser-based use also requires Playwright to download browser components.

The repository describes Crawlee as operating-system independent but supplies no separate macOS Tahoe or Apple-silicon installation command. It also does not provide a source-supported command for installing Python on macOS. If your Mac does not already have Python 3.10 or later, install a compatible Python distribution first using its first-party instructions, then return here.

Open Terminal before continuing. You can find it with Spotlight or under **Applications → Utilities → Terminal**.

Choose an installation route

There are two documented routes:

1. **Install the complete Crawlee package with all optional features.** This is the clearest route if you want the library and browser support available immediately.
2. **Use the Crawlee CLI to generate a starter project.** This route requires `uv` and lets you choose a prepared template interactively.

For a first installation, follow Route 1. Route 2 is optional and is covered later.

Route 1: install Crawlee with all features

Step 1: install the package

In Terminal, paste this exact command and press Return:

python -m pip install 'crawlee[all]'

The quotation marks are important because the package name contains square brackets. The `all` extra installs the core package plus the optional feature groups declared by the project, including BeautifulSoup, Parsel, Playwright, the Crawlee CLI, telemetry support, database integrations, Redis support, and other listed integrations. This is consequently a larger installation than the core-only package.

Wait for the command to finish. Terminal will print package download and installation progress. If it reports that your Python version is below 3.10, stop and install a supported Python version rather than trying to bypass the requirement.

This command installs the published `crawlee` package from PyPI. It does not clone the GitHub repository, and cloning is not required for ordinary use.

Step 2: install Playwright browser components

The repository’s installation sequence next requires:

playwright install

This installs the dependencies used by Playwright. Playwright is the browser-automation layer used by `PlaywrightCrawler` for pages that depend on client-side JavaScript or require browser interaction. Let the download complete before closing Terminal.

The supplied files do not document a different Apple-silicon command, Homebrew command, or macOS-specific system-package command. Use the command above exactly as shown rather than substituting an undocumented platform-specific procedure.

Step 3: verify the installation

Paste the repository’s verification command:

python -c 'import crawlee; print(crawlee.__version__)'

A successful check prints Crawlee’s installed version and returns you to the Terminal prompt. The supplied package configuration identifies the repository version as `1.10.1`, although the version you receive from PyPI can depend on what is published when you install.

If the command prints a Python traceback or says that `crawlee` cannot be imported, the verification did not succeed. Review the output from the installation command and confirm that the `python` command used for installation is the same one used for verification. The repository does not provide additional supported repair commands, so avoid guessing with elevated privileges or unrelated package-manager commands.

What is now installed

After Route 1, you have the Crawlee Python library and its declared optional features. Crawlee provides:

• HTTP-based crawling and HTML parsing.
• Headless-browser crawling through Playwright.
• Asynchronous crawling based on Python’s `asyncio`.
• Request routing, retries, proxy rotation, sessions, queues, storage, and error handling.
• A `crawlee` command-line entry point.

Browser operation is headless by default according to the supplied configuration. A crawler run creates a `storage/` directory in the current working directory. The default configuration also uses `./storage` as its storage path and purges storage on start by default. Be mindful of your current folder when running a crawler, especially if it already contains a directory named `storage`.

Crawlee can make network requests, follow links, save extracted content, and—when Playwright is used—control a browser. Only crawl sites you are authorized to access, and account for site terms, rate limits, privacy obligations, and applicable law.

Optional Route 2: create a starter project with the Crawlee CLI

The README calls the CLI the quickest way to begin from a prepared template. Its documented CLI route uses `uv`.

Step 1: check whether `uv` is available

Run:

uv --help

If help text appears, continue. If Terminal reports that `uv` is unavailable, the repository directs users to the official `uv` installation guide. No general macOS `uv` installation command is supplied in the root installation instructions, so this guide does not invent one.

Step 2: generate a project

Run the documented command:

uvx 'crawlee[cli]' create my-crawler

This starts the Crawlee CLI and lets you choose among available templates. `my-crawler` is the documented project name and will be used for the generated project directory. Answer the interactive questions shown in Terminal.

Because Route 1’s `all` extra includes the CLI feature, an already installed Crawlee package can also start project creation with:

crawlee create my-crawler

Use one creation command, not both. The first runs the CLI through `uvx`; the second uses the installed `crawlee` entry point.

The generated project’s next commands depend on the package manager selected during the prompts. Its generated README provides the corresponding instructions for Poetry, pip, or `uv`. Follow that generated README exactly. This guide does not substitute a package-module name because the real name is produced from your answers, and a made-up placeholder would not be a reliable copy-paste command.

Development installation is different

Ordinary users do not need the GitHub source tree, development dependencies, linters, type checker, test suite, or documentation website. The contributing guide requires Python 3.10 or later and `uv`, and it defines a development setup that installs all extras, pre-commit hooks, and Playwright components. That is intended for repository contributors, not for simply building crawlers.

Development tests can also be resource intensive, and the repository notes that some unit tests may be flaky in CI or sensitive to macOS resource constraints. Do not treat contributor test setup as a necessary installation check. The one-line import/version command above is the documented end-user verification.

Configuration and next steps

Crawlee supplies defaults, so no configuration is normally required just to install it. Its configuration can be changed with environment variables prefixed `CRAWLEE_`. Documented settings include storage location, logging level, memory and CPU thresholds, browser executable path, browser headless mode, and browser sandbox behavior. Do not disable the browser sandbox merely to troubleshoot installation; the supplied configuration defaults that option to false, meaning the sandbox is not disabled.

For your first crawler, use the project’s first-party Introduction tutorial and examples at `https://crawlee.dev/python/`. Choose an HTTP/BeautifulSoup crawler when pages do not require client-side JavaScript; choose `PlaywrightCrawler` when browser rendering or interaction is necessary.

Your installation is complete when the version verification command prints a version. If you created a starter project, setup is complete after you also follow the generated project’s own README for the package manager you selected.