apify/crawlee-python

Selection

Select when

• Select it for Python 3.10+ crawling projects that need HTTP and browser backends.
• Select it when persistent request queues and resumable state are important.
• Select it when proxy rotation, sessions, retries, and block handling are needed.
• Select BeautifulSoupCrawler when pages do not require client-side JavaScript.
• Select PlaywrightCrawler when content or interactions require JavaScript execution.
• Select it when crawlers must integrate directly into an asyncio application.
• Select it when generated starter projects would accelerate setup.
• Select it when outputs include both structured datasets and downloaded files.
• Select it when local execution and optional Apify deployment are both desirable.
• Inference: select it alongside Claude Code when Claude is being used to develop a Python scraping application rather than when Claude needs an immediately registered tool.

Boundaries

Avoid when

• Avoid it when a native Claude Code tool, skill, hook, or MCP integration is required.
• Avoid browser extras when plain HTTP retrieval is sufficient and dependency size matters.
• Avoid it when JavaScript-heavy automation cannot accommodate browser binaries and runtime resource use.
• Avoid it when site terms, robots policies, law, or authorization do not permit the intended collection.
• Avoid it when extraction must be fully no-code because handlers are written as Python code.
• Avoid it when Python 3.9 or earlier must be supported.
• Avoid default local storage behavior when writing a storage directory in the working directory is undesirable.
• Avoid disabling the browser sandbox unless the deployment threat model explicitly accepts the added risk.
• Avoid assuming anti-bot evasion will work reliably because the README's claim was not independently validated.

Strengths

Capabilities

Risk profile

Risks and limitations

• Uncertainty: only selected first-party files were supplied, so crawler internals, proxy logic, storage backends, CLI implementation, security policy, CI workflows, and integration tests were not comprehensively inspected.
• Uncertainty: repository update and push timestamps extend into 2026 relative to this analysis context, so current activity, version state, issue counts, and adoption signals cannot be independently reconciled.
• Uncertainty: no supplied file demonstrates native Claude Code integration, so classification as a companion development library is an inference.
• The README's claim that default crawlers appear human-like and evade modern bot protections was not independently validated and may vary by target.
• Browser crawling requires optional Playwright dependencies and browser installation, increasing setup size and runtime resource use.
• Some unit tests are documented as potentially flaky because of isolation, resource, code, or test-design issues.
• Default example runs create a storage directory in the current working directory, and purge-on-start defaults to true for storage clients.
• The configuration can disable the browser sandbox, which increases risk if enabled without compensating isolation.
• Proxy, browser, parser, database, Redis, observability, Stagehand, and AI features depend on separately versioned third-party packages or services.
• Web crawling can encounter legal, contractual, robots-policy, authentication, rate-limit, privacy, and anti-bot constraints not resolved by the library.
• No independent security audit, privacy assessment, Claude compatibility matrix, or cross-environment reliability benchmark was supplied.
• The core package requires Python 3.10 or newer.

Crawler code processes untrusted remote content and may operate authenticated browser sessions, proxies, or external services, so handlers, downloaded files, secrets, and destinations require review. Playwright configuration supports disabling the browser sandbox, but the default is false; enabling it broadens exposure. Optional dependencies and browser binaries increase the supply-chain and attack surface. The supplied files do not establish a security audit or complete threat model.