Asynchronous crawl and LLM jobs

Runs background crawl or LLM-extraction jobs with polling or retrying webhook notifications.

Parent repository GitHub

Type: workflow / service API
Repository: unclecode/crawl4ai
Readiness: Usable for development and self-hosted workflows, but deployments should use a current security release and receive careful hardening, resource controls, and legal review.
Keywords: 6

Location

Repository path

deploy/docker/README.md

Invocation

`POST /crawl/job`; `GET /crawl/job/{task_id}`; `POST /llm/job`

Setup

Installation / activation

Submit a job to the Docker API, optionally including `webhook_config`.

Keywords

jobswebhooksbackgroundcrawlLLM extractionpolling

Repository context

unclecode/crawl4ai

Apache-2.0 Python web-crawling and scraping framework that produces LLM-ready Markdown and structured data through browser automation, adaptive and deep crawling, CLI, SDK, Docker API, and a documented MCP server that exposes crawl, HTML, screenshot, PDF, JavaScript, and library-context tools directly to Claude Code.

web crawlingweb scrapingbrowser automationdata extractionRAG ingestionagent toolingself-hosted infrastructureweb-to-Markdown

Open full repository research