Crawl4AI Overview Crawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both CLI (recommended for quick tasks) and Python SDK (for programmatic control). Choose your interface: CLI ( crwl ) - Quick, scriptable commands: CLI Guide Python SDK - Full programmatic control: SDK Guide Quick Start Installation pip install crawl4ai crawl4ai-setup

Verify installation

crawl4ai-doctor CLI (Recommended)

Basic crawling - returns markdown

crwl https://example.com

Get markdown output

crwl https://example.com -o markdown

JSON output with cache bypass

crwl https://example.com -o json -v --bypass-cache

See more examples

crwl

--example

Python SDK

import

asyncio

from

crawl4ai

import

AsyncWebCrawler

async

def

main

(

)

:

async

with

AsyncWebCrawler

(

)

as

crawler

:

result

=

await

crawler

.

arun

(

"https://example.com"

)

print

(

result

.

markdown

[

:

500

]

)

asyncio

.

run

(

main

(

)

)

For SDK configuration details:

SDK Guide - Configuration

(lines 61-150)

Core Concepts

Configuration Layers

Both CLI and SDK use the same underlying configuration:

Concept

CLI

SDK

Browser settings

-B browser.yml

or

-b "param=value"

BrowserConfig(...)

Crawl settings

-C crawler.yml

or

-c "param=value"

CrawlerRunConfig(...)

Extraction

-e extract.yml -s schema.json

extraction_strategy=...

Content filter

-f filter.yml

markdown_generator=...

Key Parameters

Browser Configuration:

headless

Run with/without GUI

viewport_width/height

Browser dimensions

user_agent

Custom user agent

proxy_config

Proxy settings

Crawler Configuration:

page_timeout

Max page load time (ms)

wait_for

CSS selector or JS condition to wait for

cache_mode

bypass, enabled, disabled

js_code

JavaScript to execute

css_selector

Focus on specific element For complete parameters: CLI Config | SDK Config Output Content Every crawl returns: markdown - Clean, formatted markdown html - Raw HTML links - Internal and external links discovered media - Images, videos, audio found extracted_content - Structured data (if extraction configured) Markdown Generation (Primary Use Case) Crawl4AI excels at generating clean, well-formatted markdown: CLI

Basic markdown

crwl https://docs.example.com -o markdown

Filtered markdown (removes noise)

crwl https://docs.example.com -o markdown-fit

With content filter

crwl https://docs.example.com -f filter_bm25.yml -o markdown-fit Filter configuration:

filter_bm25.yml (relevance-based)

type : "bm25" query : "machine learning tutorials" threshold : 1.0 Python SDK from crawl4ai . content_filter_strategy import BM25ContentFilter from crawl4ai . markdown_generation_strategy import DefaultMarkdownGenerator bm25_filter = BM25ContentFilter ( user_query = "machine learning" , bm25_threshold = 1.0 ) md_generator = DefaultMarkdownGenerator ( content_filter = bm25_filter ) config = CrawlerRunConfig ( markdown_generator = md_generator ) result = await crawler . arun ( url , config = config ) print ( result . markdown . fit_markdown )

Filtered

print ( result . markdown . raw_markdown )

Original

For content filters: Content Processing (lines 2481-3101) Data Extraction 1. Schema-Based CSS Extraction (Most Efficient) No LLM required - fast, deterministic, cost-free. CLI:

Generate schema once (uses LLM)

python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

Use schema for extraction (no LLM)

crwl https://shop.com -e extract_css.yml -s product_schema.json -o json Schema format: { "name" : "products" , "baseSelector" : ".product-card" , "fields" : [ { "name" : "title" , "selector" : "h2" , "type" : "text" } , { "name" : "price" , "selector" : ".price" , "type" : "text" } , { "name" : "link" , "selector" : "a" , "type" : "attribute" , "attribute" : "href" } ] } 2. LLM-Based Extraction For complex or irregular content: CLI:

extract_llm.yml

type : "llm" provider : "openai/gpt-4o-mini" instruction : "Extract product names and prices" api_token : "your-token" crwl https://shop.com -e extract_llm.yml -o json For extraction details: Extraction Strategies (lines 4522-5429) Advanced Patterns Dynamic Content (JavaScript-Heavy Sites) CLI: crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000" Crawler config:

crawler.yml

wait_for : "css:.ajax-content" scan_full_page : true page_timeout : 60000 delay_before_return_html : 2.0 Multi-URL Processing CLI (sequential): for url in url1 url2 url3 ; do crwl " $url " -o markdown ; done Python SDK (concurrent): urls = [ "https://site1.com" , "https://site2.com" , "https://site3.com" ] results = await crawler . arun_many ( urls , config = config ) For batch processing: arun_many() Reference (lines 1057-1224) Session & Authentication CLI:

login_crawler.yml

session_id : "user_session" js_code : | document.querySelector('#username').value = 'user'; document.querySelector('#password').value = 'pass'; document.querySelector('#submit').click(); wait_for : "css:.dashboard"

Login

crwl https://site.com/login -C login_crawler.yml

Access protected content (session reused)

crwl https://site.com/protected -c "session_id=user_session" For session management: Advanced Features (lines 5429-5940) Anti-Detection & Proxies CLI:

browser.yml

headless : true proxy_config : server : "http://proxy:8080" username : "user" password : "pass" user_agent_mode : "random" crwl https://example.com -B browser.yml Common Use Cases Documentation to Markdown crwl https://docs.example.com -o markdown

docs.md E-commerce Product Monitoring

Generate schema once

python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

Monitor (no LLM costs)

crwl https://shop.com -e extract_css.yml -s schema.json -o json News Aggregation

Multiple sources with filtering

for url in news1.com news2.com news3.com ; do crwl "https:// $url " -f filter_bm25.yml -o markdown-fit done Interactive Q&A

First view content

crwl https://example.com -o markdown

Then ask questions

crwl https://example.com -q "What are the main conclusions?" crwl https://example.com -q "Summarize the key points" Resources Provided Scripts scripts/extraction_pipeline.py - Schema generation and extraction scripts/basic_crawler.py - Simple markdown extraction scripts/batch_crawler.py - Multi-URL processing Reference Documentation Document Purpose CLI Guide Command-line interface reference SDK Guide Python SDK quick reference Complete SDK Reference Full API documentation (5900+ lines) Best Practices Start with CLI for quick tasks, SDK for automation Use schema-based extraction - 10-100x more efficient than LLM Enable caching during development - --bypass-cache only when needed Set appropriate timeouts - 30s normal, 60s+ for JS-heavy sites Use content filters for cleaner, focused markdown Respect rate limits - Add delays between requests Troubleshooting JavaScript Not Loading crwl https://example.com -c "wait_for=css:.dynamic-content,page_timeout=60000" Bot Detection Issues crwl https://example.com -B browser.yml

browser.yml

headless : false viewport_width : 1920 viewport_height : 1080 user_agent : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" Content Not Extracted

Debug: see full output

crwl https://example.com -o all -v

Try different wait strategy

crwl https://example.com -c "wait_for=js:document.querySelector('.content')!==null" Session Issues

Verify session

crwl https://site.com -c "session_id=test" -o all | grep -i session For comprehensive API documentation, see Complete SDK Reference .