Bright Data - Web Scraper Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA. Setup 1. Get your API Key: Get a key from Bright Data Dashboard . 2. Create a Web Unlocker zone: Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone". 3. Set environment variables: export BRIGHTDATA_API_KEY = "your-api-key" export BRIGHTDATA_UNLOCKER_ZONE = "your-zone-name" Usage bash scripts/scrape.sh "url" Parameters: url (required): The webpage URL to scrape Examples:

Scrape a news article

bash scripts/scrape.sh "https://example.com/article"

Scrape a product page

bash scripts/scrape.sh "https://shop.example.com/product/123" Output Format Returns clean markdown content extracted from the webpage:

Page Title Main content of the page converted to markdown format...

Section Heading

More content...

Features

Bot Detection Bypass

Automatically handles anti-bot measures

CAPTCHA Solving

Bypasses CAPTCHA challenges

Clean Markdown

Returns well-formatted markdown content

JavaScript Rendering

Handles JavaScript-heavy pages Dependencies curl - For API requests