Apify Scrapers Overview Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality. Quick Decision Tree What do you want to scrape? │ ├── Social Media Posts │ ├── Twitter/X → references/twitter.md │ │ └── Script: scripts/scrape_twitter_ai_trends.py │ │ │ ├── Reddit → references/reddit.md │ │ └── Script: scripts/scrape_reddit_ai_tech.py │ │ │ ├── LinkedIn → references/linkedin.md │ │ └── Script: scripts/scrape_linkedin_posts.py │ │ │ ├── Instagram → references/instagram.md │ │ └── Script: scripts/scrape_instagram.py │ │ └── Modes: profile, posts, hashtag, reels, comments │ │ │ ├── Facebook → references/facebook.md │ │ └── Script: scripts/scrape_facebook.py │ │ └── Modes: page, posts, reviews, groups, marketplace │ │ │ ├── TikTok → references/multi-platform.md │ │ └── Script: scripts/scrape_multi_platform.py │ │ │ └── YouTube → references/multi-platform.md │ └── Script: scripts/scrape_multi_platform.py │ ├── Business/Places │ ├── Google Maps businesses → references/google-maps.md │ │ └── Script: scripts/scrape_google_maps.py │ │ └── Modes: search, place, reviews │ │ │ └── Contact info from websites → references/contact-enrichment.md │ └── Script: scripts/scrape_contact_info.py │ └── Extract: emails, phone numbers, social profiles │ ├── Auto-detect URL type → references/url-detect.md │ └── Script: scripts/scrape_content_by_url.py │ ├── Trend Analysis (NEW) │ └── Enriched trend analysis → workflows/trend-analysis.md │ └── Script: scripts/analyze_trends.py │ └── Features: velocity scoring, lifecycle staging, opportunity scoring │ └── Workflows (multi-step) ├── Lead generation → workflows/lead-generation.md ├── Influencer discovery → workflows/influencer-discovery.md ├── Competitor analysis → workflows/competitor-intel.md ├── Trend analysis → workflows/trend-analysis.md └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md └── Script: scripts/scrape_competitor_ads.py └── Platforms: Facebook Ads Library, Google Ads Transparency └── Features: Spend estimates, creative analysis, benchmarking Environment Setup

Required in .env

APIFY_TOKEN

apify_api_xxxxx Get your API key: https://console.apify.com/account/integrations Common Usage Patterns Scrape Twitter Trends python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50 Scrape Reddit Discussions python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100 Scrape LinkedIn Author python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30 Auto-detect and Scrape URL python scripts/scrape_content_by_url.py "https://x.com/user/status/123456" Scrape Instagram Profile python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20 Scrape Instagram Hashtag python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50 Scrape Instagram Reels python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30 Scrape Facebook Page python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50 Scrape Facebook Reviews python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100 Scrape Facebook Marketplace python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30 Scrape Google Maps Businesses python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50 Scrape Google Maps Reviews python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100 Extract Contact Info from Websites python scripts/scrape_contact_info.py "https://example.com" --depth 2 Bulk Contact Enrichment python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json Scrape Competitor Ads (Single Competitor) python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30 Compare Multiple Competitors' Ads python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json Discover Advertisers by Keyword python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200 Filter Competitor Ads by Media Type python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7 Analyze Trends (NEW)

Analyze specific topic with enrichments

python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90

Discover trending topics in category

python scripts/analyze_trends.py --category technology --discover --top 50

Compare multiple trends

python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare

Export HTML trend report

python scripts/analyze_trends.py

"sustainable fashion"

--format

html

--output

trend_report.html

Cost Estimates

Platform

Actor

Cost per Item

Twitter

kaitoeasyapi/twitter-x-data-tweet-scraper

~$0.00025

Reddit

trudax/reddit-scraper

~$0.001-0.005

LinkedIn

harvestapi/linkedin-post-search

~$0.01-0.05

YouTube

streamers/youtube-scraper

~$0.01-0.05

TikTok

clockworks/tiktok-scraper

~$0.005

Instagram (profile)

apify/instagram-profile-scraper

~$0.005

Instagram (posts)

apify/instagram-post-scraper

~$0.002-0.005

Instagram (hashtag)

apify/instagram-hashtag-scraper

~$0.002-0.005

Instagram (reels)

apify/instagram-reel-scraper

~$0.005-0.01

Instagram (comments)

apify/instagram-comment-scraper

~$0.001-0.003

Facebook (page)

apify/facebook-pages-scraper

~$0.005-0.01

Facebook (posts)

apify/facebook-posts-scraper

~$0.003-0.005

Facebook (reviews)

apify/facebook-reviews-scraper

~$0.002-0.005

Facebook (groups)

apify/facebook-groups-scraper

~$0.005-0.01

Facebook (marketplace)

apify/facebook-marketplace-scraper

~$0.005-0.01

Google Maps (search)

compass/crawler-google-places

~$0.01-0.02

Google Maps (place)

compass/google-maps-business-scraper

~$0.01

Google Maps (reviews)

compass/google-maps-reviews-scraper

~$0.003-0.005

Contact Enrichment

lukaskrivka/contact-info-scraper

~$0.01-0.03

Google Trends

apify/google-trends-scraper

~$0.01

Trend Analysis (multi)

Multiple actors

~$0.50-1.50/run

Facebook Ads Library

apify/facebook-ads-scraper

~$0.75/1K ads

Facebook Ads (alt)

curious_coder/facebook-ads-library-scraper

~$0.50/1K ads

Google Ads Transparency

lexis-solutions/google-ads-scraper

~$1.00/1K ads

Google Ads (alt)

xtech/google-ad-transparency-scraper

~$0.80/1K ads

Output Location

All scraped data saves to

.tmp/

with timestamped filenames:

.tmp/twitter_ai_trends_YYYYMMDD.json

.tmp/reddit_ai_tech_YYYYMMDD.json

.tmp/linkedin_posts_YYYYMMDD_HHMMSS.json

Security Notes

Credential Handling

Store

APIFY_TOKEN

in

.env

file (never commit to git)

Rotate API tokens periodically via Apify Console

Never log or print API tokens in script output

Use environment variables, not hardcoded values

Data Privacy

Scraped data contains only publicly available content

Social media posts may include PII (names, handles, profile info)

Data is stored locally in

.tmp/

directory

No data is retained by Apify after actor run completes

Consider data minimization - only scrape what you need

Access Scopes

Apify tokens have full account access (no granular scopes)

Use separate Apify accounts for different projects if needed

Monitor usage via Apify Console dashboard

Compliance Considerations

Terms of Service

Respect each platform's ToS (Twitter, Reddit, LinkedIn)

Rate Limiting

Actors have built-in rate limiting to avoid bans

Robots.txt

Some actors may bypass robots.txt - use responsibly

GDPR

Scraped PII may be subject to GDPR if EU residents

Ethical Use

Only scrape public data; never bypass authentication

Proxy Ethics

Residential proxies should be used ethically Troubleshooting Common Issues Issue: Actor run failed Symptoms: Script terminates with "Actor run failed" or timeout error Cause: Invalid actor ID, insufficient proxy credits, or actor configuration issue Solution: Verify the actor ID is correct in the script Check Apify Console for actor run logs Ensure proxy settings match actor requirements Try running with default proxy settings first Issue: Empty results returned Symptoms: Script completes but returns 0 items Cause: Content blocked by platform, invalid query, or proxy being detected Solution: Try a different proxy type (residential vs datacenter) Simplify the search query Reduce the number of results requested Check if the platform is blocking scraping attempts Issue: Rate limited by platform Symptoms: Script fails with 429 errors or "rate limited" messages Cause: Too many requests in a short time period Solution: Add delays between requests (actor settings) Reduce concurrent requests Use proxy rotation Wait and retry after a cooldown period Issue: Invalid API token Symptoms: Authentication error or "invalid token" message Cause: Token expired, revoked, or incorrectly set Solution: Regenerate API token in Apify Console Verify token is correctly set in .env file Check for leading/trailing whitespace in token Ensure APIFY_TOKEN environment variable is loaded Issue: Proxy connection errors Symptoms: Connection timeout or proxy errors Cause: Proxy pool exhausted or geo-restriction issues Solution: Switch proxy type (basic, residential, or datacenter) Verify proxy credit balance in Apify Console Try a different proxy country/region Disable proxy to test if that's the root cause Resources Platform References references/twitter.md - Twitter/X scraping details references/reddit.md - Reddit scraping with subreddit targeting references/linkedin.md - LinkedIn post scraping (author or search mode) references/instagram.md - Instagram profile, posts, hashtag, reels, and comments scraping references/facebook.md - Facebook page, posts, reviews, groups, and marketplace scraping references/multi-platform.md - TikTok and YouTube scraping references/url-detect.md - Auto-detect URL type and scrape Business/Places References references/google-maps.md - Google Maps business search, place details, and reviews references/contact-enrichment.md - Extract emails, phone numbers, and social profiles from websites Workflow References workflows/lead-generation.md - Multi-step lead generation workflow workflows/influencer-discovery.md - Find and analyze influencers across platforms workflows/competitor-intel.md - Competitive intelligence gathering workflow workflows/trend-analysis.md - Enriched multi-platform trend analysis with scoring Integration Patterns Scrape and Enrich Skills: apify-scrapers → parallel-research Use case: Scrape social media posts, then enrich with deep research Flow: Scrape Twitter/Reddit for mentions of a topic Extract company names or URLs from posts Use parallel-research to get detailed info on each company Scrape and Summarize Skills: apify-scrapers → content-generation Use case: Create newsletter content from social media trends Flow: Scrape trending AI posts from Twitter Pass scraped data to content-generation summarize Generate a formatted newsletter section Scrape and Archive Skills: apify-scrapers → google-workspace Use case: Save scraped data to Google Drive for team access Flow: Scrape LinkedIn posts from target accounts Format data as CSV or JSON Upload to Google Drive client folder via google-workspace Trend Analysis + Content Strategy Skills: apify-scrapers (trend-analysis) → content-generation Use case: Identify trending topics and create content strategy Flow: Run trend analysis: python scripts/analyze_trends.py "AI productivity" --sources all Review lifecycle stage and opportunity score Use content-generation to create content for high-opportunity trends Focus on emerging trends with high velocity scores Competitive Trend Monitoring Skills: apify-scrapers (trend-analysis) → parallel-research Use case: Monitor competitor visibility in trending topics Flow: Analyze industry trends: python scripts/analyze_trends.py --category "your-industry" --discover Compare your brand vs competitors in those trends Use parallel-research for deep dive on gaps Generate competitive intelligence report