Web Scraper Skill

Web Scraper Skill (Apify + Firecrawl)

This skill helps Openclaw scrape and extract data from websites using two powerful APIs:

Quick Decision Guide: Apify vs Firecrawl

| Use Case | Recommended Tool | |---|---| | Scrape a single page into markdown/JSON | Firecrawl /scrape | | Crawl an entire website (follow links) | Firecrawl /crawl | | Map all URLs on a site | Firecrawl /map | | Search web + scrape results | Firecrawl /search | | Scrape Instagram / TikTok / Twitter | Apify (social actors) | | Scrape Google Maps / reviews | Apify (compass/crawler-google-places) | | Scrape Amazon products | Apify (apify/amazon-scraper) | | Scrape Google Search results | Apify (apify/google-search-scraper) | | Custom actor / any Apify Store actor | Apify |

Authentication

Both APIs require API keys passed via headers. Always ask the user for their key if not provided.

Firecrawl: Authorization: Bearer fc-YOUR_API_KEY Apify: Authorization: Bearer YOUR_APIFY_TOKEN (or ?token=YOUR_TOKEN in URL)

Firecrawl API Reference

Base URL: https://api.firecrawl.dev/v2

1. Scrape a Single Page

POST /v2/scrape
Authorization: Bearer fc-YOUR_API_KEY
Content-Type: application/json
{
  "url": "https://example.com",
  "formats": ["markdown"],          // Options: markdown, html, rawHtml, links, screenshot, json
  "onlyMainContent": true,          // Strips nav/footer/ads
  "waitFor": 0,                     // ms to wait before scraping (for JS-heavy pages)
  "timeout": 30000,                 // ms
  "blockAds": true,
  "proxy": "auto"                   // "auto", "basic", or "stealth"
}

2. Crawl an Entire Website

POST /v2/crawl
{
  "url": "https://docs.example.com",
  "limit": 50,                      // Max pages
  "maxDepth": 3,
  "allowExternalLinks": false,
  "scrapeOptions": {
    "formats": ["markdown"],
    "onlyMainContent": true
  }
}

Poll status:

GET /v2/crawl/{crawl-job-id}

3. Map a Website's URLs

POST /v2/map
{ "url": "https://example.com" }

4. Search + Scrape in One Call

POST /v2/search
{
  "query": "best web scraping tools 2025",
  "limit": 5,
  "scrapeOptions": { "formats": ["markdown"] }
}

5. Batch Scrape Multiple URLs

POST /v2/batch/scrape
{
  "urls": ["https://a.com", "https://b.com"],
  "formats": ["markdown"]
}

Apify API Reference

Base URL: https://api.apify.com/v2 Auth: Pass token as query param ?token=YOUR_TOKEN or in Authorization header.

Core Workflow

1. Run an Actor (Async)

POST /v2/acts/{actorId}/runs?token=YOUR_TOKEN
Content-Type: application/json
{ ...actor-specific input... }

{
  "data": {
    "id": "RUN_ID",
    "status": "RUNNING",
    "defaultDatasetId": "DATASET_ID"
  }
}

Common Actor IDs:

2. Poll Run Status

GET /v2/acts/{actorId}/runs/{runId}?token=YOUR_TOKEN

3. Fetch Results

GET /v2/datasets/{datasetId}/items?token=YOUR_TOKEN&format=json

4. Run Synchronously (≤5 minutes)

POST /v2/acts/{actorId}/run-sync-get-dataset-items?token=YOUR_TOKEN
Content-Type: application/json
{ ...actor input... }

Common Actor Inputs

Google Search Scraper:

{ "queries": "web scraping tools", "maxPagesPerQuery": 1, "resultsPerPage": 10 }

Google Maps Scraper:

{ "searchStringsArray": ["restaurants in Mumbai"], "maxCrawledPlaces": 20 }

Web Scraper (generic):

{
  "startUrls": [{ "url": "https://example.com" }],
  "pageFunction": "async function pageFunction(context) { const $ = context.jQuery; return { title: $('title').text() }; }",
  "maxPagesPerCrawl": 10
}

Output Handling

Code Templates

See references/code-templates.md for ready-to-run Python and JavaScript code for both APIs.

Error Handling

Best Practices

1. Start small — test with 1 URL or a small limit before scaling 2. Use onlyMainContent: true in Firecrawl to remove nav/footer noise 3. Choose async for large jobs — don't use sync endpoints for crawls with 50+ pages 4. Store API keys securely — never hardcode them; use environment variables 5. Check rate limits — Firecrawl: varies by plan; Apify: 250k requests/min global 6. Prefer Firecrawl for LLM pipelines — markdown output is clean and ready for RAG/AI 7. Prefer Apify for social/structured data — specialized actors handle anti-bot better

1. Start small — test with 1 URL or a small limit before scaling 2. Use onlyMainContent: true in Firecrawl to remove nav/footer noise 3. Choose async for large jobs — don't use sync endpoints for crawls with 50+ pages 4. Store API keys securely — never hardcode them; use environment variables 5. Check rate limits — Firecrawl: varies by plan; Apify: 250k requests/min global 6. Prefer Firecrawl for LLM pipelines — markdown output is clean and ready for RAG/AI 7. Prefer Apify for social/structured data — specialized actors handle anti-bot better