🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Website Scraper Pro
πŸ¦€ ClawHub

Website Scraper Pro

by @youpele52

Run a local script to scrape a single web page into clean markdown or deterministic JSON with Crawl4AI. Use when: user needs direct page retrieval from a URL...

Versionv0.1.1
Downloads764
Installs1
#clawhub
TERMINAL
clawhub install website-scraper-pro

πŸ“– About This Skill

name: website-scraper-pro description: "Run a local script to scrape a single web page into clean markdown or deterministic JSON with Crawl4AI. Use when: user needs direct page retrieval from a URL, JS-aware single-page scraping, or deterministic query-focused narrowing without internal AI processing. Invoke by reading this SKILL.md then running: uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py " homepage: https://github.com/unclecode/crawl4ai metadata: {"clawdbot":{"emoji":"🌐","requires":{"bins":["uv"]}}}

Skill: Website Scraper Pro

When to use

  • The user wants the content of a single web page from a specific URL.
  • The user wants clean markdown extracted from an article, docs page, blog post, or landing page.
  • The user wants a JS-aware scrape for a page that depends on client-side rendering.
  • The user wants deterministic query-focused narrowing of one page without using an AI model inside the skill.
  • The user wants structured JSON output with markdown, title, links, and metadata.

    • When NOT to use

  • The user wants a broad web search across multiple sources.
  • The user wants a site-wide crawl, recursive crawl, or multi-page extraction workflow.
  • The user wants AI summarization, synthesis, or answer generation inside the scraper itself.
  • The user wants authenticated browser automation or interactive form submission.

    • Commands

    Scrape a page to markdown

    uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py

    Scrape a JS-heavy page

    uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py  --js

    Scrape a page and narrow by query

    uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py  --query ""

    Return deterministic JSON

    uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py  --format json

    Examples

    # Default markdown scrape
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com
    JS-aware scrape
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --js
    Query-focused retrieval
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --query "documentation examples"
    JSON output
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --format json

    Output

  • Default output is clean markdown for a single page.
  • --query keeps the output deterministic and non-LLM.
  • --format json returns deterministic JSON with fields such as title, url, markdown, links, and metadata when available.

    • Notes

  • This v1 does not use AI models internally. It is a deterministic retrieval tool only.
  • The skill is single-page only. It does not do deep crawling, site maps, schema extraction, or RAG.
  • uv run reads the inline # /// script dependency block in main.py and installs crawl4ai in an isolated environment.
  • If browser setup is missing, run one-time setup commands such as:
    • - uv run --with crawl4ai crawl4ai-setup - uv run --with crawl4ai python -m playwright install chromium
  • Do NOT use web search for this workflow when a direct URL is available.
  • Call uv run src/main.py directly as shown above.

    • ⚑ When to Use

    TriggerAction
    - The user wants clean markdown extracted from an article, docs page, blog post, or landing page.
    - The user wants a JS-aware scrape for a page that depends on client-side rendering.
    - The user wants deterministic query-focused narrowing of one page without using an AI model inside the skill.
    - The user wants structured JSON output with markdown, title, links, and metadata.

    πŸ’‘ Examples

    # Default markdown scrape
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com
    JS-aware scrape
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --js
    Query-focused retrieval
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --query "documentation examples"
    JSON output
    
uv run /root/.openclaw/workspace/skills/website-scraper-pro/src/main.py https://example.com --format json

    πŸ“‹ Tips & Best Practices

  • This v1 does not use AI models internally. It is a deterministic retrieval tool only.
  • The skill is single-page only. It does not do deep crawling, site maps, schema extraction, or RAG.
  • uv run reads the inline # /// script dependency block in main.py and installs crawl4ai in an isolated environment.
  • If browser setup is missing, run one-time setup commands such as:
    • - uv run --with crawl4ai crawl4ai-setup - uv run --with crawl4ai python -m playwright install chromium
  • Do NOT use web search for this workflow when a direct URL is available.
  • Call uv run src/main.py directly as shown above.