🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί proxy-web-fetch
πŸ¦€ ClawHub

proxy-web-fetch

by @whyhit2005

Proxy Web Page Fetch Tool - Fetches and parses web page content into structured Markdown or text via the OpenClaw Manager proxy. Use when: - Need to fetch an...

Versionv1.0.0
Downloads692
#writing#legal#productivity
TERMINAL
clawhub install proxy-web-fetch

πŸ“– About This Skill

name: proxy-web-fetch description: | Proxy Web Page Fetch Tool - Fetches and parses web page content into structured Markdown or text via the OpenClaw Manager proxy. Use when: - Need to fetch and read the content of a web page by URL - Need to convert web pages to Markdown or plain text format - Need to extract page content with or without images - Need to get page metadata (title, description, keywords) - Need to control caching, image retention, or summary options for fetched content - User asks to "read a URL", "fetch a page", "grab the content of a web page", "scrape" or "crawl" a URL This skill routes all fetch requests through the Manager Web Fetch Proxy (configured via WEB_FETCH_PROXY_URL env var, required), which handles API key management automatically β€” no manual configuration needed. Do NOT confuse with web search β€” this skill fetches a specific URL's content, it does not perform keyword searches. metadata: { "openclaw": { "requires": { "bins": ["curl"], "envs": ["WEB_FETCH_PROXY_URL"] }, }, }

Proxy Web Page Fetch

Fetch and parse web page content via the OpenClaw Manager Web Fetch Proxy. The Manager handles API key injection from encrypted storage automatically β€” no manual key configuration needed.

The proxy URL is configured via the WEB_FETCH_PROXY_URL environment variable (required). If not set, the skill will not be available.

Quick Start

Basic cURL Usage

curl --request POST \
  --url "${WEB_FETCH_PROXY_URL}/" \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.example.com"
  }'

Script Usage

A wrapper shell script is provided for convenience.

# Basic Fetch (returns Markdown by default)
./scripts/proxy_fetch.sh --url "https://www.example.com"
Fetch as plain text, no cache

./scripts/proxy_fetch.sh \
  --url "https://docs.python.org/3/" \
  --format text \
  --no-cache
Fetch with image and link summaries

./scripts/proxy_fetch.sh \
  --url "https://news.example.com/article" \
  --images-summary \
  --links-summary
Fetch without images, disable GFM

./scripts/proxy_fetch.sh \
  --url "https://blog.example.com/post" \
  --no-images \
  --no-gfm

Authentication

No authentication required β€” the proxy reads API keys internally from the Manager's encrypted secrets store.

API Parameter Reference

| Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | url | string | βœ… | - | URL of the web page to fetch | | timeout | integer | - | 20 | Request timeout in seconds | | no_cache | boolean | - | false | Disable caching (true/false) | | return_format | string | - | markdown | Return format: markdown or text | | retain_images | boolean | - | true | Retain images in output (true/false) | | no_gfm | boolean | - | false | Disable GitHub Flavored Markdown (true/false) | | keep_img_data_url | boolean | - | false | Keep image data URLs (true/false) | | with_images_summary | boolean | - | false | Include images summary (true/false) | | with_links_summary | boolean | - | false | Include links summary (true/false) |

Response Structure

The proxy returns JSON with the parsed page content.

{
  "id": "task-id",
  "created": 1704067200,
  "request_id": "request-id",
  "model": "model-name",
  "reader_result": {
    "title": "Page Title",
    "description": "Brief page description",
    "url": "https://www.example.com",
    "content": "Parsed page content (Markdown or text)",
    "external": {
      "stylesheet": {}
    },
    "metadata": {
      "keywords": "page, keywords",
      "viewport": "width=device-width",
      "description": "Meta description",
      "format-detection": "telephone=no"
    }
  }
}

Key Response Fields

| Field | Description | |-------|-------------| | reader_result.content | Main parsed content (body text, images, links) | | reader_result.title | Page title | | reader_result.description | Brief page description | | reader_result.url | Original page URL | | reader_result.metadata | Page metadata (keywords, viewport, etc.) |

Common Use Cases

| Scenario | Command | |----------|---------| | Read a documentation page | --url | | Extract text only (no images) | --url --no-images --format text | | Force fresh fetch (bypass cache) | --url --no-cache | | Get content with all summaries | --url --images-summary --links-summary | | Long page with extended timeout | --url --timeout 60 |

Environment Requirements

  • OpenClaw Manager must be running with the Web Fetch Proxy enabled.
  • WEB_FETCH_PROXY_URL environment variable must be set to the proxy URL (required, no default).
  • curl command must be available in your system path.

    • πŸ’‘ Examples

    Basic cURL Usage

    curl --request POST \
  --url "${WEB_FETCH_PROXY_URL}/" \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.example.com"
  }'

    Script Usage

    A wrapper shell script is provided for convenience.

    # Basic Fetch (returns Markdown by default)
./scripts/proxy_fetch.sh --url "https://www.example.com"
    Fetch as plain text, no cache
    
./scripts/proxy_fetch.sh \
  --url "https://docs.python.org/3/" \
  --format text \
  --no-cache
    Fetch with image and link summaries
    
./scripts/proxy_fetch.sh \
  --url "https://news.example.com/article" \
  --images-summary \
  --links-summary
    Fetch without images, disable GFM
    
./scripts/proxy_fetch.sh \
  --url "https://blog.example.com/post" \
  --no-images \
  --no-gfm