🎁 Get the FREE AI Skills Starter Guide — Subscribe →
BytesAgainBytesAgain
BytesAgain › Skills › web-reader-pro
🦀 ClawHub

web-reader-pro

by @0xcjl

Advanced web content extraction skill for OpenClaw using multi-tier fallback strategy (Jina → Scrapling → WebFetch) with intelligent routing, caching, qualit...

Versionv1.0.0
Downloads456
Stars⭐ 2
#writing#legal#gaming
TERMINAL
clawhub install web-reader-pro

📖 About This Skill

name: web-reader-pro description: "Advanced web content extraction skill for OpenClaw using multi-tier fallback strategy (Jina → Scrapling → WebFetch) with intelligent routing, caching, quality scoring, and domain learning. Use when: reading article content, extracting web page text, scraping dynamic JS-heavy pages, or fetching WeChat official account articles." metadata: author: 0xcjl version: "1.0.0"

Web Reader Pro - OpenClaw Skill

Overview

Web Reader Pro is an advanced web content extraction skill for OpenClaw that uses a multi-tier fallback strategy with intelligent routing, caching, and quality assessment.

Features

1. Three-Tier Fallback Strategy

  • Tier 1: Jina Reader API - Fast, reliable, best for most websites
  • Tier 2: Scrapling + Playwright - Dynamic content rendering for JS-heavy sites
  • Tier 3: WebFetch Fallback - Basic extraction for simple pages

    • 2. Jina Quota Monitoring

  • Tracks API call count with persistent counter
  • Warning alerts when approaching quota limits
  • Automatic fallback to lower-tier methods when quota exhausted

    • 3. Smart Cache Layer

  • Short-term caching (configurable TTL, default 1 hour)
  • Cache key based on URL hash
  • Reduces redundant API calls

    • 4. Extraction Quality Scoring

  • Scores based on: word count, title detection, content density
  • Minimum quality threshold (default: 200 words + valid title)
  • Auto-escalation to next tier if quality below threshold

    • 5. Domain-Level Routing Learning

  • Learns optimal extraction tier per domain
  • Persists learned routes in local JSON database
  • Adapts based on historical success rates

    • 6. Retry with Exponential Backoff

  • Configurable max retries per tier (default: 3)
  • Exponential backoff: 1s, 2s, 4s, 8s...
  • Respects rate limits and transient failures

    • Installation

    # Install dependencies
pip install -r requirements.txt
    Install Scrapling (requires Node.js)
    
./scripts/install_scrapling.sh
    Or install Scrapling manually
    
npm install -g @scrapinghub/scrapling

    Usage

    Basic Usage

    from scripts.web_reader_pro import WebReaderPro
    reader = WebReaderPro()
result = reader.fetch("https://example.com")
print(result['title'])
print(result['content'])

    Advanced Configuration

    reader = WebReaderPro(
    jina_api_key="your-jina-key",      # Optional: set via env JINA_API_KEY
    cache_ttl=3600,                      # Cache TTL in seconds (default: 3600)
    quality_threshold=200,               # Min word count for quality (default: 200)
    max_retries=3,                       # Max retries per tier (default: 3)
    enable_learning=True,                # Enable domain learning (default: True)
    scrapling_path="/usr/local/bin/scrapling"  # Path to scrapling binary
)

    Result Format

    {
    "title": "Page Title",
    "content": "Extracted content in markdown...",
    "url": "https://example.com",
    "tier_used": "jina|scrapling|webfetch",
    "quality_score": 85,
    "cached": False,
    "domain_learned_tier": "jina",
    "extracted_at": "2024-01-01T00:00:00Z"
}

    Environment Variables

    | Variable | Description | Default | |----------|-------------|---------| | JINA_API_KEY | Jina Reader API key | Required for Tier 1 | | WEB_READER_CACHE_DIR | Cache directory path | ~/.openclaw/cache/web-reader-pro/ | | WEB_READER_LEARNING_DB | Learning database path | ~/.openclaw/data/web-reader-pro/routes.json | | WEB_READER_JINA_QUOTA | Jina quota limit | 100000 |

    API Reference

    WebReaderPro.fetch(url, force_refresh=False)

    Fetch and extract content from a URL.

    Parameters:

  • url (str): Target URL
  • force_refresh (bool): Bypass cache if True

    • Returns: Dict with title, content, metadata

    WebReaderPro.fetch_with_tier(url, preferred_tier)

    Fetch using a specific tier (bypassing automatic selection).

    Parameters:

  • url (str): Target URL
  • preferred_tier (str): "jina", "scrapling", or "webfetch"

    • WebReaderPro.get_jina_status()

    Get current Jina API quota usage.

    Returns: Dict with count, limit, percentage, warnings

    WebReaderPro.clear_cache(url=None)

    Clear cache for specific URL or all URLs.

    Parameters:

  • url (str, optional): Specific URL to clear, or None for all

    • WebReaderPro.get_domain_routes()

    Get learned domain-to-tier mappings.

    Returns: Dict of domain -> preferred tier

    Tier Comparison

    | Tier | Speed | JS Rendering | Best For | Cost | |------|-------|--------------|----------|------| | Jina | Fast | No | Static pages, articles | API calls | | Scrapling | Medium | Yes | SPAs, dynamic content | CPU | | WebFetch | Fastest | No | Simple pages, fallbacks | Free |

    License

    MIT

    💡 Examples

    Basic Usage

    from scripts.web_reader_pro import WebReaderPro
    reader = WebReaderPro()
result = reader.fetch("https://example.com")
print(result['title'])
print(result['content'])

    Advanced Configuration

    reader = WebReaderPro(
    jina_api_key="your-jina-key",      # Optional: set via env JINA_API_KEY
    cache_ttl=3600,                      # Cache TTL in seconds (default: 3600)
    quality_threshold=200,               # Min word count for quality (default: 200)
    max_retries=3,                       # Max retries per tier (default: 3)
    enable_learning=True,                # Enable domain learning (default: True)
    scrapling_path="/usr/local/bin/scrapling"  # Path to scrapling binary
)