Emerging Topic Scout

Emerging Topic Scout

A real-time monitoring system for identifying "incubation period" research hotspots in biological and medical sciences before they are defined by mainstream journals.

Overview

This skill continuously monitors:

It uses trend analysis algorithms to detect sudden spikes in topic frequency, cross-platform mentions, and emerging keyword clusters.

⚠️ Network Access Notice

bioRxiv and medRxiv are currently protected by Cloudflare JavaScript Challenge, which prevents programmatic RSS access. As a workaround, this skill now supports arXiv q-bio (Quantitative Biology) as an alternative data source.

Recommended usage:

# Use arXiv for reliable data fetching
python scripts/main.py --sources arxiv --days 30
bioRxiv/medRxiv may return 0 results due to Cloudflare protection
python scripts/main.py --sources biorxiv medrxiv --days 30  # May not work

Installation

cd /Users/z04030865/.openclaw/workspace/skills/emerging-topic-scout
pip install -r scripts/requirements.txt

Usage

Basic Scan (Recommended: Use arXiv)

python scripts/main.py --sources arxiv --days 7 --output json

Legacy bioRxiv/medRxiv (May not work due to Cloudflare)

python scripts/main.py --sources biorxiv medrxiv --days 7 --output json

Advanced Configuration (arXiv Recommended)

python scripts/main.py \
  --sources arxiv \
  --keywords "CRISPR,gene editing,machine learning" \
  --days 14 \
  --min-score 0.7 \
  --output markdown \
  --notify

Legacy Configuration (bioRxiv/medRxiv - May not work)

python scripts/main.py \
  --sources biorxiv medrxiv \
  --keywords "CRISPR,gene editing,long COVID" \
  --days 14 \
  --min-score 0.7 \
  --output markdown \
  --notify
Note: bioRxiv/medRxiv may return 0 results due to Cloudflare protection
Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| --sources | list | arxiv | Data sources to monitor (arxiv recommended due to Cloudflare issues with biorxiv/medrxiv) |
| --keywords | string | (auto-detect) | Comma-separated keywords to track |
| --days | int | 7 | Lookback period in days |
| --min-score | float | 0.6 | Minimum trending score (0-1) |
| --max-topics | int | 20 | Maximum topics to return |
| --output | string | markdown | Output format: json, markdown, csv |
| --notify | flag | false | Send notification for high-priority topics |
| --config | path | config.yaml | Path to configuration file |
Output Format
JSON Output

Markdown Output

Emerging Topics Report - 2026-02-06

🔥 High Priority Topics

1. Gene Editing Therapy (Score: 0.89)

#### Key Papers 1. "New CRISPR variant shows promise" - Smith J. et al. - DOI: 10.1101/2026.01.15.xxxxx - Source: bioRxiv

Configuration File
Create config.yaml for persistent settings:

trending: min_papers_threshold: 5 velocity_window_days: 3 novelty_weight: 0.4 momentum_weight: 0.6

keywords: auto_detect: true custom_trackers: - "artificial intelligence" - "machine learning" - "single cell" - "spatial transcriptomics"

output: default_format: markdown save_history: true history_path: "./data/history.json"

notifications: enabled: false high_score_threshold: 0.8

Trending Score Algorithm
The trending score (0-1) is calculated using:

Where:

API Endpoints
bioRxiv API
Base: https://api.biorxiv.org/
Details: /details/[server]/[DOI]/[format]
Publication: /pub/[DOI]/[format]
medRxiv API
Same structure as bioRxiv
Data Storage
Historical data is stored in data/history.json for:
Trend comparison
Velocity calculation
Duplicate detection
Examples
Example 1: Quick Daily Scan (arXiv - Recommended)

Example 2: Daily Scan with bioRxiv (May not work)

Note: May return 0 results due to Cloudflare protection

Example 2: Weekly Deep Analysis

python scripts/main.py \
  --days 7 \
  --min-score 0.7 \
  --max-topics 50 \
  --output json \
  > weekly_report.json

Example 3: Track Specific Research Area

python scripts/main.py \
  --keywords "Alzheimer,neurodegeneration,amyloid" \
  --days 30 \
  --min-score 0.5

Known Issues

bioRxiv/medRxiv Cloudflare Protection

Attempted Solutions: 1. ✅ Added browser User-Agent headers → Failed (Cloudflare detects bot) 2. ✅ Added complete browser headers (Accept, Accept-Language, etc.) → Failed 3. ❌ Browser automation (Selenium/Playwright) → Not implemented (complex, heavy dependency)

Workaround: ✅ Use arXiv instead

Usage:

# Recommended: Use arXiv
python scripts/main.py --sources arxiv --days 30
Not working: bioRxiv/medRxiv
python scripts/main.py --sources biorxiv medrxiv --days 30  # Returns 0 papers

Troubleshooting

Rate Limiting

Missing Papers (0 results from bioRxiv/medRxiv)

RSS Feed Access Denied

Low Trending Scores

References

See references/README.md for:

License

MIT License - Part of OpenClaw Skills Collection

Risk Assessment

| Risk Indicator | Assessment | Level | |----------------|------------|-------| | Code Execution | Python scripts with tools | High | | Network Access | External API calls | High | | File System Access | Read/write data | Medium | | Instruction Tampering | Standard prompt guidelines | Low | | Data Exposure | Data handled securely | Medium |

Security Checklist

Prerequisites

# Python dependencies
pip install -r requirements.txt

Evaluation Criteria

Success Metrics

Test Cases

Lifecycle Status

Example 1: Quick Daily Scan (arXiv - Recommended)

python scripts/main.py --sources arxiv --days 1 --output markdown

Example 2: Daily Scan with bioRxiv (May not work)

python scripts/main.py --sources biorxiv --days 1 --output markdown
Note: May return 0 results due to Cloudflare protection
Example 2: Weekly Deep Analysis

Example 3: Track Specific Research Area

# Python dependencies
pip install -r requirements.txt

Rate Limiting

Missing Papers (0 results from bioRxiv/medRxiv)

RSS Feed Access Denied

Low Trending Scores