finviz-crawler

Run the crawler

# Default: ~/workspace/finviz/, 7-day expiry
python3 scripts/finviz_crawler.py
Custom paths and settings
python3 scripts/finviz_crawler.py --db /path/to/finviz.db --articles-dir /path/to/articles/
Keep 30 days of articles
python3 scripts/finviz_crawler.py --expiry-days 30
Never auto-delete (keep everything)
python3 scripts/finviz_crawler.py --expiry-days 0
Custom crawl interval (default: 300s)
python3 scripts/finviz_crawler.py --sleep 600

Query articles

# Last 24 hours of headlines
python3 scripts/finviz_query.py --hours 24
Titles only (compact, good for LLM summarization)
python3 scripts/finviz_query.py --hours 12 --titles-only
With full article content
python3 scripts/finviz_query.py --hours 12 --with-content
List downloaded articles with content status
python3 scripts/finviz_query.py --list-articles --hours 24
Database stats
python3 scripts/finviz_query.py --stats

Manage tickers

# List all tracked tickers
python3 scripts/finviz_query.py --list-tickers
Add single ticker (auto-generates keywords from symbol)
python3 scripts/finviz_query.py --add-ticker NVDA
Add with custom keywords
python3 scripts/finviz_query.py --add-ticker "NVDA:nvidia,jensen huang"
Add multiple tickers (batch)
python3 scripts/finviz_query.py --add-ticker NVDA TSLA AAPL
python3 scripts/finviz_query.py --add-ticker "NVDA:nvidia,jensen" "TSLA:tesla,elon musk"
Remove tickers (batch)
python3 scripts/finviz_query.py --remove-ticker NVDA TSLA
Custom DB path
python3 scripts/finviz_query.py --list-tickers --db /path/to/finviz.db

Tickers are stored in the tickers table inside finviz.db alongside articles. The crawler reads this table each cycle to know which ticker pages to scrape.

Configuration

| Setting | CLI flag | Env var | Default | |---------|----------|---------|---------| | Database path | --db | — | ~/workspace/finviz/finviz.db | | Articles directory | --articles-dir | — | ~/workspace/finviz/articles/ | | Crawl interval | --sleep | — | 300 (5 min) | | Article expiry | --expiry-days | FINVIZ_EXPIRY_DAYS | 7 days | | Timezone | — | FINVIZ_TZ or TZ | System default |

| Setting | CLI flag | Env var | Default | |---------|----------|---------|---------| | Database path | --db | — | ~/workspace/finviz/finviz.db | | Articles directory | --articles-dir | — | ~/workspace/finviz/articles/ | | Crawl interval | --sleep | — | 300 (5 min) | | Article expiry | --expiry-days | FINVIZ_EXPIRY_DAYS | 7 days | | Timezone | — | FINVIZ_TZ or TZ | System default |