Sitemap Generator
by @johnnywang2001
Generate XML sitemaps by crawling a website. Use when a user needs to create a sitemap.xml for SEO, audit site structure, discover all pages on a domain, or...
clawhub install sitemap-generatorπ About This Skill
name: sitemap-generator description: Generate XML sitemaps by crawling a website. Use when a user needs to create a sitemap.xml for SEO, audit site structure, discover all pages on a domain, or generate a sitemap for submission to Google Search Console or other search engines. Handles BFS crawling with configurable depth, page limits, and polite delays.
Sitemap Generator
Crawl any website and produce a standards-compliant XML sitemap ready for search engine submission.
Quick Start
python3 scripts/sitemap_gen.py https://example.com
Output: sitemap.xml in the current directory.
Commands
# Basic β crawl and write sitemap.xml
python3 scripts/sitemap_gen.py https://example.comCustom output path
python3 scripts/sitemap_gen.py https://example.com -o /tmp/sitemap.xmlLimit crawl scope
python3 scripts/sitemap_gen.py https://example.com --max-pages 500 --max-depth 3Polite crawling with delay
python3 scripts/sitemap_gen.py https://example.com --delay 1.0Set SEO hints
python3 scripts/sitemap_gen.py https://example.com --changefreq daily --priority 0.8Verbose progress
python3 scripts/sitemap_gen.py https://example.com -vPipe to stdout
python3 scripts/sitemap_gen.py https://example.com -o -
Options
| Flag | Default | Description |
|------|---------|-------------|
| --output, -o | sitemap.xml | Output file path (use - for stdout) |
| --max-pages | 200 | Maximum pages to crawl |
| --max-depth | 5 | Maximum link depth from start URL |
| --delay | 0.2 | Seconds between requests |
| --timeout | 10 | Request timeout in seconds |
| --changefreq | weekly | Sitemap changefreq hint |
| --priority | 0.5 | Sitemap priority hint (0.0β1.0) |
| --verbose, -v | off | Print crawl progress to stderr |
Dependencies
pip install requests beautifulsoup4
Notes
π‘ Examples
python3 scripts/sitemap_gen.py https://example.com
Output: sitemap.xml in the current directory.
βοΈ Configuration
| Flag | Default | Description |
|------|---------|-------------|
| --output, -o | sitemap.xml | Output file path (use - for stdout) |
| --max-pages | 200 | Maximum pages to crawl |
| --max-depth | 5 | Maximum link depth from start URL |
| --delay | 0.2 | Seconds between requests |
| --timeout | 10 | Request timeout in seconds |
| --changefreq | weekly | Sitemap changefreq hint |
| --priority | 0.5 | Sitemap priority hint (0.0β1.0) |
| --verbose, -v | off | Print crawl progress to stderr |