Add Directories

---

name: add-directories description: Use when adding new AI/startup directories to directories.json from a URL, pasted text, or GitHub awesome-list. Parses, deduplicates, appends new entries, then runs classify pipeline (analyze, cleanup, browser verify). Does NOT cover submission — use submit-directories skill for that.

---

Add Directories

Workflow

1. Parse the input (URL or pasted text) into a list of directories 2. Deduplicate against existing entries in directories.json 3. Append new entries with required fields 4. Classify by running the analysis and verification pipeline 5. Discover forms for submission targets 6. Submit via automation or manual browser interaction

Step 1: Parse Input

From URL

Fetch the page and extract directory entries. Look for patterns like:

Name + URL pairs in lists, tables, or cards

Structured data (JSON-LD, markdown tables, CSV)

Repeated DOM patterns with links

From GitHub Topics/Repos

Use gh CLI to explore curated lists:

gh repo clone / to clone awesome-lists

Parse README.md for directory links (markdown link format)

Check for JSON/YAML data files with directory entries

Can also create PRs to add your product to these lists

From Pasted Text

Parse lines/rows. Common formats:

Name - https://url.com or Name | https://url.com

Markdown links: Name

Markdown tables with Name and URL columns

Plain URLs (one per line) — derive name from domain

CSV/TSV with headers

Extract at minimum: name and url (submission or homepage).

Step 2: Deduplicate

Load directories.json and check each parsed entry against existing ones by:

Exact URL match (normalize: strip trailing slash, lowercase domain)

Domain match (same domain = likely duplicate)

Name match (case-insensitive)

Report duplicates to the user and skip them.

Step 3: Append New Entries

For each new directory, create an entry with this structure:

{
  "categories": ["General"],
  "description": "",
  "is_active": true,
  "name": "Directory Name",
  "pricing_type": "free",
  "slug": "directory-name",
  "submission_url": "https://example.com/submit",
  "url": "https://example.com/submit"
}

Field rules:

slug: lowercase name, spaces to hyphens, strip special chars

submission_url and url: use the submission/signup URL if available, otherwise homepage

description: leave empty string (will be filled later or by user)

categories: default ["General"] unless context provides a category

pricing_type: default "free" unless explicitly marked paid

is_active: always true for new entries

Save the updated directories.json.

Step 4: Classify

Run the pipeline scripts in order using the project venv at .venv/:

# 1. HTTP-level analysis (auth, captcha, pricing signals, dead domains)
.venv/bin/python analyze_directories.py
2. Cleanup obvious failures + build browser check list
.venv/bin/python cleanup_and_categorize.py
3. Browser verification with Playwright (10 concurrent workers)
.venv/bin/python browser_verify.py
4. Deep recheck any remaining unknowns
.venv/bin/python browser_verify.py --recheck-unknown

Each script reads/writes directories.json. Steps 3-4 use browser_check_list.json as intermediate state (generated by step 2).

After completion, report the summary: how many added, and the auth/status breakdown for the new entries.

Step 5: Discover Forms

For directories that are active and have auth_type = none or auth_type = email_password:

# Discover form fields on submission pages
.venv/bin/python discover_forms.py

This visits each submission URL with Playwright, extracts form fields via DOM queries, and updates submission_plan.json with discovered fields and form paths.

Step 6: Submit

Automated Submission

Configure the PRODUCT dict in submit_directories.py with your details (search for YOUR_ placeholders), then:

# Auto-submit to all discovered directories
.venv/bin/python submit_directories.py

The script uses heuristic field mapping (matching field names/labels to product data) and handles file uploads for logo/screenshot.

Manual Browser Submission (via Playwright MCP)

For directories that need manual interaction (captcha, OAuth, complex forms), use the Playwright browser tools:

1. Navigate to the submission URL 2. Take a snapshot to understand the page structure 3. Fill form fields using browser_fill_form or browser_type 4. Handle OAuth flows by switching tabs when Google login popups open 5. Upload files via browser_file_upload 6. Click submit and verify confirmation

GitHub PR Submissions

Some directories accept submissions via GitHub PRs to awesome-lists:

1. Fork the repo: gh repo fork / 2. Clone and create a branch 3. Add your product entry following the repo's format 4. Push and create PR: gh pr create

Notes

Pipeline Scripts

analyze_directories.py uses ThreadPoolExecutor with plain HTTP — fast first pass

cleanup_and_categorize.py triages errors (dead domains, invalid URLs, Facebook groups) and builds browser_check_list.json

browser_verify.py uses async Playwright with 10 concurrent tabs; --recheck-unknown does a deep DOM pass on active unknowns only

discover_forms.py uses async Playwright with 10 concurrent tabs; extracts form field names, types, labels, and paths

submit_directories.py uses async Playwright with 5 concurrent tabs; heuristic field mapping with file upload support

All scripts are idempotent — safe to re-run

Common Submission Blockers

When evaluating or submitting to directories, watch for these issues:

| Blocker | Frequency | How to Detect | |---|---|---| | Paid listing required | ~20% | Look for pricing page, Stripe/PayPal links, "$" on submit page | | reCAPTCHA / Turnstile | ~10% | iframe[src*=recaptcha] or [data-turnstile] elements | | Broken captcha | ~2% | "Invalid site key" errors, disabled submit buttons | | Login/account required | ~15% | Redirect to /login or /register on submit URL | | Business email required | ~3% | Rejects gmail/yahoo domains (e.g., SoftwareSuggest) | | Reciprocal link required | ~5% | Old web directories require backlink before listing | | Newsletter-only forms | ~10% | Page looks like submit but is actually email signup | | Backend API broken | ~2% | Form submits but returns GraphQL/API errors | | Domain parked/dead | ~8% | No content, parking page, DNS failure | | Cloudflare blocked | ~3% | Challenge page, 403 errors |

Automation Tips

Simple HTML forms have highest auto-submit success rate

reCAPTCHA v3 (invisible) sometimes passes; v2 (checkbox) never does automatically

Google Forms are reliably automatable

Rich text editors (TinyMCE, Quill) need browser_evaluate to set content

Cloudinary/custom upload widgets often break automation — use manual browser

Cross-origin OAuth popups: Switch tabs with browser_tabs action to handle Google login

Combobox/select fields: Use browser_click on the dropdown, then click the option

Multi-step forms: Take snapshot after each step to see new fields

Submission Plan Structure

Each entry in submission_plan.json contains:

{
  "directory_name": "Example AI",
  "submission_url": "https://example.com/submit",
  "status": "discovered",
  "copy": {
    "title": "Product Title Variation",
    "description": "Product description variation for this directory."
  },
  "discovered_fields": [...],
  "form_path": "form#submit-form",
  "credentials": {
    "email": "YOUR_EMAIL",
    "name": "YOUR_NAME",
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD"
  }
}

Status values: discovered, submitted, skipped, skipped_paid, timeout, no_form_found, no_fields_matched, submit_timeout, captcha, cloudflare_blocked, domain_parked, skipped_login_required, deferred.

Best ROI Directory Types (for AI/SaaS products)

1. AI tool directories with simple forms (FutureTools, SaaSHub, AItools.inc, etc.) 2. Startup directories with Google Form submissions 3. GitHub awesome-lists accepting PRs (free, high-quality backlinks) 4. NoCode/SaaS aggregators (NoCodeList, NoCodeDevs) 5. General web directories with DA≥30 (for SEO value)

Security Note

Before pushing to GitHub, ensure all personal data is stripped:

Search for YOUR_ placeholders in submission_plan.json and submit_directories.py

Never commit real emails, passwords, or API keys

The .playwright-mcp/ folder may contain console logs with personal data — add to .gitignore

Pipeline Scripts

analyze_directories.py uses ThreadPoolExecutor with plain HTTP — fast first pass

cleanup_and_categorize.py triages errors (dead domains, invalid URLs, Facebook groups) and builds browser_check_list.json

browser_verify.py uses async Playwright with 10 concurrent tabs; --recheck-unknown does a deep DOM pass on active unknowns only

discover_forms.py uses async Playwright with 10 concurrent tabs; extracts form field names, types, labels, and paths

submit_directories.py uses async Playwright with 5 concurrent tabs; heuristic field mapping with file upload support

All scripts are idempotent — safe to re-run

Common Submission Blockers

When evaluating or submitting to directories, watch for these issues:

| Blocker | Frequency | How to Detect | |---|---|---| | Paid listing required | ~20% | Look for pricing page, Stripe/PayPal links, "$" on submit page | | reCAPTCHA / Turnstile | ~10% | iframe[src*=recaptcha] or [data-turnstile] elements | | Broken captcha | ~2% | "Invalid site key" errors, disabled submit buttons | | Login/account required | ~15% | Redirect to /login or /register on submit URL | | Business email required | ~3% | Rejects gmail/yahoo domains (e.g., SoftwareSuggest) | | Reciprocal link required | ~5% | Old web directories require backlink before listing | | Newsletter-only forms | ~10% | Page looks like submit but is actually email signup | | Backend API broken | ~2% | Form submits but returns GraphQL/API errors | | Domain parked/dead | ~8% | No content, parking page, DNS failure | | Cloudflare blocked | ~3% | Challenge page, 403 errors |

Automation Tips

Simple HTML forms have highest auto-submit success rate

reCAPTCHA v3 (invisible) sometimes passes; v2 (checkbox) never does automatically

Google Forms are reliably automatable

Rich text editors (TinyMCE, Quill) need browser_evaluate to set content

Cloudinary/custom upload widgets often break automation — use manual browser

Cross-origin OAuth popups: Switch tabs with browser_tabs action to handle Google login

Combobox/select fields: Use browser_click on the dropdown, then click the option

Multi-step forms: Take snapshot after each step to see new fields

Submission Plan Structure

Each entry in submission_plan.json contains:

{
  "directory_name": "Example AI",
  "submission_url": "https://example.com/submit",
  "status": "discovered",
  "copy": {
    "title": "Product Title Variation",
    "description": "Product description variation for this directory."
  },
  "discovered_fields": [...],
  "form_path": "form#submit-form",
  "credentials": {
    "email": "YOUR_EMAIL",
    "name": "YOUR_NAME",
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD"
  }
}

Status values: discovered, submitted, skipped, skipped_paid, timeout, no_form_found, no_fields_matched, submit_timeout, captcha, cloudflare_blocked, domain_parked, skipped_login_required, deferred.

Best ROI Directory Types (for AI/SaaS products)

1. AI tool directories with simple forms (FutureTools, SaaSHub, AItools.inc, etc.) 2. Startup directories with Google Form submissions 3. GitHub awesome-lists accepting PRs (free, high-quality backlinks) 4. NoCode/SaaS aggregators (NoCodeList, NoCodeDevs) 5. General web directories with DA≥30 (for SEO value)

Security Note

Before pushing to GitHub, ensure all personal data is stripped:

Search for YOUR_ placeholders in submission_plan.json and submit_directories.py

Never commit real emails, passwords, or API keys

The .playwright-mcp/ folder may contain console logs with personal data — add to .gitignore