Technical SEO Mastery

Technical SEO Mastery

> Complete technical SEO audit, fix, and monitoring system. From crawlability to Core Web Vitals to international SEO — everything search engines need to find, crawl, index, and rank your site.

When to Use

Phase 1: Quick Health Check (5-Minute Triage)

Before a full audit, run a rapid assessment. Fetch the site and check:

quick_health:
  url: "[domain]"
  checks:
    robots_txt: "[accessible / blocked / missing]"
    sitemap_xml: "[found / missing / errors]"
    https: "[yes / mixed content / no]"
    mobile_viewport: "[set / missing]"
    page_load: "[fast <2s / moderate 2-4s / slow >4s]"
    h1_present: "[yes / missing / multiple]"
    canonical: "[set / missing / self-referencing]"
    structured_data: "[present / missing / errors]"
  
  severity: "[healthy / needs work / critical]"
  priority_fix: "[top issue to address first]"

Severity guide:

Phase 2: Crawlability Audit

2.1 Robots.txt Analysis

Fetch [domain]/robots.txt and evaluate:

robots_txt_audit:
  exists: true/false
  valid_syntax: true/false
  issues:
    - type: "[blocked_important_page / missing_sitemap / wildcard_block / syntax_error]"
      detail: "[specific line or pattern]"
      severity: "critical/warning/info"
      fix: "[exact fix]"
  
  checks:
    - "Sitemap directive present"
    - "No accidental blocking of CSS/JS/images"
    - "No blocking of important page directories"
    - "Correct user-agent targeting (Googlebot, Bingbot, etc.)"
    - "No conflicting rules (allow + disallow same path)"
    - "Crawl-delay only if needed (slows indexing)"

Common mistakes: | Mistake | Impact | Fix | |---------|--------|-----| | Disallow: / blocking everything | No pages indexed | Remove or narrow scope | | Blocking CSS/JS | Poor rendering = ranking drop | Allow: /assets/ | | No sitemap reference | Slower discovery | Add Sitemap: directive | | Multiple sitemaps not declared | Partial crawling | Declare all sitemaps | | Blocking search/filter pages poorly | Crawl waste | Use Disallow: /*? patterns |

Recommended template:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

2.2 XML Sitemap Audit

Fetch sitemap(s) and evaluate:

sitemap_audit:
  url: "[sitemap URL]"
  type: "[index / single / none]"
  url_count: 0
  issues:
    - type: "[missing / too_large / includes_noindex / stale_lastmod / broken_urls]"
      severity: "critical/warning/info"
      fix: "[specific action]"
  
  quality_checks:
    - "All URLs return 200 (no 404s, 301s, or 5xx)"
    - "No noindex pages included"
    - "lastmod dates are accurate (not all same date)"
    - "Under 50,000 URLs per sitemap file"
    - "Under 50MB uncompressed per file"
    - "Sitemap index if >50K URLs"
    - "Matches canonical URLs (not alternate versions)"
    - "Images/video sitemaps if applicable"

Sitemap best practices:

2.3 Crawl Budget Optimization

crawl_budget_analysis:
  total_pages: 0
  indexable_pages: 0
  crawl_waste_ratio: "[indexable / total — target >80%]"
  
  waste_sources:
    - source: "[faceted navigation / pagination / parameters / duplicate content / thin pages]"
      page_count: 0
      action: "[noindex / canonical / robots block / parameter handling / consolidate]"
  
  optimization_priority:
    1: "Remove/noindex thin and duplicate pages"
    2: "Consolidate parameter variations with canonicals"
    3: "Implement pagination best practices (rel=next or load-more)"
    4: "Fix redirect chains (max 1 hop)"
    5: "Eliminate orphan pages or link them into site structure"

Crawl budget matters when: Site has >10K pages OR crawl rate is notably low in Search Console.

Phase 3: Indexability Audit

3.1 Index Status Check

For each important page type, verify:

indexability_check:
  page_type: "[homepage / product / blog / category / landing]"
  sample_url: "[URL]"
  
  signals:
    meta_robots: "[index,follow / noindex / nofollow / none]"
    x_robots_tag: "[present / absent — check HTTP headers]"
    canonical: "[self / points to other URL / missing]"
    http_status: "[200 / 301 / 302 / 404 / 410 / 5xx]"
    in_sitemap: true/false
    internal_links_to: "[count of internal links pointing here]"
    robots_txt_allowed: true/false
  
  verdict: "[indexable / blocked / conflicting signals]"
  fix: "[action if not indexable]"

3.2 Common Indexing Blockers

| Blocker | Detection | Fix | Priority | |---------|-----------|-----|----------| | noindex meta tag | Check | Remove tag or move to correct pages | P0 | | X-Robots-Tag: noindex header | Check HTTP response headers | Remove header from server config | P0 | | Canonical pointing elsewhere | Check | Fix to self-reference or correct target | P0 | | Blocked in robots.txt | Cross-reference robots.txt | Update robots.txt rules | P0 | | Not in sitemap | Check sitemap inclusion | Add to sitemap | P1 | | No internal links (orphan) | Crawl internal link graph | Add contextual internal links | P1 | | Soft 404 (200 with no content) | Check page content | Return proper 404 or add content | P1 | | Duplicate content | Compare page similarity | Canonical or consolidate | P2 | | Thin content (<200 words, no value) | Word count + quality check | Expand or merge with related page | P2 |

3.3 Redirect Audit

redirect_audit:
  chains_found: 0  # A→B→C (should be A→C)
  loops_found: 0   # A→B→A (broken)
  temporary_redirects: 0  # 302s that should be 301s
  
  rules:
    - "Max 1 redirect hop (no chains)"
    - "Use 301 for permanent moves, 308 for POST-preserving"
    - "302 only for genuinely temporary redirects"
    - "Update internal links to point to final destination"
    - "Redirect HTTP → HTTPS at server level"
    - "Redirect www ↔ non-www consistently"

Phase 4: Core Web Vitals & Performance

4.1 Core Web Vitals Targets

| Metric | Good | Needs Improvement | Poor | |--------|------|-------------------|------| | LCP (Largest Contentful Paint) | ≤2.5s | 2.5-4.0s | >4.0s | | INP (Interaction to Next Paint) | ≤200ms | 200-500ms | >500ms | | CLS (Cumulative Layout Shift) | ≤0.1 | 0.1-0.25 | >0.25 |

4.2 LCP Fix Playbook

lcp_diagnosis:
  current_lcp: "[Xs]"
  lcp_element: "[identify the largest element — usually hero image or heading]"
  
  fix_priority:
    1:
      issue: "Slow server response (TTFB >800ms)"
      fixes:
        - "Enable server-side caching (Redis, Varnish)"
        - "Use CDN for static assets"
        - "Optimize database queries"
        - "Upgrade hosting if shared"
    2:
      issue: "Render-blocking resources"
      fixes:
        - "Defer non-critical CSS: "
        - "Async/defer JavaScript: