Production Readiness

---

name: production-readiness model: reasoning description: Meta-skill that orchestrates logging, monitoring, error handling, performance, security, deployment, and testing skills to ensure a service is fully production-ready before launch. Use before first deploy, major releases, quarterly reviews, or after incidents.

---

Production Readiness (Meta-Skill)

Coordinates all operational concerns into a single readiness review. Instead of duplicating domain expertise, this skill routes to specialized skills and agents for each area, then synthesizes results into a unified go/no-go assessment.

Installation

OpenClaw / Moltbot / Clawbot

npx clawhub@latest install production-readiness

---

Purpose

Ensure a service is production-ready by systematically checking every operational concern — logging, error handling, performance, security, deployment, testing, and documentation — before traffic hits it.

A production-ready service:

Fails gracefully under load and partial outages

Observes itself with structured logs, metrics, and traces

Recovers automatically from transient failures

Communicates health to orchestrators and operators

Documents operations so on-call engineers can respond without tribal knowledge

---

When to Use

| Trigger | Context | |---------|---------| | Before first deploy | New service going to production for the first time | | Before major release | Significant feature or architectural change shipping | | Quarterly production review | Scheduled audit of existing services | | After incident | Post-incident hardening to prevent recurrence | | Dependency upgrade | Major framework, runtime, or infrastructure change | | Team handoff | Transferring ownership of a service to another team |

---

Orchestration Flow

Run each area sequentially or in parallel. Each step delegates to a specialized skill or agent — this skill does not re-implement their logic.

┌─────────────────────────────────────────────────┐
│              Production Readiness Review         │
├─────────────────────────────────────────────────┤
│                                                  │
│  1. Logging & Observability ──► logging-observability skill
│  2. Error Handling ───────────► error-handling-patterns skill
│  3. Performance ──────────────► performance-agent
│  4. Security ─────────────────► security-review meta-skill
│  5. Deployment ───────────────► deployment-agent + docker-expert skill
│  6. Testing ──────────────────► testing-workflow meta-skill
│  7. Documentation ────────────► /generate-docs command
│                                                  │
│  ──► Synthesize results into go/no-go report     │
└─────────────────────────────────────────────────┘

Step Details

1. Logging & Observability — Structured logging, log levels, correlation IDs, metrics endpoints, distributed tracing, alerting rules 2. Error Handling — Global error boundaries, retry policies, dead-letter queues, error classification, user-facing error messages 3. Performance — Load testing results, P95/P99 latency baselines, memory/CPU profiling, database query analysis, caching strategy 4. Security — Auth/authz verification, input validation, dependency audit, secrets management, OWASP top-10 review 5. Deployment — Container hardening, rollback strategy, blue-green/canary configuration, infrastructure-as-code review 6. Testing — Unit/integration/e2e coverage, contract tests, chaos/failure injection, smoke tests in staging 7. Documentation — API docs, runbooks, architecture diagrams, on-call playbooks, ADRs for key decisions

---

Skill Routing Table

| Concern | Skill / Agent | Path | |---------|--------------|------| | Logging & Observability | logging-observability | ai/skills/tools/logging-observability/SKILL.md | | Error Handling | error-handling-patterns | ai/skills/backend/error-handling-patterns/SKILL.md | | Performance | performance-agent | ai/agents/performance/ | | Security | security-review | ai/skills/meta/security-review/SKILL.md | | Deployment (containers) | docker-expert | ai/skills/devops/docker/SKILL.md | | Deployment (pipelines) | deployment-agent | ai/agents/deployment/ | | Testing | testing-workflow | ai/skills/testing/testing-workflow/SKILL.md | | Rate Limiting | rate-limiting-patterns | ai/skills/backend/rate-limiting-patterns/SKILL.md | | Documentation | /generate-docs | ai/commands/documentation/ |

> Routing rule: Read the target skill first, follow its instructions, then return results here for synthesis.

---

Production Readiness Checklist

Health & Lifecycle

[ ] Health check endpoint (/healthz or /health) returns dependency status

[ ] Readiness probe distinguishes "starting" from "ready to serve"

[ ] Liveness probe detects deadlocks and unrecoverable states

[ ] Graceful shutdown drains in-flight requests before exit

[ ] Startup probe allows for slow initialization without false restarts

Resilience

[ ] Circuit breakers on all external service calls

[ ] Retry with exponential backoff and jitter on transient failures

[ ] Rate limiting configured per endpoint and per client

[ ] Backpressure mechanisms prevent cascade failures under load

[ ] Timeouts set on every outbound call (HTTP, DB, queue)

[ ] Bulkhead isolation separates critical from non-critical paths

Configuration & Secrets

[ ] All configuration externalized (env vars, config service, or feature flags)

[ ] No secrets in code, images, or environment variable defaults

[ ] Secrets loaded from a vault (e.g., AWS Secrets Manager, HashiCorp Vault)

[ ] Configuration changes do not require redeployment

[ ] Feature flags in place for high-risk changes

Data Safety

[ ] Backup strategy defined and tested (RPO/RTO documented)

[ ] Restore procedure verified in a non-production environment

[ ] Database migrations are backward-compatible and reversible

[ ] Data retention policies implemented and enforced

Operational Readiness

[ ] Runbooks exist for top 5 most likely failure scenarios

[ ] SLOs defined (availability, latency, error rate) with error budgets

[ ] SLAs communicated to dependent teams or customers

[ ] On-call rotation staffed and escalation path documented

[ ] Dashboards show golden signals (latency, traffic, errors, saturation)

[ ] Alerting rules configured with appropriate thresholds and severity

---

Maturity Levels

| Level | Name | Requirements | |-------|------|-------------| | L1 | MVP | Health check, basic logging, error handling, manual deploy, unit tests, README | | L2 | Stable | Structured logging, metrics, graceful shutdown, CI/CD pipeline, integration tests, runbooks | | L3 | Resilient | Distributed tracing, circuit breakers, auto-scaling, chaos testing, SLOs, on-call rotation | | L4 | Optimized | Adaptive rate limiting, predictive alerting, canary deploys, full observability, error budgets, postmortem culture |

Progression Guidance

L1 → L2: Add structured logging, metrics endpoint, and a CI/CD pipeline. Write runbooks for known failure modes.

L2 → L3: Instrument distributed tracing. Add circuit breakers to external calls. Define SLOs and set up on-call.

L3 → L4: Implement canary deployments. Adopt error budgets. Run regular game days. Build predictive alerting.

---

Incident Response

On-Call Rotation

Minimum two engineers per rotation (primary + secondary)

Handoff includes review of recent deploys, open issues, and known risks

Escalation targets defined: primary → secondary → engineering lead → VP Eng

Escalation Matrix

| Severity | Response Time | Escalation After | Stakeholder Notification | |----------|--------------|-------------------|--------------------------| | SEV-1 (outage) | 15 min | 30 min | Immediate — exec + customers | | SEV-2 (degraded) | 30 min | 1 hour | Within 1 hour — eng lead | | SEV-3 (minor) | 4 hours | Next business day | Daily standup | | SEV-4 (cosmetic) | Next sprint | N/A | Backlog |

Postmortem Template

## Incident: [Title]
Date: YYYY-MM-DD | Duration: X hours | Severity: SEV-N
Summary
One-paragraph description of what happened and impact.
Timeline
HH:MM — First alert fired
HH:MM — Engineer paged, investigation started
HH:MM — Root cause identified
HH:MM — Mitigation applied
HH:MM — Full resolution confirmed
Root Cause
What broke and why. Link to code/config change if applicable.
Impact
Users affected: N
Revenue impact: $X (if applicable)
SLO budget consumed: X%
Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Fix X  | @eng  | YYYY-MM-DD | Open |
Lessons Learned
What went well
What went poorly
Where we got lucky

---

NEVER Do

1. NEVER skip health checks — every service must expose health endpoints; no exceptions for "simple" services 2. NEVER store secrets in code or container images — use a secrets manager; never default env vars with real values 3. NEVER deploy without a rollback plan — if you cannot roll back in under 5 minutes, you are not ready to deploy 4. NEVER ignore error budget violations — when the error budget is exhausted, freeze feature work and fix reliability 5. NEVER treat logging as optional — a service without structured logging is a service you cannot debug at 3 AM 6. NEVER go to production without runbooks — if on-call cannot resolve the top 5 failure modes without the original author, the service is not production-ready