ML Engineering

ML & AI Engineering System

Complete methodology for building, deploying, and operating production ML/AI systems — from experiment to scale.

---

Phase 1: Problem Framing

Before writing any code, define the ML problem precisely.

ML Problem Brief

problem_brief:
  business_objective: ""          # What business metric improves?
  success_metric: ""              # Quantified target (e.g., "reduce churn 15%")
  baseline: ""                    # Current performance without ML
  ml_task_type: ""                # classification | regression | ranking | generation | clustering | anomaly_detection | recommendation
  prediction_target: ""           # What exactly are we predicting?
  prediction_consumer: ""         # Who/what uses the prediction? (API | dashboard | email | automated action)
  latency_requirement: ""         # real-time (<100ms) | near-real-time (<1s) | batch (minutes-hours)
  data_available: ""              # What data exists today?
  data_gaps: ""                   # What's missing?
  ethical_considerations: ""      # Bias risks, fairness requirements, privacy
  kill_criteria:                  # When to abandon the ML approach
    - "Baseline heuristic achieves >90% of ML performance"
    - "Data quality too poor after 2 weeks of cleaning"
    - "Model can't beat random by >10% on holdout set"

ML vs Rules Decision

| Signal | Use Rules | Use ML | |--------|-----------|--------| | Logic is explainable in <10 rules | ✅ | ❌ | | Pattern is too complex for humans | ❌ | ✅ | | Training data >1,000 labeled examples | — | ✅ | | Needs to adapt to new patterns | ❌ | ✅ | | Must be 100% auditable/deterministic | ✅ | ❌ | | Pattern changes faster than you can update rules | ❌ | ✅ |

Rule of thumb: Start with rules/heuristics. Only add ML when rules fail to capture the pattern.

---

Phase 2: Data Engineering for ML

Data Quality Assessment

Score each data source (0-5 per dimension):

| Dimension | 0 (Terrible) | 5 (Excellent) | |-----------|--------------|----------------| | Completeness | >50% missing | <1% missing | | Accuracy | Known errors, no validation | Validated against source of truth | | Consistency | Different formats, duplicates | Standardized, deduplicated | | Timeliness | Months stale | Real-time or daily refresh | | Relevance | Weak proxy for target | Direct signal for prediction | | Volume | <100 samples | >10,000 samples per class |

Minimum score to proceed: 18/30. Below 18 → fix data first, don't build models.

Feature Engineering Patterns

feature_types:
  numerical:
    - raw_value           # Use as-is if normally distributed
    - log_transform       # Right-skewed distributions (revenue, counts)
    - standardize         # z-score for algorithms sensitive to scale (SVM, KNN, neural nets)
    - bin_to_categorical  # When relationship is non-linear and data is limited
  categorical:
    - one_hot             # <20 categories, tree-based models handle natively
    - target_encoding     # High-cardinality (>20 categories), use with K-fold to prevent leakage
    - embedding           # Very high-cardinality (user IDs, product IDs) with deep learning
  temporal:
    - lag_features        # Value at t-1, t-7, t-30
    - rolling_statistics  # Mean, std, min, max over windows
    - time_since_event    # Days since last purchase, hours since login
    - cyclical_encoding   # sin/cos for hour-of-day, day-of-week, month
  text:
    - tfidf               # Simple, interpretable, good baseline
    - sentence_embeddings # semantic similarity, modern NLP
    - llm_extraction      # Use LLM to extract structured fields from unstructured text
  interaction:
    - ratios              # Feature A / Feature B (e.g., clicks/impressions = CTR)
    - differences         # Feature A - Feature B (e.g., price - competitor_price)
    - polynomial          # A * B, A^2 (use sparingly, high-cardinality features)

Feature Store Design

feature_store:
  offline_store:         # For training — batch computed, stored in data warehouse
    storage: "BigQuery | Snowflake | S3+Parquet"
    compute: "Spark | dbt | SQL"
    refresh: "daily | hourly"
  online_store:          # For serving — low-latency lookups
    storage: "Redis | DynamoDB | Feast online"
    latency_target: "<10ms p99"
    refresh: "streaming | near-real-time"
  registry:              # Feature metadata
    naming: "{entity}_{feature_name}_{window}_{aggregation}"  # e.g., user_purchase_count_30d_sum
    documentation:
      - description
      - data_type
      - source_table
      - owner
      - created_date
      - known_issues

Data Leakage Prevention Checklist

[ ] No future information in features (time-travel check)

[ ] Train/val/test split done BEFORE feature engineering

[ ] Target encoding uses only training fold statistics

[ ] No features derived from the target variable

[ ] Temporal splits for time-series (no random shuffle)

[ ] Holdout set created BEFORE any EDA

[ ] Duplicates removed BEFORE splitting (same entity not in train AND test)

[ ] Normalization/scaling fit on train, applied to val/test

---

Phase 3: Experiment Management

Experiment Tracking Template

experiment:
  id: "EXP-{YYYY-MM-DD}-{NNN}"
  hypothesis: ""                 # "Adding user tenure features will improve churn prediction AUC by >2%"
  dataset_version: ""            # Hash or version of training data
  features_used: []              # List of feature names
  model_type: ""                 # Algorithm name
  hyperparameters: {}            # All hyperparams logged
  training_time: ""              # Wall clock
  metrics:
    primary: {}                  # The one metric that matters
    secondary: {}                # Supporting metrics
  baseline_comparison: ""        # Delta vs baseline
  verdict: "promoted | archived | iterate"
  notes: ""
  artifacts:
    - model_path: ""
    - notebook_path: ""
    - confusion_matrix: ""

Model Selection Guide

| Task | Start With | Scale To | Avoid | |------|-----------|----------|-------| | Tabular classification | XGBoost/LightGBM | Neural nets only if >100K samples | Deep learning on <10K samples | | Tabular regression | XGBoost/LightGBM | CatBoost for high-cardinality cats | Linear regression without feature engineering | | Image classification | Fine-tune ResNet/EfficientNet | Vision Transformer if >100K images | Training from scratch | | Text classification | Fine-tune BERT/RoBERTa | LLM few-shot if labeled data scarce | Bag-of-words for nuanced tasks | | Text generation | GPT-4/Claude API | Fine-tuned Llama/Mistral for cost | Training from scratch | | Time series | Prophet/ARIMA baseline → LightGBM | Temporal Fusion Transformer | LSTM without strong reason | | Recommendation | Collaborative filtering baseline | Two-tower neural | Complex models on <1K users | | Anomaly detection | Isolation Forest | Autoencoder if high-dimensional | Supervised methods without labeled anomalies | | Search/ranking | BM25 baseline → Learning to Rank | Cross-encoder reranking | Pure keyword without semantic |

Hyperparameter Tuning Strategy

1. Manual first — understand 3-5 most impactful parameters 2. Bayesian optimization (Optuna) — 50-100 trials for production models 3. Grid search — only for final fine-tuning of 2-3 parameters 4. Random search — better than grid for >4 parameters

Key hyperparameters by model:

| Model | Critical Params | Typical Range | |-------|----------------|---------------| | XGBoost | learning_rate, max_depth, n_estimators, min_child_weight | 0.01-0.3, 3-10, 100-1000, 1-10 | | LightGBM | learning_rate, num_leaves, feature_fraction, min_data_in_leaf | 0.01-0.3, 15-255, 0.5-1.0, 5-100 | | Neural Net | learning_rate, batch_size, hidden_dims, dropout | 1e-5 to 1e-2, 32-512, arch-dependent, 0.1-0.5 | | Random Forest | n_estimators, max_depth, min_samples_leaf | 100-1000, 5-30, 1-20 |

---

Phase 4: Model Evaluation

Metric Selection by Task

| Task | Primary Metric | When to Use | Watch Out For | |------|---------------|-------------|---------------| | Binary classification (balanced) | F1-score | Equal importance of precision/recall | — | | Binary classification (imbalanced) | PR-AUC | Rare positive class (<5%) | ROC-AUC hides poor performance on minority | | Multi-class | Macro F1 | All classes equally important | Micro F1 if class frequency = importance | | Regression | MAE | Outliers should not dominate | RMSE penalizes large errors more | | Ranking | NDCG@K | Top-K results matter most | MAP if binary relevance | | Generation | Human eval + automated | Quality is subjective | BLEU/ROUGE alone are insufficient | | Anomaly detection | Precision@K | False positives are expensive | Recall if missing anomalies is dangerous |

Evaluation Rigor Checklist

[ ] Metrics computed on TRUE holdout (never seen during training OR tuning)

[ ] Cross-validation for small datasets (<10K samples)

[ ] Stratified splits for imbalanced classes

[ ] Temporal split for time-dependent data

[ ] Confidence intervals reported (bootstrap or cross-val)

[ ] Performance broken down by important segments (geography, user cohort, etc.)

[ ] Fairness metrics across protected groups

[ ] Comparison against simple baseline (majority class, mean prediction, rules)

[ ] Error analysis: examined top 50 worst predictions manually

[ ] Calibration plot for probabilistic predictions

Offline-to-Online Gap Analysis

Before deploying, verify these don't cause train-serving skew:

| Check | Offline | Online | Action | |-------|---------|--------|--------| | Feature computation | Batch SQL | Real-time API | Verify same logic, test with replay | | Data freshness | Point-in-time snapshot | Latest value | Document acceptable staleness | | Missing values | Imputed in pipeline | May be truly missing | Handle gracefully in serving | | Feature distributions | Training period | Current period | Monitor drift post-deploy |

---

Phase 5: Model Deployment

Deployment Pattern Decision Tree

Is latency < 100ms required?
├── Yes → Is model < 500MB?
│   ├── Yes → Embedded serving (FastAPI + model in memory)
│   └── No → Model server (Triton, TorchServe, vLLM)
└── No → Is it a batch prediction?
    ├── Yes → Batch pipeline (Spark, Airflow + offline inference)
    └── No → Async queue (Celery/SQS → worker → result store)

Production Serving Checklist

serving_config:
  model:
    format: ""                    # ONNX | TorchScript | SavedModel | safetensors
    version: ""                   # Semantic version
    size_mb: null
    load_time_seconds: null
  infrastructure:
    compute: ""                   # CPU | GPU (T4/A10/A100/H100)
    instances: null               # Min/max for autoscaling
    autoscale_metric: ""          # RPS | latency_p99 | GPU_utilization
    autoscale_target: null
  api:
    endpoint: ""
    input_schema: {}              # Pydantic model or JSON schema
    output_schema: {}
    timeout_ms: null
    rate_limit: null
  reliability:
    health_check: "/health"
    readiness_check: "/ready"     # Model loaded and warm
    graceful_shutdown: true
    circuit_breaker: true
    fallback: ""                  # Rules-based fallback when model is down

Containerization Template

# Multi-stage build for minimal image
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY model/ ./model/
COPY src/ ./src/
Non-root user
RUN useradd -m appuser && chown -R appuser /app
USER appuser
Health check
HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["uvicorn", "src.serve:app", "--host", "0.0.0.0", "--port", "8080"]

A/B Testing for Models

ab_test:
  name: ""
  hypothesis: ""
  primary_metric: ""              # Business metric (revenue, engagement, etc.)
  guardrail_metrics: []           # Metrics that must NOT degrade
  traffic_split:
    control: 50                   # Current model
    treatment: 50                 # New model
  minimum_sample_size: null       # Power analysis: use statsmodels or online calculator
  minimum_runtime_days: null      # At least 1 full business cycle (7 days min)
  decision_criteria:
    ship: "Treatment > control by >X% with p<0.05 AND no guardrail regression"
    iterate: "Promising signal but not significant — extend test or refine model"
    kill: "No improvement after 2x minimum runtime OR guardrail breach"

---

Phase 6: LLM Engineering

LLM Application Architecture

┌─────────────────────────────────────────────┐
│              Application Layer               │
│  (Prompt templates, chains, output parsers)  │
├─────────────────────────────────────────────┤
│              Orchestration Layer              │
│  (Routing, fallback, retry, caching)         │
├─────────────────────────────────────────────┤
│              Model Layer                     │
│  (API calls, fine-tuned models, embeddings)  │
├─────────────────────────────────────────────┤
│              Data Layer                      │
│  (Vector store, context retrieval, memory)   │
└─────────────────────────────────────────────┘

Model Selection for LLM Tasks

| Task | Best Option | Cost-Effective Option | When to Fine-Tune | |------|------------|----------------------|-------------------| | General reasoning | Claude Opus / GPT-4o | Claude Sonnet / GPT-4o-mini | Never for general reasoning | | Classification | Fine-tuned small model | Few-shot with Sonnet | >1,000 labeled examples + high volume | | Extraction | Structured output API | Regex + LLM fallback | Consistent format needed at scale | | Summarization | Claude Sonnet | GPT-4o-mini | Domain-specific style needed | | Code generation | Claude Sonnet | Codestral / DeepSeek | Internal codebase conventions | | Embeddings | text-embedding-3-large | text-embedding-3-small | Domain-specific vocab (medical, legal) |

RAG System Architecture

rag_pipeline:
  ingestion:
    chunking:
      strategy: "semantic"         # semantic | fixed_size | recursive
      chunk_size: 512              # tokens (512-1024 for most use cases)
      overlap: 50                  # tokens overlap between chunks
      metadata_to_preserve:
        - source_document
        - page_number
        - section_heading
        - date_created
    embedding:
      model: "text-embedding-3-large"
      dimensions: 1536             # Or 256/512 with Matryoshka for cost savings
    vector_store: "Pinecone | Weaviate | pgvector | Qdrant"
  retrieval:
    strategy: "hybrid"             # dense | sparse | hybrid (recommended)
    top_k: 10                      # Retrieve more, then rerank
    reranking:
      model: "Cohere rerank | cross-encoder"
      top_n: 3                     # Final context chunks
    filters: []                    # Metadata filters (date range, source, etc.)
  generation:
    model: ""
    system_prompt: |
      Answer based ONLY on the provided context.
      If the context doesn't contain the answer, say "I don't have enough information."
      Cite sources using [Source: document_name, page X].
    temperature: 0.1               # Low for factual, higher for creative
    max_tokens: null

RAG Quality Checklist

[ ] Chunking preserves semantic meaning (not cutting mid-sentence)

[ ] Metadata enables filtering (dates, sources, categories)

[ ] Retrieval returns relevant chunks (test with 50+ queries manually)

[ ] Reranking improves precision (compare with/without)

[ ] System prompt prevents hallucination (tested with adversarial queries)

[ ] Sources are cited and verifiable

[ ] Handles "I don't know" gracefully

[ ] Latency acceptable (<3s for interactive, <30s for complex)

[ ] Cost per query tracked and within budget

LLM Cost Optimization

| Strategy | Savings | Trade-off | |----------|---------|-----------| | Prompt caching | 50-90% on repeated prefixes | Requires cache-friendly prompt design | | Model routing (small → large) | 40-70% | Slightly higher latency, need router logic | | Batch API | 50% | Hours of delay, batch-only workloads | | Shorter prompts | Linear with token reduction | May reduce quality | | Fine-tuned small model | 80-95% vs large model API | Training cost + maintenance | | Semantic caching | 50-80% for similar queries | May return stale/wrong cached result | | Output token limits | Proportional | May truncate useful information |

---

Phase 7: Model Monitoring

Monitoring Dashboard

monitoring:
  model_performance:
    metrics:
      - name: "primary_metric"         # Same as offline evaluation
        threshold: null                 # Alert if below
        window: "1h | 1d | 7d"
      - name: "prediction_distribution"
        alert: "KL divergence > 0.1 from training distribution"
    latency:
      p50_ms: null
      p95_ms: null
      p99_ms: null
      alert_threshold_ms: null
    throughput:
      requests_per_second: null
      error_rate_threshold: 0.01       # Alert if >1% errors
  data_drift:
    method: "PSI | KS-test | JS-divergence"
    features_to_monitor: []            # Top 10 most important features
    check_frequency: "hourly | daily"
    alert_threshold: null              # PSI > 0.2 = significant drift
  concept_drift:
    method: "performance_degradation"
    ground_truth_delay: ""             # How long until we get labels?
    proxy_metrics: []                  # Metrics available before ground truth
    retraining_trigger: ""             # When to retrain

Drift Response Playbook

| Drift Type | Detection | Severity | Response | |------------|-----------|----------|----------| | Feature drift (input distribution shifts) | PSI > 0.1 | Warning | Investigate cause, monitor performance | | Feature drift (PSI > 0.25) | PSI > 0.25 | Critical | Retrain on recent data within 24h | | Concept drift (relationship changes) | Performance drop >5% | Critical | Retrain with new labels, review features | | Label drift (target distribution changes) | Chi-square test | Warning | Verify label quality, check for data issues | | Prediction drift (output distribution shifts) | KL divergence | Warning | May indicate upstream data issue |

Automated Retraining Pipeline

retraining:
  triggers:
    - type: "scheduled"
      frequency: "weekly | monthly"
    - type: "performance"
      condition: "primary_metric < threshold for 24h"
    - type: "drift"
      condition: "PSI > 0.2 on any top-10 feature"
  pipeline:
    1_data_validation:
      - check_completeness
      - check_distribution_shift
      - check_label_quality
    2_training:
      - use_latest_N_months_data
      - same_hyperparameters_as_production   # Unless scheduled tuning
      - log_all_metrics
    3_evaluation:
      - compare_vs_production_model
      - must_beat_production_on_primary_metric
      - must_not_regress_on_guardrail_metrics
      - evaluate_on_golden_test_set
    4_deployment:
      - canary_deployment: 5%
      - monitor_for: "4h minimum"
      - auto_rollback_if: "error_rate > 2x baseline"
      - gradual_rollout: "5% → 25% → 50% → 100%"
    5_notification:
      - log_retraining_event
      - notify_team_on_failure
      - update_model_registry

---

Phase 8: MLOps Infrastructure

ML Platform Components

| Component | Purpose | Tools | |-----------|---------|-------| | Experiment tracking | Log runs, compare results | MLflow, W&B, Neptune | | Feature store | Centralized feature management | Feast, Tecton, Hopsworks | | Model registry | Version, stage, approve models | MLflow Registry, SageMaker | | Pipeline orchestration | DAG-based ML workflows | Airflow, Prefect, Dagster, Kubeflow | | Model serving | Low-latency inference | Triton, TorchServe, vLLM, BentoML | | Monitoring | Drift, performance, data quality | Evidently, Whylogs, Great Expectations | | Vector store | Embedding storage for RAG | Pinecone, Weaviate, pgvector, Qdrant | | GPU management | Training and inference compute | K8s + GPU operator, RunPod, Modal |

CI/CD for ML

ml_cicd:
  on_code_change:
    - lint_and_type_check
    - unit_tests (data transforms, feature logic)
    - integration_tests (pipeline end-to-end on sample data)
  on_data_change:
    - data_validation (Great Expectations / custom)
    - feature_pipeline_run
    - smoke_test_predictions
  on_model_change:
    - full_evaluation_suite
    - bias_and_fairness_check
    - performance_regression_test
    - model_size_and_latency_check
    - security_scan (model file, dependencies)
    - staging_deployment
    - integration_test_in_staging
    - approval_gate (manual for major versions)
    - canary_deployment

Model Registry Workflow

┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│  Development │ ───→ │   Staging    │ ───→ │  Production  │
│              │      │              │      │              │
│ - Experiment │      │ - Eval suite │      │ - Canary     │
│ - Log metrics│      │ - Load test  │      │ - Monitor    │
│ - Compare    │      │ - Approval   │      │ - Rollback   │
└──────────────┘      └──────────────┘      └──────────────┘

Promotion criteria:

Dev → Staging: Beats current production on offline metrics

Staging → Production: Passes load test + integration test + human approval

Auto-rollback: Error rate >2x OR latency >2x OR primary metric drops >5%

---

Phase 9: Responsible AI

Bias Detection Checklist

[ ] Training data represents all demographic groups proportionally

[ ] Performance metrics broken down by protected attributes

[ ] Equal opportunity: similar true positive rates across groups

[ ] Calibration: predicted probabilities match actual rates per group

[ ] No proxy features for protected attributes (ZIP code → race)

[ ] Fairness metric selected and threshold defined BEFORE training

[ ] Disparate impact ratio >0.8 (80% rule)

[ ] Edge cases tested: what happens with unusual inputs?

Model Card Template

model_card:
  model_name: ""
  version: ""
  date: ""
  owner: ""
  description: ""
  intended_use: ""
  out_of_scope_uses: ""
  training_data:
    source: ""
    size: ""
    date_range: ""
    known_biases: ""
  evaluation:
    metrics: {}
    datasets: []
    sliced_metrics: {}             # Performance by subgroup
  limitations: []
  ethical_considerations: []
  maintenance:
    retraining_schedule: ""
    monitoring: ""
    contact: ""

---

Phase 10: Cost & Performance Optimization

GPU Selection Guide

| Use Case | GPU | VRAM | Cost/hr (cloud) | Best For | |----------|-----|------|-----------------|----------| | Fine-tune 7B model | A10G | 24GB | ~$1 | LoRA/QLoRA fine-tuning | | Fine-tune 70B model | A100 80GB | 80GB | ~$4 | Full fine-tuning medium models | | Serve 7B model | T4 | 16GB | ~$0.50 | Inference at scale | | Serve 70B model | A100 40GB | 40GB | ~$2 | Large model inference | | Train from scratch | H100 | 80GB | ~$8 | Pre-training, large-scale training |

Inference Optimization Techniques

| Technique | Speedup | Quality Impact | Complexity | |-----------|---------|---------------|------------| | Quantization (INT8) | 2-3x | <1% degradation | Low | | Quantization (INT4/GPTQ) | 3-4x | 1-3% degradation | Medium | | Batching | 2-10x throughput | None | Low | | KV-cache optimization | 20-40% memory savings | None | Medium | | Speculative decoding | 2-3x for LLMs | None (mathematically exact) | High | | Model distillation | 5-10x smaller model | 2-5% degradation | High | | ONNX Runtime | 1.5-3x | None | Low | | TensorRT | 2-5x | <1% | Medium | | vLLM (PagedAttention) | 2-4x throughput for LLMs | None | Low |

Cost Tracking Template

ml_costs:
  training:
    compute_cost_per_run: null
    runs_per_month: null
    data_storage_monthly: null
    experiment_tracking: null
  inference:
    cost_per_1k_predictions: null
    daily_volume: null
    monthly_cost: null
    cost_per_query_breakdown:
      compute: null
      model_api_calls: null
      vector_db: null
      data_transfer: null
  optimization_targets:
    cost_per_prediction: null      # Target
    monthly_budget: null
    cost_reduction_goal: ""

---

Phase 11: ML System Quality Rubric

Score your ML system (0-100):

| Dimension | Weight | 0-2 (Poor) | 3-4 (Good) | 5 (Excellent) | |-----------|--------|-----------|------------|----------------| | Problem framing | 15% | No clear business metric | Defined success metric | Kill criteria + baseline + ROI estimate | | Data quality | 15% | Ad-hoc data, no validation | Automated quality checks | Feature store + lineage + versioning | | Experiment rigor | 15% | No tracking, one-off notebooks | MLflow/W&B tracking | Reproducible pipelines + proper evaluation | | Model performance | 15% | Barely beats baseline | Significant improvement | Calibrated, fair, robust to edge cases | | Deployment | 10% | Manual deployment | CI/CD for models | Canary + auto-rollback + A/B testing | | Monitoring | 15% | No monitoring | Basic metrics dashboard | Drift detection + auto-retraining + alerts | | Documentation | 5% | Nothing documented | Model card exists | Full model card + runbooks + decision log | | Cost efficiency | 10% | No cost tracking | Budget exists | Optimized inference + cost-per-prediction tracking |

Scoring:

80-100: Production-grade ML system

60-79: Good foundations, missing operational maturity

40-59: Prototype quality, not ready for production

<40: Science project, needs fundamental rework

---

Common Mistakes

| Mistake | Fix | |---------|-----| | Optimizing model before fixing data | Data quality > model complexity. Always. | | Using accuracy on imbalanced data | Use PR-AUC, F1, or domain-specific metric | | No baseline comparison | Always start with simple heuristic baseline | | Training on future data | Temporal splits for time-series, strict leakage checks | | Deploying without monitoring | No model in production without drift detection | | Fine-tuning when prompting works | Try few-shot prompting first — fine-tune only for scale/cost | | GPU for everything | CPU inference is often sufficient and 10x cheaper | | Ignoring calibration | If probabilities matter (risk scoring), calibrate | | One-time model deployment | ML is a continuous system — plan for retraining from day 1 | | Premature scaling | Prove value with batch predictions before building real-time serving |

---

Quick Commands

"Frame ML problem" → Phase 1 brief

"Assess data quality" → Phase 2 scoring

"Select model" → Phase 3 guide

"Evaluate model" → Phase 4 checklist

"Deploy model" → Phase 5 serving config

"Build RAG" → Phase 6 RAG architecture

"Set up monitoring" → Phase 7 dashboard

"Optimize costs" → Phase 10 tracking

"Score ML system" → Phase 11 rubric

"Detect drift" → Phase 7 playbook

"A/B test model" → Phase 5 template

"Create model card" → Phase 9 template