Mlops Experiment Tracker
by @charlie-morrison
Track and compare ML experiments across MLflow, Weights & Biases, or local logs. Organizes runs, compares metrics, identifies best models, detects training a...
clawhub install mlops-experiment-trackerπ About This Skill
name: cm-mlops-experiment-tracker description: Track and compare ML experiments across MLflow, Weights & Biases, or local logs. Organizes runs, compares metrics, identifies best models, detects training anomalies, and recommends hyperparameter configurations. Use when asked to review ML experiments, compare model runs, analyze training metrics, find best hyperparameters, audit experiment tracking setup, or troubleshoot training regressions. Triggers on "mlflow", "wandb", "weights and biases", "experiment tracking", "ml experiments", "model comparison", "hyperparameter", "training metrics", "model registry", "experiment runs". metadata: tags: ["mlops", "machine-learning", "experiment-tracking", "mlflow", "wandb", "hyperparameter", "model-comparison", "training", "metrics", "data-science"]
MLOps Experiment Tracker
Track, compare, and analyze ML experiments across any tracking backend. Reviews experiment organization, compares run metrics, identifies best-performing models, detects training anomalies, and recommends hyperparameter strategies. Acts as a senior ML engineer auditing your experiment workflow.
Usage
Basic: Analyze the ML experiments in /path/to/mlruns/
Focused: Compare hyperparameters across the top 5 runs | Find training runs with anomalous loss curves | Review model registry for promotion readiness
How It Works
Step 1: Discover Tracking Backend
# Detect MLflow, W&B, or local logs
find /path/to/project -name "mlruns" -type d
grep -r "import mlflow\|import wandb" /path/to/project --include="*.py"
find /path/to/project -name "metrics.json" -o -name "params.json"
Adapts to MLflow (file store, SQLite, PostgreSQL), W&B (local/cloud), local CSV/JSON logs, or TensorBoard event files.
Step 2: Parse Experiment Runs
Extracts structured data per run β parameters, metrics, artifacts, tags, duration, status:
Experiment: "fraud-detection-v2" (ID: 3) | 47 runs | 5 failed Run: run_abc123 | FINISHED | 2h 14m
Params: model=xgboost, lr=0.01, depth=8, estimators=500
Metrics: auc_roc=0.9423, f1=0.8832, precision=0.8712, recall=0.8956
Artifacts: model/, confusion_matrix.png, feature_importance.json
Tags: engineer=jsmith, dataset_version=v3.2, purpose=hyperparameter_sweep
Step 3: Compare Runs and Rank Models
Top 5 Runs by AUC-ROC:
Rank | Run ID | AUC-ROC | F1 | Duration
1 | run_abc123 | 0.9423 | 0.8832 | 2h 14m
2 | run_def456 | 0.9401 | 0.8801 | 2h 31m
3 | run_ghi789 | 0.9387 | 0.8790 | 1h 58m FINDING: Top 3 within 0.004 AUC β statistically equivalent
RECOMMEND: Choose run_abc123 β best accuracy at shortest training time
Pareto-Optimal Runs (accuracy vs cost):
run_ghi789: AUC=0.9387, cost=1h 58m <-- Best efficiency
run_abc123: AUC=0.9423, cost=2h 14m <-- Best accuracy
run_xyz999: AUC=0.9290, cost=0h 42m <-- Best budget option
Step 4: Analyze Hyperparameter Impact
Hyperparameter Importance:
learning_rate | -0.72 correlation | [0.001, 0.1] tested
max_depth | +0.58 correlation | [3, 12] tested
n_estimators | +0.45 correlation | [100, 1000] tested
subsample | +0.12 correlation | [0.6, 1.0] tested KEY: learning_rate dominates. Best range: [0.005, 0.02]
WARN: max_depth > 10 shows overfitting (train_loss << val_loss)
NEXT SWEEP: lr=[0.005, 0.02], depth=[7, 9], fix subsample=0.8
Use Bayesian optimization β 60% fewer runs for same coverage
Step 5: Detect Training Anomalies
FAIL: run_pqr678 β Severe overfitting
train_loss=0.012, val_loss=0.389 (32x gap). Diverged at epoch 18.
FIX: Add regularization or early stopping (patience=10) FAIL: run_stu901 β Training diverged
Loss exploded epoch 12: 0.45 -> 847.32. Cause: lr=0.1 too high.
FIX: Reduce lr by 10x, add gradient clipping
WARN: run_vwx234 β Loss plateau for 20 epochs
Wasted ~1h 20m compute. FIX: early stopping (min_delta=0.001)
WARN: 5 runs show oscillation β batch_size=16 likely too small
Step 6: Audit Experiment Organization
FAIL: 12 runs have no tags β cannot determine purpose or engineer
FAIL: Inconsistent param naming: "lr" (15), "learning_rate" (28), "LR" (4)
FAIL: No git commit linked to 31/47 runs β cannot reproduce
WARN: 3 experiments named "test", "debug", "experiment1"
WARN: 18 runs missing artifact logging (no saved model/config)
Step 7: Model Registry Readiness
Candidate: run_abc123 (AUC-ROC: 0.9423)
[x] Model artifact saved [x] Hyperparameters logged
[x] Data version recorded [x] Holdout metrics present
[ ] No model signature β input/output schema unknown
[ ] No pip requirements β dependency versions unknown
[ ] No edge case testing [ ] No fairness/bias metrics Verdict: NOT READY β fix signature and dependency pinning first
Step 8: Final Report
# ML Experiment Analysis ReportExperiment Health Score: 62/100
Run organization: 5/10 Hyperparameter search: 7/10
Training quality: 6/10 Reproducibility: 4/10
Model registry: 5/10 Resource efficiency: 6/10Critical Actions
1. Standardize parameter naming across training scripts
2. Enable early stopping β save ~80h compute on future sweeps
3. Add git commit tracking for reproducibility
4. Log model signatures before production promotion
5. Narrow hyperparameter search based on importance analysis
Output
Tips for Best Results
π‘ Examples
Basic: Analyze the ML experiments in /path/to/mlruns/
Focused: Compare hyperparameters across the top 5 runs | Find training runs with anomalous loss curves | Review model registry for promotion readiness