Prometheus

Prometheus

Production Prometheus setup covering scrape configuration, service discovery, recording rules, alert rules, and operational best practices for infrastructure and application monitoring.

When to Use

| Scenario | Example | |----------|---------| | Set up metrics collection | New service needs Prometheus scraping | | Configure service discovery | K8s pods, file-based, or static targets | | Create recording rules | Pre-compute expensive PromQL queries | | Design alert rules | SLO-based alerts for availability and latency | | Production deployment | HA setup with retention and storage planning | | Troubleshoot scraping | Targets down, metrics missing, relabeling issues |

Architecture

Applications ──(/metrics)──→ Prometheus Server ──→ AlertManager → Slack/PD
      ↑                           │
  client libraries          ├──→ Grafana (dashboards)
  (prom client)             └──→ Thanos/Cortex (long-term storage)

Installation

Kubernetes (Helm)

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageVolumeSize=50Gi

Core Configuration

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: production
    region: us-west-2
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
rule_files:
  - /etc/prometheus/rules/*.yml
scrape_configs:
  # Self-monitoring
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  # Node exporters
  - job_name: node-exporter
    static_configs:
      - targets: ["node1:9100", "node2:9100", "node3:9100"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"
  # Application metrics (TLS)
  - job_name: my-app
    scheme: https
    metrics_path: /metrics
    tls_config:
      ca_file: /etc/prometheus/ca.crt
    static_configs:
      - targets: ["app1:9090", "app2:9090"]

Service Discovery

Kubernetes Pods (Annotation-Based)

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels:
          [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Pod annotations to enable scraping:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"

File-Based Discovery

scrape_configs:
  - job_name: file-sd
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.json"]
        refresh_interval: 5m

targets/production.json:

[{
  "targets": ["app1:9090", "app2:9090"],
  "labels": { "env": "production", "service": "api" }
}]

Discovery Method Comparison

| Method | Best For | Dynamic | |--------|----------|---------| | static_configs | Fixed infrastructure, dev | No | | file_sd_configs | CM-managed inventories | Yes (file watch) | | kubernetes_sd_configs | K8s workloads | Yes (API watch) | | consul_sd_configs | Consul service mesh | Yes (Consul watch) | | ec2_sd_configs | AWS EC2 instances | Yes (API poll) |

Recording Rules

Pre-compute expensive queries for dashboard and alert performance:

# /etc/prometheus/rules/recording_rules.yml
groups:
  - name: api_metrics
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
      - record: job:http_error_rate:ratio
        expr: job:http_errors:rate5m / job:http_requests:rate5m
      - record: job:http_duration:p95
        expr: >
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )
  - name: resource_metrics
    interval: 30s
    rules:
      - record: instance:node_cpu:utilization
        expr: >
          100 - (avg by (instance)
            (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
      - record: instance:node_memory:utilization
        expr: >
          100 - ((node_memory_MemAvailable_bytes
            / node_memory_MemTotal_bytes) * 100)
      - record: instance:node_disk:utilization
        expr: >
          100 - ((node_filesystem_avail_bytes
            / node_filesystem_size_bytes) * 100)

Naming Convention

level:metric_name:operations

| Part | Example | Meaning | |------|---------|---------| | level | job:, instance: | Aggregation level | | metric_name | http_requests | Base metric | | operations | :rate5m, :ratio | Applied functions |

Alert Rules

# /etc/prometheus/rules/alert_rules.yml
groups:
  - name: availability
    rules:
      - alert: ServiceDown
        expr: up{job="my-app"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "{{ $labels.job }} down for >1 minute"
      - alert: HighErrorRate
        expr: job:http_error_rate:ratio > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error rate {{ $value | humanizePercentage }} for {{ $labels.job }}"
      - alert: HighP95Latency
        expr: job:http_duration:p95 > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency {{ $value }}s for {{ $labels.job }}"
  - name: resources
    rules:
      - alert: HighCPU
        expr: instance:node_cpu:utilization > 80
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "CPU {{ $value }}% on {{ $labels.instance }}"
      - alert: HighMemory
        expr: instance:node_memory:utilization > 85
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "Memory {{ $value }}% on {{ $labels.instance }}"
      - alert: DiskSpaceLow
        expr: instance:node_disk:utilization > 90
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Disk {{ $value }}% on {{ $labels.instance }}"

Alert Severity Guide

| Severity | Threshold | Response | |----------|-----------|----------| | critical | Service down, data loss risk | Page on-call immediately | | warning | Degraded, approaching limit | Investigate within hours | | info | Notable but not urgent | Review in next business day |

Validation

# Validate config syntax
promtool check config prometheus.yml
Validate rule files
promtool check rules /etc/prometheus/rules/*.yml
Test a query
promtool query instant http://localhost:9090 'up'
Reload config without restart
curl -X POST http://localhost:9090/-/reload

Best Practices

| Practice | Detail | |----------|--------| | Naming: prefix_name_unit | Snake_case, _total for counters, _seconds/_bytes for units | | Scrape intervals 15–60s | Shorter wastes resources and storage | | Recording rules for dashboards | Pre-compute anything queried repeatedly | | Monitor Prometheus itself | prometheus_tsdb_*, scrape_duration_seconds | | HA deployment | 2+ instances scraping same targets | | Retention planning | Match --storage.tsdb.retention.time to disk capacity | | Federation for scale | Global Prometheus aggregates from regional instances | | Long-term storage | Thanos or Cortex for >30d retention |

Troubleshooting Quick Reference

| Problem | Diagnosis | Fix | |---------|-----------|-----| | Target shows DOWN | Check /targets page for error | Fix firewall, verify endpoint, check TLS | | Metrics missing | Query up{job="x"} | Verify scrape config, check /metrics endpoint | | High cardinality | prometheus_tsdb_head_series growing | Drop high-cardinality labels with metric_relabel_configs | | Storage filling up | Check prometheus_tsdb_storage_* | Reduce retention, add disk, enable compaction | | Slow queries | Check prometheus_engine_query_duration_seconds | Add recording rules, reduce range, limit series | | Config not applied | Check prometheus_config_last_reload_successful | Fix syntax, POST /-/reload |

NEVER Do

| Anti-Pattern | Why | Do Instead | |-------------|-----|------------| | Scrape interval < 5s | Overwhelms targets and storage | Use 15–60s intervals | | High-cardinality labels (user ID, request ID) | Explodes TSDB series count | Use logs for high-cardinality data | | Alert without for duration | Fires on transient spikes | Always set for: 1m minimum | | Skip recording rules | Dashboards compute expensive queries every load | Pre-compute with recording rules | | Store secrets in prometheus.yml | Config often in Git | Use file-based secrets or env substitution | | Ignore up metric | Miss targets silently going down | Alert on up == 0 for all jobs | | Single Prometheus instance in prod | Single point of failure | Run 2+ replicas with shared targets | | Unbounded retention | Disk fills, Prometheus crashes | Set explicit --storage.tsdb.retention.time |

Templates

| Template | Description | |----------|-------------| | templates/prometheus.yml | Full config with static, file-based, and K8s discovery | | templates/alert-rules.yml | 25+ alert rules by category | | templates/recording-rules.yml | Pre-computed metrics for HTTP, latency, resources, SLOs |

| Practice | Detail | |----------|--------| | Naming: prefix_name_unit | Snake_case, _total for counters, _seconds/_bytes for units | | Scrape intervals 15–60s | Shorter wastes resources and storage | | Recording rules for dashboards | Pre-compute anything queried repeatedly | | Monitor Prometheus itself | prometheus_tsdb_*, scrape_duration_seconds | | HA deployment | 2+ instances scraping same targets | | Retention planning | Match --storage.tsdb.retention.time to disk capacity | | Federation for scale | Global Prometheus aggregates from regional instances | | Long-term storage | Thanos or Cortex for >30d retention |

# Validate config syntax
promtool check config prometheus.yml
Validate rule files
promtool check rules /etc/prometheus/rules/*.yml
Test a query
promtool query instant http://localhost:9090 'up'
Reload config without restart
curl -X POST http://localhost:9090/-/reload