Monitoring client websites with Prometheus and Grafana

For the first few years of freelancing, my monitoring strategy was "wait for the client to tell me something is down." That stopped being acceptable around the time I had more than five sites to look after. Now every server I manage runs Prometheus and Grafana, and I know about problems before they become support tickets.

Here is the setup.

The architecture

One monitoring server runs Prometheus, Grafana, and Alertmanager. Each client server runs node_exporter (system metrics) and blackbox_exporter (uptime probes). PHP applications expose custom metrics at a /metrics endpoint.

# monitoring-server docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=90d'

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

volumes:
  prometheus_data:
  grafana_data:

Prometheus configuration

# prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - "alerts/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets:
          - 'client1.example.com:9100'
          - 'client2.example.com:9100'
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+).*'
        target_label: instance

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://client1.example.com
          - https://client2.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

Alerting rules that actually matter

# alerts/websites.yml
groups:
  - name: websites
    rules:
      - alert: SiteDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is unreachable"

      - alert: SlowResponse
        expr: probe_duration_seconds > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} responding slowly ({{ $value | humanizeDuration }})"

      - alert: SSLExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 14
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate for {{ $labels.instance }} expires in less than 14 days"

      - alert: HighDiskUsage
        expr: (node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk usage above 85% on {{ $labels.instance }}"

Custom PHP metrics in Yii2

Install the Prometheus PHP client and expose an endpoint:

composer require promphp/prometheus_client_php

// controllers/MetricsController.php
class MetricsController extends Controller
{
    public function behaviors(): array
    {
        return [
            'access' => [
                'class' => AccessControl::class,
                'rules' => [
                    ['allow' => true, 'ips' => ['monitoring.server.ip']],
                ],
            ],
        ];
    }

    public function actionIndex(): string
    {
        $registry = CollectorRegistry::getDefault();

        // Track active jobs in queue
        $gauge = $registry->getOrRegisterGauge('app', 'queue_size', 'Jobs in queue');
        $gauge->set(Queue::find()->where(['status' => 'pending'])->count());

        $renderer = new RenderTextFormat();
        Yii::$app->response->format = Response::FORMAT_RAW;
        Yii::$app->response->headers->set('Content-Type', RenderTextFormat::MIME_TYPE);
        return $renderer->render($registry->getMetricFamilySamples());
    }
}

What changes when you have this

The first week after I set this up, I caught a disk filling up on a client server before it caused any issues. The week after, I caught an SSL certificate that someone had forgotten to renew. Neither client ever knew there was a problem. That is the point.

The Slack alerts mean I see problems within two minutes, from my phone, without anyone having to report them. For a freelancer managing multiple clients, that is the difference between being reactive and being professional.