Configuring Prometheus Alerts

Log in to the Prometheus server.

Edit the Prometheus config file:

sudo vi /etc/prometheus/prometheus.yml

Add a path for rules files to the Prometheus config:

rule_files:
  - "/etc/prometheus/rules/*.yml"

Create the rules directory:

sudo mkdir -p /etc/prometheus/rules/

Create a new rules file for your alerting rule:

sudo vi /etc/prometheus/rules/limedrop-web.yml

Implement an alerting rule to issue an alert when the server goes down:

groups:
- name: limedrop-web
  rules:
  - alert: WebServerDown
    expr: up{instance="limedrop-web:9100"} == 0
    labels:
      severity: critical
    annotations:
      summary: Web Server Down

Restart Prometheus to reload the configuration:

sudo systemctl restart prometheus

Access Prometheus in a browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.

Click Alerts. You should see your WebServerDown alert listed.

Edit the Alertmanager configuration file:

sudo vi /etc/alertmanager/alertmanager.yml

Add a new node to the routing table:

route:

  ...

  routes:
  - receiver: 'web.hook'
    group_wait: 30s
    match_re:
      alertname: WebServerDown

Restart Alertmanager to reload the configuration:

sudo systemctl restart alertmanager

You can test your alert by shutting down the limedrop-web exporter to simulate the server going down. To do this, stop the limedrop_web_exporter service:

sudo systemctl stop limedrop_web_exporter

After a few moments, check the Alertmanager page in your browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. If the alert is firing, you should see the WebServerDown alert appear.

If you wish, you can start limedrop_web_exporter again and watch the alert disappear in Alertmanager:

sudo systemctl start limedrop_web_exporter