Creating Alerting Rules

Edit prometheus-rules-config-map.yml and add the Redis alerting rules. It should look like this when we're done:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: prometheus-rules-conf
  namespace: monitoring
data:
  redis_rules.yml: |
    groups:
    - name: redis_rules
      rules:
      - record: redis:command_call_duration_seconds_count:rate2m
        expr: sum(irate(redis_command_call_duration_seconds_count[2m])) by (cmd, environment)
      - record: redis:total_requests:rate2m
        expr: rate(redis_commands_processed_total[2m])
  redis_alerts.yml: |
    groups:
    - name: redis_alerts
      rules:
      - alert: RedisServerDown
        expr: redis_up{app="media-redis"} == 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: Redis Server {{ $labels.instance }} is down!
      - alert: RedisServerGone
        expr:  absent(redis_up{app="media-redis"})
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: No Redis servers are reporting!

Now, apply the changes that were made to prometheus-rules-config-map.yml:

kubectl apply -f prometheus-rules-config-map.yml

1. List the pods to find the name of the Prometheus pod:

kubectl get pods -n monitoring

2. Delete the Prometheus pod:

kubectl delete pods <POD_NAME> -n monitoring

3. In a new browser tab, navigate to the Expression browser:

http://<IP>:30080

4. Click on the Alerts link to verify that the two Redis alerts are showing as green.