Configuring Prometheus to Use Service Discovery

Prepare the environment and create the monitoring namespace as below:

sudo su -
cd /root/prometheus
./bootstrap.sh
kubectl get pods -n monitoring

Edit prometheus-config-map.yml and add in the two service discovery targets:

vi prometheus-config-map.yml

When we're done, the whole file should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

Now, apply the changes that were made to prometheus-config-map.yml:

kubectl apply -f prometheus-config-map.yml

1. List the pods to find the name of the Prometheus pod:

kubectl get pods -n monitoring

2. Delete the Prometheus pod:

kubectl delete pods <POD_NAME> -n monitoring

3. Open up a new web browser tab, and navigate to the Expression browser. This will be at the public IP of the lab server, on port 30000:

http://<IP>:30000

4. Click on Status, and select Target from the dropdown. We should see two targets in there.