Kubernetes Monitoring with Prometheus

Log in to the Kubernetes server.

Install kube-state-metrics:

git clone https://github.com/kubernetes/kube-state-metrics.git

cd kube-state-metrics/

git checkout v1.8.0

kubectl apply -f kubernetes

Create a NodePort service to expose the metrics service outside the cluster, making it accessible to the Prometheus server:

cd ..

vi kube-state-metrics-nodeport-svc.yml

Define the NodePort service in a YAML file:

kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: kube-state-nodeport
spec:
  selector:
    k8s-app: kube-state-metrics
  ports:
  - protocol: TCP
    port: 8080
    nodePort: 30000
  type: NodePort

Create the NodePort service and test it:

kubectl apply -f kube-state-metrics-nodeport-svc.yml

curl localhost:30000/metrics

Log in to the Prometheus server.

Edit the Prometheus config:

sudo vi /etc/prometheus/prometheus.yml

Under the scrape_configs section, add a scrape configuration for Kubernetes:

- job_name: 'Kubernetes'
  static_configs:
  - targets: ['limedrop-kube:30000']

Restart Prometheus to load the new configuration:

sudo systemctl restart prometheus

Use the expression browser to verify you can see Kubernetes metrics in Prometheus. You can access the expression browser in a web browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.

Run a query to view some Kubernetes metric data:

kube_pod_status_ready{namespace="default",condition="true"}