How to Monitor Kubernetes Cluster with Prometheus and Grafana

If you read the previous article about kops and how to make a cluster, this article will be a continuation of it. You will need kubernetes cluster on which you will install Prometheus and Grafana. You might need to upgrade this cluster though, and I will show how in this article.

Upgrading cluster with Kops

For successful operation of Prometheus we need Kubernetes 1.7.3, and we are actually going to upgrade to 1.7.5.

To setup the upgrade, we need to edit yaml of the cluster with the following command:

kops edit cluster

Then we will change the line that says kubernetesVersion to look like this:

kubernetesVersion: 1.7.5

Next, we will try out configuration for validity

kops update cluster

And then actually update cluster by running the same command with flag --yes.

kops update cluster --yes

After the kops finish this process, you will probably be prompted to do rolling update as well. You can do it with following commands:

kops rolling-update cluster

Again you need to say --yes to commit the changes

kops rolling-update cluster --yes

This will reboot all the AWS instances, and when they are back up, you will want to validate cluster again to see if all went well.

 kops validate cluster

Setting up Prometheus and Grafana

We are done with updating the cluster and we now need to install prometheus and grafana on top of it. We will start by cloning the repository.

git clone https://github.com/camilb/prometheus-kubernetes.git

Next , we will change directory into it.

cd prometheus-kubernetes/

We will edit service file of prometheus to change from ClusterIP to LoadBalancer

nano k8s/prometheus/prometheus.svc.ss.yaml

Just find clusterIP and change to LoadBalancer, to look similar to (but not exactly) like this:

apiVersion: v1
kind: Service
metadata:
 annotations:
 prometheus.io/scrape: 'true'
 labels:
 name: prometheus
 name: prometheus
 namespace: monitoring
spec:
 selector:
 app: prometheus
 type: LoadBalancer
 ports:
 - name: prometheus
 protocol: TCP
 port: 9090
 targetPort: 9090

We will also edit the grafana service, but in this case we need need to add type field bellow spec field.

nano k8s/grafana/grafana.svc.de.yaml

spec:
 type: LoadBalancer
 ports:
 - port: 3000
Now we are ready to go, Execute ./init.sh command and work the prompt as follows

Enter Grafana version [4.5.0]:
 Enter Prometheus version [v2.0.0-beta.3]:
 Enter Alert Manager version [v0.8.0]:
 Enter Node Exporter version [v0.14.0]:
 Enter Kube State Metrics version [v1.0.1]:
 Enter Dockerhub username []: yourdockerhubusername

Do you want to set up an SMTP relay?
 Y/N [N]: n

Do you want to set up slack alerts?
 Y/N [N]: n

Do you want to monitor EC2 instances in your AWS account?
 Y/N [N]: n

Creating 'monitoring' namespace.
 namespace "monitoring" created

Using RBAC?
 [y/N]: y

Now wait few minutes and then look for grafana and prometheus services. When it is done, you need to get address of LoadBalancer in order to access the web UI. You can do it with this command

kubectl describe svc -n monitoring prometheus grafana

The lines you are looking are LoadBalancer Ingress and Port. Port for grafana is 3000 and for prometheus is 9090. You need them to form the url together with load balancer. For example like this

The password for start is admin on username admin. You will probably want to change that. As we see, prometheus is happily monitoring our kubernetes cluster and sending data to Grafana dashboard. When you decide to remove grafana and prometheus, you can do so by executing ./remove.sh script while being in the repository. How to remove kops cluster was already shown in the previous article, but let's say again, here is command

kops delete cluster --name cluster2.k8s.local --yes

Off course, change the name of the cluster for you actual. With that we conclude this tutorial and wish you good day.