How to Setup Monitoring for Docker Containers using Prometheus

Prometheus is a next-generation open source monitoring system from SoundCloud and is designed for monitoring such as the generation and collection of metrics, graphing the resulting data on dashboards, and alerting on anomalies etc. In this tutorial, we will install/configure following component:

i)  Prometheus as a Docker service although it can be installed as a binary or built from source.
ii) Prometheus Node Exporter which exposes the metrics of the host machine.
iii) Grafana the face of Prometheus.
iv)  Prometheus alertmanager which will generate alerts based on some criteria and send it to email, hipchat, slacks etc.
v) cAdvisior - collects, aggregates, processes, and exports information about running containers.

Components

The primary function of prometheus is to query docker services on predefined metrics, create graphs, query database, to check health status of services and to notify the alertmanager based on alert rules. The execution of notification is done by alertmanager and routes alerts to different channels like hipchat, slacks, email etc. The alertmanager can also suppress notification if it has the same source so as to avoid spam. Node exporter collects system matrices like CPU/memory/storage usage for the host machine and exports these to prometheus in format that it understands. Lastly cAdvisor scrapes information about containers inside the host system and send this data to the prometheus.

Requirements

To install the above components, make sure you have installed these two pre-requisites i) docker ii) docker-compose.

Install Prometheus

Let us create a very simple docker-compose.yml configuration file and prometheus configuration file prometheus.yml to install prometheus.

# docker-compose.yml
version: '2'
services:
  prometheus:
     image: prom/prometheus:latest
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml            
     command:
       - '--config.file=/etc/prometheus/prometheus.yml'
     ports:
       - '9090:9090'

Prometheus configuration file-

# prometheus.yml
global:
 scrape_interval: 5s
 external_labels:
   monitor: 'my-monitor'
rule_files:
 - 'alert.rules'
scrape_configs:
 - job_name: 'prometheus'
   static_configs:
     - targets: ['123.45.67.89:9090']

Inside docker-compose.yml the mapping of prometheus config file into the container as a volume is done under volume tag and adding a config parameter to the command pointing to this file. let Us launch prometheus using following command.

root@demohost:~#  docker-compose up

Now point your browser to http://Target-IP-adress:9090 to confirm that server is running and click ‘Status’ to check the configuration file that we have provided.

The raw metrics can be checked by visiting http://Target-IP-adress:9090/metrics Next click the target option from the status menu, you will find that prometheus state is UP.

Install Node exporter

The Node Exporter exposes the prometheus metrics of the host machine in which it is running and shows the machine’s file system, networking devices, processor, memory usages and others features as well. Node exporter can be run as a docker container while reporting stats for the host system. We will append configuration setting to the existing docker-compose.yml and prometheus.yml to bring up life to node-exporter.

# docker-compose.yml
version: '2'
services:
  prometheus:
     image: prom/prometheus:latest
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml           
     command:
       - '--config.file=/etc/prometheus/prometheus.yml'
     ports:
       - '9090:9090'
  node-exporter:
     image: prom/node-exporter
     ports:
       - '9100:9100'

Prometheus configuration file-

# prometheus.yml
global:
 scrape_interval: 5s
 external_labels:
   monitor: 'my-monitor'
rule_files:
 - 'alert.rules'
scrape_configs:
 - job_name: 'prometheus'
   static_configs:
     - targets: ['123.45.67.89:9090']
 - job_name: 'node-exporter'
   static_configs:
     - targets: ['node-exporter:9100']

In prometheus.yml, we reference node-exporter by its service name that have already defined in docker-compose.yml
Restart docker-composer

root@demohost:~#  docker-compose restart

Now click the target option from the status menu, you will find that node exporter state is UP and healthy.

We will now execute a query to test the rich features of prometheus.  Click 'Graph' from the menu-bar and then select any metrics from the drop down box and click 'Execute'. Press 'Graph' tab to view the graph.

Install Grafana

As of now we have installed and configured prometheus server and Node exporter. Let us now set-up Grafana which is a graphical interface with a dashboard that supports prometheus as a back-end to query for the data to generate the graph.

Although prometheus has a built in graph features that can be access through its web interface but grafana offers a much more powerful features. Append the following lines to docker-compose.yml to setup grafana dashboard.

……
……
  grafana:
     image: grafana/grafana
     user: "0" #[ Execute  id -u in terminal to find the user id from where you are invoking docker compose]
     volumes:
       - ./grafana_db:/var/lib/grafana
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=yourpass
     depends_on:
       - prometheus
     ports:
       - '3000:3000'
…….
…….

Point your browser to http://Prometheus-IP:3000 to access grafana. Login to grafana with the user 'admin' and password as 'admin_password'. You will be taken to grafana dashboard.

Once you have successfully logged in to the grafana dashboard, click grafana icon followed by 'data sources' and finally 'Add data source' as shown in the image below.

In the add data source page under config tab, provide a name of the data source, type as prometheus. In Http settings, provide URL as the prometheus server IP and access as direct and click 'Add'.

Once data source is added, click the dashboard tab and import the data source that you have created just now.  You will find the newly created dashboard with revision 1.

Now click the home icon and select the prometheus dashboard that you have created in the previous step.

To ensure data actually persists in docker volumes we add following two lines in docker-compose.yml Make sure you have these two directories, prometheus_db and grafana_db in the working directory.

volumes:
- ./prometheus_db:/var/lib/prometheus

...............

...............
volumes:
- ./grafana_db:/var/lib/grafana

Alert / Notification

We will now configure alert rules to detect events based on metric criteria which has been track by prometheus. Here we will use alert-manager component to send notification to a hipchat account when alerts are triggered.

To setup notification, we need to configure three files-
-    Alert.rules to define rules on which alert will be fired
-    Map this file with the container in docker-compose.yml
-    Edit Prometheus.yml to add alertmanager as a service.

We will add the following two alert rules  1) service_down 2) high_load  in alert.rules

groups:
- name: alert.rules
  rules:
  - alert: service_down
    expr: up == 0
  - alert: high_load
    expr: node_load1 > 0.5
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} is under high
        load.'
      summary: Instance {{ $labels.instance }} under high load

To map the alert rules in docker-compose.yml, add the following line to the volumes section of the prometheus service in docker-compose.yml

version: '2'
services:
  prometheus:
     image: prom/prometheus:latest
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml
       - ./prometheus_db:/var/lib/prometheus
       - ./alert.rules:/etc/prometheus/alert.rules
………….
………….

At last edit prometheus.yml to tell the prometheus about the path of rule files.

rule_files:
 - 'alert.rules'

......................

......................

Let us restart the containers with the following command-

root@demohost:~# docker-compose restart

Once docker is up, click alerts in prometheus dashboard, you can view the alerts.

As we have already defined an alert by the name high_load with load average greater than 0.5, let us trigger this alert by creating a load with the following command.

root@demohost:~#  docker run --rm -it busybox sh -c "while true; do :; done"

After few seconds, we will see the alert high_load is active.

Remember alerts themselves are metrics means alerts can be viewed in grafana dashboard. To do that, click add row->click hamburger menu from the left->Add panel->graph . Select panel data source as prometheus , place mouse cursor in the metric lookup box, you will find ALERT at the top of the metrics.

Select 'prometheus' from the drop-box and check 'alerts' from metric lookup

High load alerts is now visible in grafana dashboard.

Now coming back to alert manager, we will append few lines for it in the docker-compose.yml just like we did it for prometheus and exporter. Our final configuration for docker-compose.yml looks like this.

# docker-compose.yml
version: '2'
services:
  prometheus:
     image: prom/prometheus:latest
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml
       - ./prometheus_db:/var/lib/prometheus
       - ./alert.rules:/etc/prometheus/alert.rules
     command:
       - '--config.file=/etc/prometheus/prometheus.yml'
     ports:
       - '9090:9090'
     depends_on:
       - alertmanager

  node-exporter:
     image: prom/node-exporter
     ports:
       - '9100:9100'

  grafana:
     image: grafana/grafana
     user: "0"
     volumes:
       - ./grafana_db:/var/lib/grafana
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=yourpass
     depends_on:
       - prometheus
     ports:
       - '3000:3000'

  alertmanager:
     image: prom/alertmanager:latest
     volumes:
       - ./alertmanager.yml:/alertmanager.yml
     command:
       - '--config.file=/etc/alertmanager/alertmanager.yml'
     ports:
       - '9093:9093'

This is all need to be done to configure alert manager and to connect Prometheus but we still need to create alertmanager.yml. Before that let us configure an alert receiver.

Configure Alert receiver

Login to your hipchat account->select the room->integrations  Now click “Build your own integration's”

Give a name to the integration and click create.

Note down room no and authentication token that we will use in configuring alertmanager.conf

This is our alertmanager.yml which is configured to send alerts to a hipchat account. You can add more receivers like slack, email etc in this file.

global:

  hipchat_auth_token: 'xxx'
  hipchat_api_url: 'https://api.hipchat.com/'

route:
  group_by: [cluster]
  receiver: team-hipchat
  routes:
  - match:
      severity: hipchat
    receiver: team-hipchat

receivers:
  - name: team-hipchat
    hipchat_configs:
    - auth_token: 'yyy'
      room_id: 12345
      message_format: html
      notify: true

Once an alert is triggered by prometheus, the alert-manager will send the alert to the above hipchat account. You can verify it by navigating to the chat history.

cAdvisor

To install cAdvisor, append the following lines to the docker-compose.yml

..............
..............
  cadvisor:
     image: google/cadvisor:latest
     ports:
       - '8080:8080'
     volumes:
       - /:/rootfs:ro
       - /var/run:/var/run:rw
       - /sys:/sys:ro
       - /var/lib/docker/:/var/lib/docker:ro

Also make an entry of cAdvisor service to prometheus.yml

..............
..............
 - job_name: 'cAdvisor'
   static_configs:
     - targets: ['123.45.67.89:8080']

That's all, we are done with cAdvisor. Restart docker-compose

root@demohost:~# docker-compose restart

Point your browser to http://target-IP:9090/targets to check the status of cAdvisor. If there is no error its state will be UP.

To check the container metrics, click graphs from the top menu bar and select any of the container parameter from the drop-down list and click execute. Select 'graph' tab to view the graph based on container metrics that you have selected just now.

Conclusion

In this tutorial, we configured prometheus server, configured a metric scrapping target, configured alertmanager and enabled notifications to hipchat all using docker. We have also configured cAdvisor to monitor containers inside the host machine. Prometheus is an awesome monitoring service that gets metrics data from node-manager and container advisor and access these metrics from grafana. There is no better way to monitor applications and containerized services.