Prometheus is a next-generation open source monitoring system from SoundCloud and is designed for monitoring such as the generation and collection of metrics, graphing the resulting data on dashboards, and alerting on anomalies etc. In this tutorial, we will install/configure following component:
i) Prometheus as a Docker service although it can be installed as a binary or built from source.
ii) Prometheus Node Exporter which exposes the metrics of the host machine.
iii) Grafana the face of Prometheus.
iv) Prometheus alertmanager which will generate alerts based on some criteria and send it to email, hipchat, slacks etc.
v) cAdvisior - collects, aggregates, processes, and exports information about running containers.
Components
The primary function of prometheus is to query docker services on predefined metrics, create graphs, query database, to check health status of services and to notify the alertmanager based on alert rules. The execution of notification is done by alertmanager and routes alerts to different channels like hipchat, slacks, email etc. The alertmanager can also suppress notification if it has the same source so as to avoid spam. Node exporter collects system matrices like CPU/memory/storage usage for the host machine and exports these to prometheus in format that it understands. Lastly cAdvisor scrapes information about containers inside the host system and send this data to the prometheus.
Requirements
To install the above components, make sure you have installed these two pre-requisites i) docker ii) docker-compose.
Install Prometheus
Let us create a very simple docker-compose.yml configuration file and prometheus configuration file prometheus.yml to install prometheus.
# docker-compose.yml version: '2' services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml command: - '--config.file=/etc/prometheus/prometheus.yml' ports: - '9090:9090'
Prometheus configuration file-
# prometheus.yml global: scrape_interval: 5s external_labels: monitor: 'my-monitor' rule_files: - 'alert.rules' scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['123.45.67.89:9090']
Inside docker-compose.yml the mapping of prometheus config file into the container as a volume is done under volume tag and adding a config parameter to the command pointing to this file. let Us launch prometheus using following command.
root@demohost:~# docker-compose up
Now point your browser to http://Target-IP-adress:9090
to confirm that server is running and click ‘Status’ to check the configuration file that we have provided.
The raw metrics can be checked by visiting http://Target-IP-adress:9090/metrics Next click the target option from the status menu, you will find that prometheus state is UP.
Install Node exporter
The Node Exporter exposes the prometheus metrics of the host machine in which it is running and shows the machine’s file system, networking devices, processor, memory usages and others features as well. Node exporter can be run as a docker container while reporting stats for the host system. We will append configuration setting to the existing docker-compose.yml and prometheus.yml to bring up life to node-exporter.
# docker-compose.yml version: '2' services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml command: - '--config.file=/etc/prometheus/prometheus.yml' ports: - '9090:9090' node-exporter: image: prom/node-exporter ports: - '9100:9100'
Prometheus configuration file-
# prometheus.yml global: scrape_interval: 5s external_labels: monitor: 'my-monitor' rule_files: - 'alert.rules' scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['123.45.67.89:9090'] - job_name: 'node-exporter' static_configs: - targets: ['node-exporter:9100']
In prometheus.yml, we reference node-exporter by its service name that have already defined in docker-compose.yml
Restart docker-composer
root@demohost:~# docker-compose restart
Now click the target option from the status menu, you will find that node exporter state is UP and healthy.
We will now execute a query to test the rich features of prometheus. Click 'Graph' from the menu-bar and then select any metrics from the drop down box and click 'Execute'. Press 'Graph' tab to view the graph.
Install Grafana
As of now we have installed and configured prometheus server and Node exporter. Let us now set-up Grafana which is a graphical interface with a dashboard that supports prometheus as a back-end to query for the data to generate the graph.
Although prometheus has a built in graph features that can be access through its web interface but grafana offers a much more powerful features. Append the following lines to docker-compose.yml to setup grafana dashboard.
…… …… grafana: image: grafana/grafana user: "0" #[ Execute id -u in terminal to find the user id from where you are invoking docker compose] volumes: - ./grafana_db:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=yourpass depends_on: - prometheus ports: - '3000:3000' ……. …….
Point your browser to http://Prometheus-IP:3000
to access grafana. Login to grafana with the user 'admin' and password as 'admin_password'. You will be taken to grafana dashboard.
Once you have successfully logged in to the grafana dashboard, click grafana icon followed by 'data sources' and finally 'Add data source' as shown in the image below.
In the add data source page under config tab, provide a name of the data source, type as prometheus. In Http settings, provide URL as the prometheus server IP and access as direct and click 'Add'.
Once data source is added, click the dashboard tab and import the data source that you have created just now. You will find the newly created dashboard with revision 1.
Now click the home icon and select the prometheus dashboard that you have created in the previous step.
To ensure data actually persists in docker volumes we add following two lines in docker-compose.yml Make sure you have these two directories, prometheus_db and grafana_db in the working directory.
volumes: - ./prometheus_db:/var/lib/prometheus ............... ............... volumes: - ./grafana_db:/var/lib/grafana
Alert / Notification
We will now configure alert rules to detect events based on metric criteria which has been track by prometheus. Here we will use alert-manager component to send notification to a hipchat account when alerts are triggered.
To setup notification, we need to configure three files-
- Alert.rules to define rules on which alert will be fired
- Map this file with the container in docker-compose.yml
- Edit Prometheus.yml to add alertmanager as a service.
We will add the following two alert rules 1) service_down 2) high_load in alert.rules
groups: - name: alert.rules rules: - alert: service_down expr: up == 0 - alert: high_load expr: node_load1 > 0.5 annotations: description: '{{ $labels.instance }} of job {{ $labels.job }} is under high load.' summary: Instance {{ $labels.instance }} under high load
To map the alert rules in docker-compose.yml, add the following line to the volumes section of the prometheus service in docker-compose.yml
version: '2' services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./prometheus_db:/var/lib/prometheus - ./alert.rules:/etc/prometheus/alert.rules …………. ………….
At last edit prometheus.yml to tell the prometheus about the path of rule files.
rule_files: - 'alert.rules' ...................... ......................
Let us restart the containers with the following command-
root@demohost:~# docker-compose restart
Once docker is up, click alerts in prometheus dashboard, you can view the alerts.
As we have already defined an alert by the name high_load with load average greater than 0.5, let us trigger this alert by creating a load with the following command.
root@demohost:~# docker run --rm -it busybox sh -c "while true; do :; done"
After few seconds, we will see the alert high_load is active.
Remember alerts themselves are metrics means alerts can be viewed in grafana dashboard. To do that, click add row->click hamburger menu from the left->Add panel->graph . Select panel data source as prometheus , place mouse cursor in the metric lookup box, you will find ALERT at the top of the metrics.
Select 'prometheus' from the drop-box and check 'alerts' from metric lookup
High load alerts is now visible in grafana dashboard.
Now coming back to alert manager, we will append few lines for it in the docker-compose.yml just like we did it for prometheus and exporter. Our final configuration for docker-compose.yml looks like this.
# docker-compose.yml version: '2' services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./prometheus_db:/var/lib/prometheus - ./alert.rules:/etc/prometheus/alert.rules command: - '--config.file=/etc/prometheus/prometheus.yml' ports: - '9090:9090' depends_on: - alertmanager node-exporter: image: prom/node-exporter ports: - '9100:9100' grafana: image: grafana/grafana user: "0" volumes: - ./grafana_db:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=yourpass depends_on: - prometheus ports: - '3000:3000' alertmanager: image: prom/alertmanager:latest volumes: - ./alertmanager.yml:/alertmanager.yml command: - '--config.file=/etc/alertmanager/alertmanager.yml' ports: - '9093:9093'
This is all need to be done to configure alert manager and to connect Prometheus but we still need to create alertmanager.yml. Before that let us configure an alert receiver.
Configure Alert receiver
Login to your hipchat account->select the room->integrations Now click “Build your own integration's”
Give a name to the integration and click create.
Note down room no and authentication token that we will use in configuring alertmanager.conf
This is our alertmanager.yml which is configured to send alerts to a hipchat account. You can add more receivers like slack, email etc in this file.
global: hipchat_auth_token: 'xxx' hipchat_api_url: 'https://api.hipchat.com/' route: group_by: [cluster] receiver: team-hipchat routes: - match: severity: hipchat receiver: team-hipchat receivers: - name: team-hipchat hipchat_configs: - auth_token: 'yyy' room_id: 12345 message_format: html notify: true
Once an alert is triggered by prometheus, the alert-manager will send the alert to the above hipchat account. You can verify it by navigating to the chat history.
cAdvisor
To install cAdvisor, append the following lines to the docker-compose.yml
.............. .............. cadvisor: image: google/cadvisor:latest ports: - '8080:8080' volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro
Also make an entry of cAdvisor service to prometheus.yml
.............. .............. - job_name: 'cAdvisor' static_configs: - targets: ['123.45.67.89:8080']
That's all, we are done with cAdvisor. Restart docker-compose
root@demohost:~# docker-compose restart
Point your browser to http://target-IP:9090/targets to check the status of cAdvisor. If there is no error its state will be UP.
To check the container metrics, click graphs from the top menu bar and select any of the container parameter from the drop-down list and click execute. Select 'graph' tab to view the graph based on container metrics that you have selected just now.
Conclusion
In this tutorial, we configured prometheus server, configured a metric scrapping target, configured alertmanager and enabled notifications to hipchat all using docker. We have also configured cAdvisor to monitor containers inside the host machine. Prometheus is an awesome monitoring service that gets metrics data from node-manager and container advisor and access these metrics from grafana. There is no better way to monitor applications and containerized services.
A whole article about deploying prometheus in docker, and you can't copy/paste the yml, because of the indentation... Not really nice.
I realy appriciate the effort but as we cant copy paste the .yml files... this artical is of no use. SAD :(
Echo the others, please post updated files with indentation. I did manage to get it going, but it required a bit of trial and error, not knowing the syntax of the prometheus.yml file.
Also note, the command: "- '-config.file=/etc/prometheus/prometheus.yml'" is wrong. Should be two initial hyphens.
Thanks for posting this, I found it very helpful, as I was looking to move away from my zabbix monitoring following a move to a docker infrastructure.
Hi Indention's for YML files are updated for this article . Hopefully you will find it more useful
Further, You can also view the codes in github https://github.com/DwijadasDey/prometheus
Thanks