How to Install Prometheus on Ubuntu 18.04

Prometheus is a very powerful open-source monitoring and alerting system suitable for dynamic environments like in Cloud space. It is a standalone project maintained by the community with many companies and organizations contributing to its source code. Most components of Prometheus are written in Go, making them easy to build and deploy as static binaries. Rarely should you need to compile the application from code. In this guide, we will cover how to install to Prometheus on Ubuntu 18.04 server.

Prometheus is capable of recording any purely numeric time series, for both machine-centric monitorings like Linux systems as well as monitoring of highly dynamic service-oriented architectures. Visualization tools like Grafana support querying Prometheus with the Grafana data source for Prometheus included since Grafana 2.5.0.

Components of Prometheus

The Prometheus ecosystem comprises various components which together work to provide a complete monitoring and alerting system. Most of the components are optional, and you only need to install them on demand. The major components are:

  • Prometheus server - This scrapes and stores time series data from client systems and applications.
  • Exporters - used in exporting existing metrics from third-party systems as Prometheus metrics.
  • An alertmanager used to handle alerts
  • The client libraries for instrumenting application code

Features of Prometheus

The main features of Prometheus are:

  • a multi-dimensional data model with time series data identified by metric name and key/value pairs
  • It has a very flexible query language to leverage its multi-dimensional model
  • Single server nodes are autonomous and don't rely on distributed storage.
  • Prometheus time series collection uses pull model over HTTP protocol
  • It has automatic service discovery of targets and configuration files can be used for the same.
  • pushing time series is supported via an intermediary gateway
  • Prometheus has support for multiple modes of graphing and dashboarding

Installing Prometheus on Ubuntu 18.04

Since Prometheus is written in Go, its packages are distributed as binaries. As of this writing, the latest release of Prometheus is v2.2.1. You can check releases from the Official Download page.  Packages provided are for macOS, Linux, and Windows operating systems, so make sure you are downloading the correct package.

$ sudo su -
$ export RELEASE="2.2.1"
$ wget https://github.com/prometheus/prometheus/releases/download/v${RELEASE}/prometheus-${RELEASE}.linux-amd64.tar.gz

After downloading the archive, extract it using tar.

$ tar xvf prometheus-${RELEASE}.linux-amd64.tar.gz
prometheus-2.2.1.linux-amd64/
prometheus-2.2.1.linux-amd64/consoles/
prometheus-2.2.1.linux-amd64/consoles/index.html.example
prometheus-2.2.1.linux-amd64/consoles/node-cpu.html
prometheus-2.2.1.linux-amd64/consoles/node-disk.html
prometheus-2.2.1.linux-amd64/consoles/node-overview.html
prometheus-2.2.1.linux-amd64/consoles/node.html
prometheus-2.2.1.linux-amd64/consoles/prometheus-overview.html
prometheus-2.2.1.linux-amd64/consoles/prometheus.html
prometheus-2.2.1.linux-amd64/console_libraries/
prometheus-2.2.1.linux-amd64/console_libraries/menu.lib
prometheus-2.2.1.linux-amd64/console_libraries/prom.lib
prometheus-2.2.1.linux-amd64/prometheus.yml
prometheus-2.2.1.linux-amd64/LICENSE
prometheus-2.2.1.linux-amd64/NOTICE
prometheus-2.2.1.linux-amd64/prometheus
prometheus-2.2.1.linux-amd64/promtool

Change to the newly created directory from file extractions.

# cd prometheus-${RELEASE}.linux-amd64/

Create Prometheus system group

Prometheus needs its own user and group to run as.

# groupadd --system prometheus
# grep prometheus /etc/group
prometheus:x:999:

Create Prometheus system user

Now that we have Prometheus group, let's create a user and assign it the group created.

# useradd -s /sbin/nologin -r -g prometheus prometheus
# id prometheus
uid=999(prometheus) gid=999(prometheus) groups=999(prometheus)

Create configuration and data directories for Prometheus

Prometheus needs a directory to store its data and configuration files. We will create /var/lib/prometheus for data and /etc/prometheus for configuration files.

# mkdir -p /etc/prometheus/{rules,rules.d,files_sd}  /var/lib/prometheus

Copy Prometheus binary files to a directory in your $PATH

Preferred directory to put third party binaries on Linux is  /usr/local/bin/ since it is in $PATH by default and it doesn't interfere with system binaries.

# cp prometheus promtool /usr/local/bin/
# ls /usr/local/bin/
prometheus promtool

Copy consoles and console_libraries to configuration files directory

Console files and libraries need to be placed under  /etc/prometheus/ directory.

# cp -r consoles/ console_libraries/ /etc/prometheus/

Create systemd unit file:

Ubuntu 18.04 uses systemd init system by default, we need to create a Service unit file for managing Prometheus service. We will put the file under /etc/systemd/system directory. The name of the file has to end with .service 

# cat /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus systemd service unit
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Note that:

  • We bind the service to 0.0.0.0:9090. This will be accessible from all network interfaces on the server. Limit it by specifying IP address for the interface you want to use, 127.0.0.1 for local access only.
  • Configuration file specified is /etc/prometheus/prometheus.yml. We will create a basic configuration file to use next.

Create Prometheus Configuration file

This will be placed in /etc/prometheus/ directory.

# cat  /etc/prometheus/prometheus.yml
# Global config
global: 
 scrape_interval: 15s # Set the scrape interval to every 15 seconds.
 evaluation_interval: 15s # Evaluate rules every 15 seconds. 
 scrape_timeout: 15s # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.
scrape_configs:
 # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
 - job_name: 'prometheus'

# metrics_path defaults to '/metrics'
 # scheme defaults to 'http'.

static_configs:
 - targets: ['localhost:9090']

For more configuration options, refer to official Prometheus Configuration guide.

Change directory permissions to Prometheus user and group

The ownership of Prometheus files and data should be its user and group.

# chown -R prometheus:prometheus /etc/prometheus/  /var/lib/prometheus/
# chmod -R 775 /etc/prometheus/ /var/lib/prometheus/

Start and enable Prometheus service

Start and enable Prometheus service to start on boot.

# systemctl start prometheus
# systemctl enable prometheus
Created symlink from /etc/systemd/system/multi-user.target.wants/prometheus.service to /etc/systemd/system/prometheus.service.

Check status:

# systemctl status prometheus


Confirm that port 9090 is listening.

# netstat -tunlp | grep 9090
tcp6 0 0 :::9090 :::* LISTEN 2140/prometheus

# telnet 127.0.0.1 9090
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]
If you have a firewall like ufw, Open port 9090 to be able to access it from a remote device.

# ufw allow 9090
Rule added
Rule added (v6)

To allow for a specific network, use:

# ufw allow from 192.168.10.0/24 to any port 9090
Rule added

# ufw status
Status: active

To Action From
-- ------ ----
9090 ALLOW Anywhere 
9090 ALLOW 192.168.10.0/24 
9090 (v6) ALLOW Anywhere (v6)

To get Web interface, open http://ip:9090

Read also:

The next thing you may want to take a look is the configuration of Various exporters for metrics collection and visualization. The best place to start is Exporters Instrumentation page.

Josphat Mutai 12:45 am

Comments

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.

13 Comments

  1. Hello,

    I just followed your tutorial, thanks for that.
    After created the "Prometheus.yml", the service wasn't able to start. It seems that 3 spaces are needed behind "static_configs:", because this static_config is for the "job_name:prometheus"

    I saw the same config default file on prometheus website, and maybe a visual explaination will be more clear !
    https://prometheus.io/docs/prometheus/latest/getting_started/#configuring-prometheus-to-monitor-itself

    Kind regards, thanks again for this tutorial !
    Simon

  2. Hi,
    First thanks that your tutorial, it's very clear.
    However I'm facing an issue. It all goes well until I try to check status.
    Running "# systemctl status prometheus" always result in:
    "# systemctl status prometheus
    ● prometheus.service - Prometheus systemd service unit
    Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Tue 2019-06-18 08:05:22 UTC; 19min ago
    Main PID: 2249 (code=exited, status=1/FAILURE)

    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Failed with result 'exit-code'.
    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Service hold-off time over, scheduling restart.
    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5.
    Jun 18 08:05:22 ubuntu systemd[1]: Stopped Prometheus systemd service unit.
    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Start request repeated too quickly.
    Jun 18 08:05:22 ubuntu systemd[1]: prometheus.service: Failed with result 'exit-code'.
    Jun 18 08:05:22 ubuntu systemd[1]: Failed to start Prometheus systemd service unit."

    Any idea ?

  3. Doesn't work at all

    2 14:16:21 Ubuntu-1804-bionic-64-minimal systemd[1]: prometheus.service: Service hold-off time over, scheduling restart.
    Sep 22 14:16:21 Ubuntu-1804-bionic-64-minimal systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5.
    Sep 22 14:16:21 Ubuntu-1804-bionic-64-minimal systemd[1]: Stopped Prometheus systemd service unit.
    Sep 22 14:16:21 Ubuntu-1804-bionic-64-minimal systemd[1]: prometheus.service: Start request repeated too quickly.
    Sep 22 14:16:21 Ubuntu-1804-bionic-64-minimal systemd[1]: prometheus.service: Failed with result 'start-limit-hit'.

    1. One reason I could thing is prometheus will fail to start if the configuration file is incorrectly formatted as it uses YAML. If you copied then it might have issue with config.
      Also, check Syslog why it tried to restart too many times.

    2. I had the same issue and managed to start the service it by changing the yml file and removing the last two lines:
      static_configs:
      - targets: ['localhost:9090']

      Still server defaults to 9090 port.
      Not sure if this is because the version I am using (prometheus-2.14.0-rc.0.linux-amd64.tar)

  4. How do you start it in WSL2 (windows services for linux) since Ubuntu 20.04 LTS from the Windows store does not use systemd?
    Executing: /lib/systemd/systemd-sysv-install enable prometheus
    root@Rockheart:~/prometheus-2.2.1.linux-amd64# systemctl status prometheus
    System has not been booted with systemd as init system (PID 1). Can't operate.
    Failed to connect to bus: Host is down

    I tried through init.d and got an error and its using the same config file (I cut and paste it off of this site into nano).

    root@Rockheart:~/prometheus-2.2.1.linux-amd64# /etc/init.d/prometheus start
    * Starting monitoring system and time series database prometheus /etc/init.d/prometheus: 62: /lib/init/init-d-script: /usr/bin/daemon: not found
    *
    Configuration test failed. Output of config test was:
    Checking /etc/prometheus/prometheus.yml
    FAILED: parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:
    line 15: field static_configs not found in type config.plain
    [fail]

  5. Ok update on my previous comment. Copying and pasting to the prometheus.yml file was the issue. I copied the default back out from /prometheus-2/2/1/linux-amd64 to /etc/prometheus/
    But then starting it via init.d (remember in windows services for linus systemd is not supported see: https://github.com/MicrosoftDocs/WSL/issues/457 )

    /etc/init.d/prometheus: 64: /lib/init/init-d-script: /usr/bin/daemon: not found

    Finally I had to run apt-get install daemon. Then it finally started and I got the windows firewall popup and it seems the service started.

    I'm just doing this test in WSL2 on Windows 10 2004 for now before a more permanent linux install.