How to Setup Log Rotation with Logrotate on Ubuntu 18.04

Log files are intended to retain information about system activities for a reasonable period of time, but system logging daemons provide no means to control the size of log files. Here I will show you how to rotate the log files with logrotate on Ubuntu 18.04. Left unchecked, log files can, therefore, grow to consume all of the available partition space on which they reside. To prevent log files from filling up your system completely, the log messages can be rotated, meaning that when a certain threshold has been reached, the old log file is closed and a new log file is opened.

What is log rotation?

Log rotation is a process that solves these problems by periodically archiving the current log file and starting a new one. It renames and optionally compresses the current log files, delete old log files, and force the logging system to begin using new log files. It usually runs automatically through the cron utility. With log rotation, you can

  • start a new log file on a schedule, such as daily, weekly, or monthly
  • compress old log files to save disk space
  • prune old archives so that you only keep a certain number of old logs.
  • rename old log files with a date stamp so you know where to look for older logs.
  • run commands before or after a set of logs is rotated.

You can see the log rotation as the process that renames a current log file and sets up a new log file for new log entries. When a log file is rotated, the old log file is typically copied to a file that has the rotation date in it.

1) The configuration files of logrotate

On Ubuntu 18.04, the logrotate’s configuration files are /etc/logrotate.conf, along with any files in /etc/logrotate.dThe main logrotate.conf file specifies any default settings and system files to be rotated. The file contains some options that are well explained by the comments.

cat /etc/logrotate.conf 
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

The main options of logrotate are:

  • weekly: rotates the log files once a week
  • su root syslog: will archive the logs using the specific user (root) and group (syslog) to prevent issues with permissions
  • rotate 4: makes sure that four old versions of the file are saved. Rotates a given log four times before deleting it, so this keeps four weeks of logs online.
  • create: The old file is saved under a new name and a new file is created
  • compress: compress causes logrotate to compress log files to save space. This is done using gzip by default, but you can specify another program
  • include: This important option makes sure that the content of the directory /etc/logrotate.d is included. In this directory, files exist that specify how to handle some individual log files.

Each file in /etc/logrotate.d is used to add additional settings or override the default settings for specific files. The folder also contains the logrotate configuration of any packages you install that need log rotation

# ls -l /etc/logrotate.d
total 40
-rw-r--r-- 1 root root 120 Nov 2 2017 alternatives
-rw-r--r-- 1 root root 442 Oct 23 2017 apache2
-rw-r--r-- 1 root root 126 Nov 20 16:39 apport
-rw-r--r-- 1 root root 173 Apr 20 10:08 apt
-rw-r--r-- 1 root root 112 Nov 2 2017 dpkg
-rw-r--r-- 1 root root 146 Apr 17 17:35 lxd
-rw-r--r-- 1 root root 845 Jan 12 10:57 mysql-server
-rw-r--r-- 1 root root 501 Jan 14 16:19 rsyslog
-rw-r--r-- 1 root root 178 Aug 15 2017 ufw
-rw-r--r-- 1 root root 235 Apr 17 14:53 unattended-upgrades

You can see for example the logrotate configuration for the apache web server

# cat /etc/logrotate.d/apache2 
/var/log/apache2/*.log {
	daily
	missingok
	rotate 14
	compress
	delaycompress
	notifempty
	create 640 root adm
	sharedscripts
	postrotate
                if invoke-rc.d apache2 status > /dev/null 2>&1; then \
                    invoke-rc.d apache2 reload > /dev/null 2>&1; \
                fi;
	endscript
	prerotate
		if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
			run-parts /etc/logrotate.d/httpd-prerotate; \
		fi; \
	endscript
}

Logrotate contains some additionals options that you can see above:

  • daily: rotates log on basis day
  • missingok: Don’t raise an error if the log is missing
  • delaycompress: Don’t compress the file until it has already been rotated. This is to prevent corruption if the daemon doesn’t close the log file immediately.
  • notifyempty: Don't rotate the log file when it is empty
  • create 640 root adm: this will create new log files with set permissions, owner and group
  • sharedscripts: If the wildcard matches several files, run any scripts once for all files
  • postrotate/endscript: Specifies some commands or scripts that should be executed until the endscript keyword, after the logs were rotated.
  • prerotate/endscript: as postrotate/endscript, it specifies commands or scripts that should be executed before log rotating begins.

You can have some information about the additional options by using the manual

man logrotate

2) Configure log rotation with logrotate

The logrotate command typically renames (or rotates) log files on a cyclic basis; the log file will be renamed to contain a numeric or date extension, and a new log file will be created to accept system information. If logrotate is configured to keep only two copies of old log files, then after two log rotations, the oldest log file will automatically be removed.

Normally the logrotate command is installed by default on your Ubuntu 18.04 and you can check the version as below

# logrotate --version
logrotate 3.11.0

By default, logrotate runs on a daily basis by executing the shell script /etc/cron.daily/logrotate You will see that the other cron files don't contain any logrotate file

cat /etc/cron.daily/logrotate 
#!/bin/sh

# Clean non existent log file entries from status file
cd /var/lib/logrotate
test -e status || touch status
head -1 status > status.clean
sed 's/"//g' status | while read logfile date
do
    [ -e "$logfile" ] && echo "\"$logfile\" $date"
done >> status.clean
mv status.clean status

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

You can see what the logrotate configuration of application would do if it's executed with the -d parameter of the command

# logrotate -d /etc/logrotate.d/mysql-server 
reading config file /etc/logrotate.d/mysql-server
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /var/log/mysql.log /var/log/mysql/*log  after 1 days (7 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/mysql.log
error: skipping "/var/log/mysql.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
Creating new state
considering log /var/log/mysql/error.log
Creating new state
  Now: 2018-05-22 06:02
  Last rotated at 2018-05-22 06:00
  log does not need rotating (log has been already rotated)

You can configure log rotation as a root user and as a non-root user.

a) Set up logrotation for installed server packages

Logrotate configuration files as root user are created in the /etc/logrotate.d folder on a daily basis. For example, we will configure the log rotation for the monitoring tool supervisor. First, we will create the log rotation file on which we will indicate the config file that will store the logs

# vim /etc/logrotate.d/supervisor
/var/log/supervisor/superviz.log {
        daily
        create 0640 root root
        missingok
        dateext
        rotate 3
        size=1M
        notifempty
        sharedscripts
        mail [email protected]
}

Let's explain some options:

  • dateext: uses the date as extension for old versions of the log files
  • size: logs files that grow bigger than the size specified here
  • mail: to instruct logrotate to e-mail its contents to the specified address before overwriting it. You must have a mail server working

Now we can test the configuration by calling logrotate on debug mode which points to the main configuration containing our folder. The command will show some information but we will only sort those about our configuration

# logrotate /etc/logrotate.conf --debug
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file alternatives
reading config file apache2
reading config file apport
reading config file apt
reading config file dpkg
reading config file lxd
reading config file mysql-server
reading config file rsyslog
reading config file supervisor
reading config file ufw
reading config file unattended-upgrades
Reading state from file: /var/lib/logrotate/status
...
...
rotating pattern: /var/log/supervisor/supervisord.log  1048576 bytes (3 rotations)
empty log files are not rotated, old logs mailed to [email protected]
switching euid to 0 and egid to 106
considering log /var/log/supervisor/supervisord.log
Creating new state
  Now: 2018-05-22 08:16
  Last rotated at 2018-05-22 08:00
  log does not need rotating (log size is below the 'size' threshold)
switching euid to 0 and egid to 0
....
....

You can see that our configuration doesn't need rotating yet. This is because the condition size it not verified yet. Once this command runs without any errors, it means that the configuration looks good.

b) Set up log rotation for custom application

You can configure log rotation for custom applications that generate logs and run as non-root user. By default logrotate runs on daily basis so, if we need to run our application on hourly basis, we will create the logrotate configuration outside the default folder.

Let's take an example by configuring log rotation for discord. We will create a personal folder which will contain the main logrotate configuration file and a folder for log files. This application has no understanding of log rotation so logrotate will handle this by using a copy and truncate implementation

$ vim apps/logrotate.conf
/home/alain/apps/logs/discord.log {
        hourly
        copytruncate
        missingok
        dateext
        rotate 10
        compress
}

We used new options copytruncate that truncates the old log file in place after creating a copy, instead of moving the old file and creating a new one. This is useful for services that cannot be told to close their log files.

We can test our configuration but we need to specify a state file though. This file records what logrotate saw and did the last time it ran so that it knows what to do the next time it runs. Notice that we run the command as non-root user

$ logrotate /home/alain/apps/logrotate.conf --state /home/alain/apps/logrotate-state --verbose
reading config file /home/alain/apps/logrotate.conf
Reading state from file: /home/alain/apps/logrotate-state
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /home/alain/apps/logs/discord.log  hourly (10 rotations)
empty log files are rotated, old logs are removed
considering log /home/alain/apps/logs/discord.log
Creating new state
  Now: 2018-05-22 10:09
  Last rotated at 2018-05-22 10:00
  log does not need rotating (log has been already rotated)

If you try the same command after few hours, you can see some additional information in the output such as the truncating and the compressing infos

logrotate /home/alain/apps/logrotate.conf --state /home/alain/apps/logrotate-state --verbose
reading config file /home/alain/apps/logrotate.conf
Reading state from file: /home/alain/apps/logrotate-state
Allocating hash table for state file, size 64 entries
Creating new state

Handling 1 logs

rotating pattern: /home/alain/apps/logs/discord.log  hourly (10 rotations)
empty log files are rotated, old logs mailed to [email protected]
considering log /home/alain/apps/logs/discord.log
  Now: 2018-05-22 22:52
  Last rotated at 2018-05-22 10:23
  log needs rotating
rotating log /home/alain/apps/logs/discord.log, log->rotateCount is 10
dateext suffix '-2018052222'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
copying /home/alain/apps/logs/discord.log to /home/alain/apps/logs/discord.log-2018052222
truncating /home/alain/apps/logs/discord.log
compressing log with: /bin/gzip

You must retain that a logrotate configuration is defined by a file glob that matches the log files, followed by a series of instructions enclosed inside curly braces. If an option isn’t specified in the stanza attached to the name of the log file, the top level option from /etc/logrotate.conf takes priority. Log files that have been rotated are not stored anywhere; they are just gone so you should think to take measures.

Read also:

Alain Francois 8:00 am

About Alain Francois

IT Linux administrator passionate of free and open source software, I work on Linux Systems since some years on installations and deployments of certain solutions. I like to share my experiences with a wider audience by training and conferences.

Author Archive Page

Have anything to say?

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.