How Use Systemd journalctl Command To Manage Logs

November 29, 2016 | By
| Reply More

Despite some of the negative publicity, systemd have brought great new functionality to the Linux operating system. Logging is one of them. In the old days before systemd, checking logs and troubleshooting issues was sometimes very unpleasant thing to do because logs for different programs were scattered in different files and you had to check timestamps in each and every one of them. Of course there was syslog file also, but sometimes was not as verbose as needed. The systemd brings us journal daemon called journald which enables centralized management of logs. This logs can be filtered and manipulated by utility called journalctl. In this article we will go through basic usage of utilities.

What is journald

The journald is service that collects and stores logs from many sources, and make indexed structured log files that are easy to interpret. Some of the sources that journald uses

  • Kernel log messages, via kmsg
  • Simple system log messages, via the libc syslog call
  • Structured system log messages via the native Journal API
  • Standard output and standard error of system services

Idea is to have all log messages centralized, no matter from which application they are coming from.

Setting the time

In order for logs to be useful, you must have correct system time. Logs can have time by UTC or by local time. Generally both time-stamps are saved. So lets set up timezone first

sudo timedatectl set-timezone Europe/Belgrade

This command sets the time to Central European timezone, but if you are on any other spot of the globe, you just need to enter continent and press tab to see cities you might chose. It doesn't matter if it is not your city, I am not living in Belgrade either, it is just closest big city. Next lets see the status of the timezone

miki@ThinkPad-X220T:~$ timedatectl status
Local time: Mon 2016-11-14 18:40:46 CET
Universal time: Mon 2016-11-14 17:40:46 UTC
RTC time: Mon 2016-11-14 17:40:46
Time zone: Europe/Belgrade (CET, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no

1) Viewing logs

Now that we have correct time, lets continue to viewing logs.  The basic command for viewing logs is

journalctl

It will get you output similar to this:

-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 18:42:51 CET

Nov 13 17:38:07 ThinkPad-X220T systemd-journald[344]: Runtime journal (/run/log/
Nov 13 17:38:07 ThinkPad-X220T kernel: microcode: microcode updated early to rev

That is first line, and followed by many lines. We see that logs begin from last system boot. I last time rebooted the computer yesterday. By default, journald doesn't save logs across reboots, because that would make log files get to big over time. The log seems similar to syslog, but it has many more sources. We said that both UTC and your local timezone timestamps are saved, so you can also look at the same logs with UTC timestamps.

journalctl --utc

Yes, I know that logs are huge. We said that logs will be great again! But they are too great, so the journald has neat feature of filtering logs.

If you want to see logs from some point in time till now, you need to use command formatted like this:

miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:15:00"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:07:51 CET
Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session opene
Nov 14 19:17:01 ThinkPad-X220T CRON[6049]: (root) CMD ( cd / && run-parts --re
Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session close
Nov 14 19:22:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo

...

2) Filtering logs by time

This is still too big, so we can make them smaller if we add one more operand after --since, and that will be --until

miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:00:00" --until "2016-11-14 19:05:00"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:21:23 CET
Nov 14 19:01:16 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo
Nov 14 19:02:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo

In those five minutes, there were only two log massages.

Since and until flags accept the time stamps in format YYYY-MM-DD HH:MM:SS but that is not only way. If for example get a user complain that your server is not working in last 5 minutes, you can examine logs like this:

journalctl --since "5 minutes ago"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:22:21 CET
Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:21:23 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:22:20 ThinkPad-X220T apport-gtk[5977]: gdk_pixbuf_composite: assertion
Nov 14 20:22:21 ThinkPad-X220T systemd[3634]: Started Notification regarding a c

It also understands words like yesterday or today so you can look for logs since yesterday or today.

3) Setting up logging across reboots

This this all is more useful if you have the logs persistent across reboots. If you want logs to remain after boots, you need to enable this manually. First create directory where it is going to be logged

sudo mkdir -p /var/log/journal

Then edit the configuration file to log persistently.

sudo sed -i.orig 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf

After this you can reboot your machine and check if boots span across reboots with following command:

miki@ThinkPad-X220T:~$ journalctl --list-boots
-1 7917f2c3498d4de89e94701ca8049245 Mon 2016-11-14 20:47:49 CET—Mon 2016-11-14 2
0 438c0a9258aa47fcbf88daf91106aaa3 Mon 2016-11-14 20:56:55 CET—Mon 2016-11-14 2

list of boots
I rebooted the computer two times and both boots are logged. I thought that previous boot would get saved as well but it did not. So only those reboots that are done after setting persistent logs will be logged.

4) Managing logs from different boots

If you now want to see logs from this boot only, you use command

journalctl -b

You can use boot number to pick specific boot

journalctl -b -1

Or you could use boot id

journalctl -b 7917f2c3498d4de89e94701ca8049245

The time based filtering will also work

journalctl -b 0 --since "15 minutes ago"

5) Filtering by service and id

Say you want to see logs by one program only. I will look logs of fprintd

journalctl -u fprintd.service
-- Logs begin at Mon 2016-11-14 20:47:49 CET, end at Mon 2016-11-14 21:28:03 CET
Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Starting Fingerprint Authentication D
Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Started Fingerprint Authentication Da
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: user 'miki' claiming the device: 0
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: now monitoring fd 15
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: device 0 claim status 0
Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: no longer monitoring fd 15
Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: released device 0
-- Reboot --

You can also filter logs by UID. First you need to get UID of your user

miki@ThinkPad-X220T:~$ id -u miki
1000

And next you can use that uid to filter logs

journalctl _UID=1000 --since "25 minutes ago"

You can also filter by GID

miki@ThinkPad-X220T:~$ journalctl -F _GID
30
102
130
1000
114
118
104
126
109
0

Now can filter by any of those GIDs

journalctl _GID=114

Filtering logs by path to to executable:

journalctl /usr/bin/sudo

Seeing logs in real time

journalctl -f

If you want your logs to be displayed in plan text for easy copy and paste, you can use "no pager" option

journalctl --no-pager

6) Displaying kernel messages and deleting old logs

The journalctl can display kernel messages as well. By default, it will display from current boot

journalctl -k

If you want kernel messages from previous boot, you use the -b -1 flag.

journalctl -k -b -1

To check how much disk space your logs use, you can use this command

journalctl --disk-usage
Archived and active journals take up 24.0M on disk.

If your logs have grown in size, you can use vacuum cleaner to wipe the oldest one. Well not really a vacuum cleaner, but command is called vacuum.

sudo journalctl --vacuum-size=16M

This will reduce the log size to 16MB.

Conclusion

We have gone through basic journald commands that make logging great again. This will hopefully help you greatly when you next time have to troubleshoot some sticky problem with your Linux server or desktop computer with systemd. The systemd is brings us great tools and negative publicity that it got early was just backlash against changing old tools that people were used too. I already got used to systemd and I don't know how I lived without it. This is all for this article, thank you for reading and have a nice day.

Filed Under : LINUX HOWTO

Tagged With : ,

Free Linux Ebook to Download

Leave a Reply

Commenting Policy:
Promotion of your products ? Comment gets deleted.
All comments are subject to moderation.