Despite some of the negative publicity, systemd have brought great new functionality to the Linux operating system. Logging is one of them. In the old days before systemd, checking logs and troubleshooting issues was sometimes very unpleasant thing to do because logs for different programs were scattered in different files and you had to check timestamps in each and every one of them. Of course there was syslog file also, but sometimes was not as verbose as needed. The systemd brings us journal daemon called journald which enables centralized management of logs. This logs can be filtered and manipulated by utility called journalctl. In this article we will go through basic usage of utilities.
What is journald
The journald is service that collects and stores logs from many sources, and make indexed structured log files that are easy to interpret. Some of the sources that journald uses
- Kernel log messages, via kmsg
- Simple system log messages, via the libc syslog call
- Structured system log messages via the native Journal API
- Standard output and standard error of system services
Idea is to have all log messages centralized, no matter from which application they are coming from.
Setting the time
In order for logs to be useful, you must have correct system time. Logs can have time by UTC or by local time. Generally both time-stamps are saved. So lets set up timezone first
sudo timedatectl set-timezone Europe/Belgrade
This command sets the time to Central European timezone, but if you are on any other spot of the globe, you just need to enter continent and press tab to see cities you might chose. It doesn't matter if it is not your city, I am not living in Belgrade either, it is just closest big city. Next lets see the status of the timezone
miki@ThinkPad-X220T:~$ timedatectl status Local time: Mon 2016-11-14 18:40:46 CET Universal time: Mon 2016-11-14 17:40:46 UTC RTC time: Mon 2016-11-14 17:40:46 Time zone: Europe/Belgrade (CET, +0100) Network time on: yes NTP synchronized: yes RTC in local TZ: no
1) Viewing logs
Now that we have correct time, lets continue to viewing logs. The basic command for viewing logs is
journalctl
It will get you output similar to this:
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 18:42:51 CET Nov 13 17:38:07 ThinkPad-X220T systemd-journald[344]: Runtime journal (/run/log/ Nov 13 17:38:07 ThinkPad-X220T kernel: microcode: microcode updated early to rev
That is first line, and followed by many lines. We see that logs begin from last system boot. I last time rebooted the computer yesterday. By default, journald doesn't save logs across reboots, because that would make log files get to big over time. The log seems similar to syslog, but it has many more sources. We said that both UTC and your local timezone timestamps are saved, so you can also look at the same logs with UTC timestamps.
journalctl --utc
Yes, I know that logs are huge. We said that logs will be great again! But they are too great, so the journald has neat feature of filtering logs.
If you want to see logs from some point in time till now, you need to use command formatted like this:
miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:15:00" -- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:07:51 CET Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session opene Nov 14 19:17:01 ThinkPad-X220T CRON[6049]: (root) CMD ( cd / && run-parts --re Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session close Nov 14 19:22:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo ...
2) Filtering logs by time
This is still too big, so we can make them smaller if we add one more operand after --since, and that will be --until
miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:00:00" --until "2016-11-14 19:05:00" -- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:21:23 CET Nov 14 19:01:16 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo Nov 14 19:02:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo
In those five minutes, there were only two log massages.
Since and until flags accept the time stamps in format YYYY-MM-DD HH:MM:SS but that is not only way. If for example get a user complain that your server is not working in last 5 minutes, you can examine logs like this:
journalctl --since "5 minutes ago" -- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:22:21 CET Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox Nov 14 20:21:23 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox Nov 14 20:22:20 ThinkPad-X220T apport-gtk[5977]: gdk_pixbuf_composite: assertion Nov 14 20:22:21 ThinkPad-X220T systemd[3634]: Started Notification regarding a c
It also understands words like yesterday or today so you can look for logs since yesterday or today.
3) Setting up logging across reboots
This this all is more useful if you have the logs persistent across reboots. If you want logs to remain after boots, you need to enable this manually. First create directory where it is going to be logged
sudo mkdir -p /var/log/journal
Then edit the configuration file to log persistently.
sudo sed -i.orig 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf
After this you can reboot your machine and check if boots span across reboots with following command:
miki@ThinkPad-X220T:~$ journalctl --list-boots -1 7917f2c3498d4de89e94701ca8049245 Mon 2016-11-14 20:47:49 CET—Mon 2016-11-14 2 0 438c0a9258aa47fcbf88daf91106aaa3 Mon 2016-11-14 20:56:55 CET—Mon 2016-11-14 2
I rebooted the computer two times and both boots are logged. I thought that previous boot would get saved as well but it did not. So only those reboots that are done after setting persistent logs will be logged.
4) Managing logs from different boots
If you now want to see logs from this boot only, you use command
journalctl -b
You can use boot number to pick specific boot
journalctl -b -1
Or you could use boot id
journalctl -b 7917f2c3498d4de89e94701ca8049245
The time based filtering will also work
journalctl -b 0 --since "15 minutes ago"
5) Filtering by service and id
Say you want to see logs by one program only. I will look logs of fprintd
journalctl -u fprintd.service -- Logs begin at Mon 2016-11-14 20:47:49 CET, end at Mon 2016-11-14 21:28:03 CET Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Starting Fingerprint Authentication D Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Started Fingerprint Authentication Da Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: user 'miki' claiming the device: 0 Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: now monitoring fd 15 Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: device 0 claim status 0 Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: no longer monitoring fd 15 Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: released device 0 -- Reboot --
You can also filter logs by UID. First you need to get UID of your user
miki@ThinkPad-X220T:~$ id -u miki 1000
And next you can use that uid to filter logs
journalctl _UID=1000 --since "25 minutes ago"
You can also filter by GID
miki@ThinkPad-X220T:~$ journalctl -F _GID 30 102 130 1000 114 118 104 126 109 0
Now can filter by any of those GIDs
journalctl _GID=114
Filtering logs by path to to executable:
journalctl /usr/bin/sudo
Seeing logs in real time
journalctl -f
If you want your logs to be displayed in plan text for easy copy and paste, you can use "no pager" option
journalctl --no-pager
6) Displaying kernel messages and deleting old logs
The journalctl can display kernel messages as well. By default, it will display from current boot
journalctl -k
If you want kernel messages from previous boot, you use the -b -1 flag.
journalctl -k -b -1
To check how much disk space your logs use, you can use this command
journalctl --disk-usage Archived and active journals take up 24.0M on disk.
If your logs have grown in size, you can use vacuum cleaner to wipe the oldest one. Well not really a vacuum cleaner, but command is called vacuum.
sudo journalctl --vacuum-size=16M
This will reduce the log size to 16MB.
Conclusion
We have gone through basic journald commands that make logging great again. This will hopefully help you greatly when you next time have to troubleshoot some sticky problem with your Linux server or desktop computer with systemd. The systemd is brings us great tools and negative publicity that it got early was just backlash against changing old tools that people were used too. I already got used to systemd and I don't know how I lived without it. This is all for this article, thank you for reading and have a nice day.
