Despite some of the negative publicity, systemd have brought great new functionality to the Linux operating system. Logging is one of them. In the old days before systemd, checking logs and troubleshooting issues was sometimes very unpleasant thing to do because logs for different programs were scattered in different files and you had to check timestamps in each and every one of them. Of course there was syslog file also, but sometimes was not as verbose as needed. The systemd brings us journal daemon called journald which enables centralized management of logs. This logs can be filtered and manipulated by utility called journalctl. In this article we will go through basic usage of utilities.
What is journald
The journald is service that collects and stores logs from many sources, and make indexed structured log files that are easy to interpret. Some of the sources that journald uses
- Kernel log messages, via kmsg
- Simple system log messages, via the libc syslog call
- Structured system log messages via the native Journal API
- Standard output and standard error of system services
Idea is to have all log messages centralized, no matter from which application they are coming from.
Setting the time
In order for logs to be useful, you must have correct system time. Logs can have time by UTC or by local time. Generally both time-stamps are saved. So lets set up timezone first
sudo timedatectl set-timezone Europe/BelgradeThis command sets the time to Central European timezone, but if you are on any other spot of the globe, you just need to enter continent and press tab to see cities you might chose. It doesn't matter if it is not your city, I am not living in Belgrade either, it is just closest big city. Next lets see the status of the timezone
miki@ThinkPad-X220T:~$ timedatectl status
Local time: Mon 2016-11-14 18:40:46 CET
Universal time: Mon 2016-11-14 17:40:46 UTC
RTC time: Mon 2016-11-14 17:40:46
Time zone: Europe/Belgrade (CET, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no1) Viewing logs
Now that we have correct time, lets continue to viewing logs. The basic command for viewing logs is
journalctlIt will get you output similar to this:
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 18:42:51 CET
Nov 13 17:38:07 ThinkPad-X220T systemd-journald[344]: Runtime journal (/run/log/
Nov 13 17:38:07 ThinkPad-X220T kernel: microcode: microcode updated early to revThat is first line, and followed by many lines. We see that logs begin from last system boot. I last time rebooted the computer yesterday. By default, journald doesn't save logs across reboots, because that would make log files get to big over time. The log seems similar to syslog, but it has many more sources. We said that both UTC and your local timezone timestamps are saved, so you can also look at the same logs with UTC timestamps.
journalctl --utcYes, I know that logs are huge. We said that logs will be great again! But they are too great, so the journald has neat feature of filtering logs.
If you want to see logs from some point in time till now, you need to use command formatted like this:
miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:15:00"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:07:51 CET
Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session opene
Nov 14 19:17:01 ThinkPad-X220T CRON[6049]: (root) CMD ( cd / && run-parts --re
Nov 14 19:17:01 ThinkPad-X220T CRON[6044]: pam_unix(cron:session): session close
Nov 14 19:22:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo
...2) Filtering logs by time
This is still too big, so we can make them smaller if we add one more operand after --since, and that will be --until
miki@ThinkPad-X220T:~$ journalctl --since "2016-11-14 19:00:00" --until "2016-11-14 19:05:00"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:21:23 CET
Nov 14 19:01:16 ThinkPad-X220T nm-applet[4929]: ModemManager is not available fo
Nov 14 19:02:51 ThinkPad-X220T nm-applet[4929]: ModemManager is not available foIn those five minutes, there were only two log massages.
Since and until flags accept the time stamps in format YYYY-MM-DD HH:MM:SS but that is not only way. If for example get a user complain that your server is not working in last 5 minutes, you can examine logs like this:
journalctl --since "5 minutes ago"
-- Logs begin at Sun 2016-11-13 17:38:07 CET, end at Mon 2016-11-14 20:22:21 CET
Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:21:21 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:21:23 ThinkPad-X220T gnome-terminal-[7867]: Allocating size to GtkBox
Nov 14 20:22:20 ThinkPad-X220T apport-gtk[5977]: gdk_pixbuf_composite: assertion
Nov 14 20:22:21 ThinkPad-X220T systemd[3634]: Started Notification regarding a cIt also understands words like yesterday or today so you can look for logs since yesterday or today.
3) Setting up logging across reboots
This this all is more useful if you have the logs persistent across reboots. If you want logs to remain after boots, you need to enable this manually. First create directory where it is going to be logged
sudo mkdir -p /var/log/journalThen edit the configuration file to log persistently.
sudo sed -i.orig 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.confAfter this you can reboot your machine and check if boots span across reboots with following command:
miki@ThinkPad-X220T:~$ journalctl --list-boots
-1 7917f2c3498d4de89e94701ca8049245 Mon 2016-11-14 20:47:49 CET—Mon 2016-11-14 2
0 438c0a9258aa47fcbf88daf91106aaa3 Mon 2016-11-14 20:56:55 CET—Mon 2016-11-14 2
I rebooted the computer two times and both boots are logged. I thought that previous boot would get saved as well but it did not. So only those reboots that are done after setting persistent logs will be logged.
4) Managing logs from different boots
If you now want to see logs from this boot only, you use command
journalctl -bYou can use boot number to pick specific boot
journalctl -b -1Or you could use boot id
journalctl -b 7917f2c3498d4de89e94701ca8049245The time based filtering will also work
journalctl -b 0 --since "15 minutes ago"5) Filtering by service and id
Say you want to see logs by one program only. I will look logs of fprintd
journalctl -u fprintd.service
-- Logs begin at Mon 2016-11-14 20:47:49 CET, end at Mon 2016-11-14 21:28:03 CET
Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Starting Fingerprint Authentication D
Nov 14 20:48:32 ThinkPad-X220T systemd[1]: Started Fingerprint Authentication Da
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: user 'miki' claiming the device: 0
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: now monitoring fd 15
Nov 14 20:48:32 ThinkPad-X220T fprintd[3413]: device 0 claim status 0
Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: no longer monitoring fd 15
Nov 14 20:48:33 ThinkPad-X220T fprintd[3413]: released device 0
-- Reboot --You can also filter logs by UID. First you need to get UID of your user
miki@ThinkPad-X220T:~$ id -u miki
1000And next you can use that uid to filter logs
journalctl _UID=1000 --since "25 minutes ago"You can also filter by GID
miki@ThinkPad-X220T:~$ journalctl -F _GID
30
102
130
1000
114
118
104
126
109
0Now can filter by any of those GIDs
journalctl _GID=114Filtering logs by path to to executable:
journalctl /usr/bin/sudoSeeing logs in real time
journalctl -fIf you want your logs to be displayed in plan text for easy copy and paste, you can use "no pager" option
journalctl --no-pager6) Displaying kernel messages and deleting old logs
The journalctl can display kernel messages as well. By default, it will display from current boot
journalctl -kIf you want kernel messages from previous boot, you use the -b -1 flag.
journalctl -k -b -1To check how much disk space your logs use, you can use this command
journalctl --disk-usage
Archived and active journals take up 24.0M on disk.If your logs have grown in size, you can use vacuum cleaner to wipe the oldest one. Well not really a vacuum cleaner, but command is called vacuum.
sudo journalctl --vacuum-size=16MThis will reduce the log size to 16MB.
Conclusion
We have gone through basic journald commands that make logging great again. This will hopefully help you greatly when you next time have to troubleshoot some sticky problem with your Linux server or desktop computer with systemd. The systemd is brings us great tools and negative publicity that it got early was just backlash against changing old tools that people were used too. I already got used to systemd and I don't know how I lived without it. This is all for this article, thank you for reading and have a nice day.
