Sar command is used to produce statistical reports on a variety of resources including CPU usage, memory utilization, I/O devices consumption, network monitor, swap, and load averages.
SAR stands for System Activity Reporter, very useful tools for any Linux administrator to monitor the system performance and to investigate the bottlenecks. This tool is provided by the sysstat package.
In this tutorial, we will go through how to install sysstat package and use sar command on Linux.
Install sysstat package
The sysstat package contains many utilities which are a collection of performance monitoring tools for Linux system. Sar is a tool which belong to sysstat package.
Installing sysstat package on RHEL and CentOS
$ sudo yum install sysstatOnce the sysstat has been installed successfully, verifying the installation by command:
$ which sar
/usr/bin/sarBy default, sysstat will store the server statistics in the file '/var/log/sa/saDD' (DD is day of month, for example: /var/log/sa/sa08) and will keep the data for 28 days. If you want to keep the data more that, you need to modify the sysstat configuration file '/etc/sysconfig/sysstat'.
$ sudo cat /etc/sysconfig/sysstat
HISTORY=28Modify the value of HISTORY with the number of days you want to keep the logs.
Installing sysstat package on Ubuntu and Debian
$ sudo apt install sysstatVerifying the installation by command:
$ which sar
/usr/bin/sarOn Ubuntu, we need to enable it while CentOS is not needed.
Using your favorite editor (vim, nano or gedit) to edit the following file:
$ sudo nano /etc/default/sysstatAnd set ENABLED to true:
ENABLED="true"Restart the service
$ sudo service sysstat restartOn Ubuntu, the configuration file is '/etc/sysstat/sysstat' and the data will be kept in 7 days by default.
$ cat /etc/sysstat/sysstat
HISTORY=7From Fedora 21, sysstat comes by default and all you need to do is running below commands:
$ sudo systemctl enable sysstat.service
$ sudo systemctl start sysstat.serviceThere is a cron job for sysstat which runs in every 10 minutes to collect data and will store it in the corresponding file /var/log/sa/saDD. If you want to change the interval of this cron run, you can modify it in '/etc/cron.d/sysstat'.
For systemd, the timer unit file calls sysstat-collect.service (/usr/lib/systemd/system/sysstat-collect.service) every 10 minutes to collect the stats.
$ cat /etc/cron.d/sysstat
# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2Sar command usage
The basic syntax for sar command usage is as follows:
sar [option] [-o filename] [-f filename] [interval] [count]Where:
interval: the period in seconds.
count: number of times to make output statistics.
1. Find CPU activity
By default (with no other options given) sar command will report the CPU activity of the server. Also, the option -u can be used to get the CPU utilization report.
If you want to get the CPU activity report in every 2 seconds for 3 times, you can use the following sar command:
$ sar -u 2 3
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 09/06/2020 _x86_64_ (2 CPU)
22:26:54 CPU %user %nice %system %iowait %steal %idle
22:26:56 all 4,03 0,00 0,50 0,00 0,00 95,47
22:26:58 all 8,08 0,00 0,51 0,00 0,00 91,41
22:27:00 all 12,50 0,00 1,00 0,00 0,00 86,50
Average: all 8,21 0,00 0,67 0,00 0,00 91,11
Where:
%user denotes the percentage of CPU utilization that occurred while executing at the user level (user space)
%system denotes the percentage of CPU utilization that occurred while executing at the system level (kernel space)
%idle is the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
If you want to get the output in a file in binary form, you need to append -o filename with the sar command. For example:
$ sar -u 2 3 -o log-fileAlso, you can get the CPU activity report of a particular day, say: June 09th, 2020. You can find it as:
$ sudo sar -u -f /var/log/sa/sa092. Find CPU usage for each processor
The -u option provides the CPU usage on all processors. If you want to find the CPU activity on all processors separately, you need to use the -P option.
$ sar -P ALL 1 1
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
05:38:18 CPU %user %nice %system %iowait %steal %idle
05:38:19 all 3,03 0,00 0,00 0,00 0,00 96,97
05:38:19 0 3,96 0,00 0,99 0,00 0,00 95,05
05:38:19 1 3,00 0,00 0,00 0,00 0,00 97,00
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 3,03 0,00 0,00 0,00 0,00 96,97
Average: 0 3,96 0,00 0,99 0,00 0,00 95,05
Average: 1 3,00 0,00 0,00 0,00 0,00 97,00
3. Find Memory usage
You can find the memory usage (used and free memory of the server) over time using the -r switch.
$ sar -r 1 3
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
05:41:04 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
05:41:05 855300 3025220 77,96 0 1495276 2426864 30,55 1265008 1158172 100
05:41:06 854800 3025720 77,97 0 1495276 2426864 30,55 1265132 1158172 100
05:41:07 855032 3025488 77,97 0 1495276 2426864 30,55 1265012 1158172 100
Average: 855044 3025476 77,97 0 1495276 2426864 30,55 1265051 1158172 100Where:
kbmemfree indicates the amount of free memory available in kilobytes.
kbmemused indicates the amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.
If you want to gather the memory usage for a particular day say: Sept 05, you need to run it as below command:
$ sar -r -f /var/log/sa/sa05
4. Find Swapping activity
You can check the swap usage of the machine using the -W option.
$ sar -W 1 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:31:12 PM pswpin/s pswpout/s
03:31:13 PM 16.16 0.00
03:31:14 PM 16.00 0.00
03:31:15 PM 15.84 0.00
Average: 16.00 0.00Where:
pswpin/s indicates the total number of swap pages the system brought in per second.
pswpout/s indicates the total number of swap pages the system brought out per second.
If you want to gather the swap usages for a particular day say Sept 05, you need to run it as following sar command:
$ sar -W -f /var/log/sa/sa055. Find load averages over time
You can find the load averages overtime using the -q option.
$ sar -q 1 3
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
06:16:13 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
06:16:14 0 329 0,00 0,04 0,09 0
06:16:15 0 329 0,00 0,04 0,09 0
06:16:16 0 329 0,00 0,04 0,09 0
Average: 0 329 0,00 0,04 0,09 0Where:
runq-sz: It shows the run queue length (number of tasks waiting for run time).
plist-sz: It shows the number of tasks in the task list.
ldavg-1/5/15: System load average for the last minute/ past 5 minutes/ past 15 minutes.
If you want to gather the load averages for a particular day, say Sept 05, you need to run it following sar command:
$ sar -q -f /var/log/sa/sa056. Report of mounted filesystems
You can get the statistics for currently filesystems using the -F option
$ sar -F 2 3
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
06:30:18 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM
06:30:20 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root
06:30:20 796 218 21,52 21,52 523947 341 0,07 /dev/sda1
06:30:20 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM
06:30:22 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root
06:30:22 796 218 21,52 21,52 523947 341 0,07 /dev/sda1
06:30:22 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM
06:30:24 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root
06:30:24 796 218 21,52 21,52 523947 341 0,07 /dev/sda1
Summary: MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM
Summary 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root
Summary 796 218 21,52 21,52 523947 341 0,07 /dev/sda1If you want to collect the mounted filesystems for a particular day, say June 10th.
$ sar -F -f /var/log/sa/sa107. Report details of inode, kernel tables and file tables
You can get the details of inode, processes, kernel threads and file tables used by the system by using sar command with -v option.
$ sar -v 2 3
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
06:57:23 dentunusd file-nr inode-nr pty-nr
06:57:25 160833 2400 132081 3
06:57:27 160833 2400 132081 3
06:57:29 160833 2400 132081 3
Average: 160833 2400 132081 3Where:
dentunusd: number of unused cache entries in the directory cache
file-nr: numbers of in-use file handles
inode-nr: inode handlers
pty-nr: pseudo terminals
8. Check CPU bottleneck
You can use sar command to find out which interrupt number possible causing CPU bottleneck.
In the below example #9 was the highest excluding system interrupt #0.
$ sar -I XALL 2 10
02:07:10 AM INTR intr/s
02:07:12 AM 0 992.57
02:07:12 AM 1 0.00
02:07:12 AM 2 0.00
02:07:12 AM 3 0.00
02:07:12 AM 4 0.00
02:07:12 AM 5 0.00
02:07:12 AM 6 0.00
02:07:12 AM 7 0.00
02:07:12 AM 8 0.00
02:07:12 AM 9 350.50Details in '/proc/interrupts' file will also find helpful for you. Interrupt halts processing momentarily so that I/O or other operations can occur. Processing resumes after the specific operation takes place. So it is very important that each device installed in machine must be provided with an interrupt setting that does not conflict with the settings used by the hardware and other peripherals.
$ sudo cat /proc/interrupts
CPU0 CPU1
0: 48 0 IO-APIC-edge timer
1: 54 0 IO-APIC-edge i8042
8: 1 0 IO-APIC-edge rtc0
9: 0 0 IO-APIC-fasteoi acpi
12: 35 116 IO-APIC-edge i8042
14: 0 0 IO-APIC-edge ata_piix
15: 0 0 IO-APIC-edge ata_piix
16: 118 2513 IO-APIC-fasteoi vmwgfx
24: 0 0 PCI-MSI-edge PCIe PME, pciehp
25: 0 0 PCI-MSI-edge PCIe PME, pciehp
26: 0 0 PCI-MSI-edge PCIe PME, pciehp
27: 0 0 PCI-MSI-edge PCIe PME, pciehp
NMI: 0 0 Non-maskable interrupts
LOC: 35392807 14792833 Local timer interrupts
SPU: 0 0 Spurious interrupts
PMI: 0 0 Performance monitoring interrupts
IWI: 2677624 215297 IRQ work interrupts
...The first column refers to the IRQ number. The next column reports the type of interrupt and the last column contains the name of the device that is located to IRQ.
9. Report network statistics
In order to get the report of the network statistics, using sar command with -n option. For example:
$ sar -n DEV
Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU)
19:20:23 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
19:20:24 ens192 11,00 4,00 1,80 2,75 0,00 0,00 0,00
19:20:24 lo 79,00 79,00 43,62 43,62 0,00 0,00 0,00
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
Average: ens192 11,00 4,00 1,80 2,75 0,00 0,00 0,00
Average: lo 79,00 79,00 43,62 43,62 0,00 0,00 0,00Where:
DEV: statistics from the network devices are reported
IFACE: name of the interface (In the above example: ens192)
rxpck/s / txpck/s: total number of packets received / transmitted per second
There are some other options than DEV, such as:
NFS: show NFS client activities
SOCK: indicate the sockets in use
TCP: indicate TCP v4 network traffic
UDP: indicate UDP v4 network traffic
If you want to gather the network statistics for a particular day say June 09, you need to run it as follows:
$ sar -n DEV -f /var/log/sa/sa09Sar graph
The above sections explained the way to use sar from the terminal. kSar is a java-based GUI application that visualizes your sar data.
Installing ksar:
$ sudo apt install openjdk-11-jdk$ sudo yum install epel-release
$ sudo yum install java-openjdkDownload ksar and unpack it:
$ wget https://sourceforge.net/projects/ksar/files/latest/download
$ mv download kSar.zip
$ unzip kSar.zipNow cd into dir
$ cd kSar-*Now make run.sh file executable and run it as root.
$ chmod +x run.sh
$ sudo su
./run.shNow we can test by running local command. We go to Data > Run Local Command and there we can use default command.
Next, we can look at the graphs by clicking on the left panel and looking in the right side the displayed graph:
It is also possible to load graphs from file. First, we need to run sar:
LC_ALL=C sar -A > /tmp/sar.data.txtAnd then in menus, we select Data > Load from text file and find the file in /tmp or wherever we have saved it.
Conclusion
In this tutorial, we learned how sar command can be used to help you notice the usage of system resources and improve performance while working with your operating system.
From sar examples, it's clear that it is a great tool to collect, report, or save system activity information.
Thanks for reading and please leave your suggestion in the below comment section.
Related Read: 10 Linux iostat Command to Report CPU and I/O Statistics
Great tutorial. To analyze sar report, follow https://github.com/sumitbiswasgit/sarbox
Hi Sumit,
Nice tool.