Sar Command in Linux with Examples

Sar command is used to produce statistical reports on a variety of resources including CPU usage, memory utilization, I/O devices consumption, network monitor, swap, and load averages.

SAR stands for System Activity Reporter, very useful tools for any Linux administrator to monitor the system performance and to investigate the bottlenecks. This tool is provided by the sysstat package.

In this tutorial, we will go through how to install sysstat package and use sar command on Linux.

Install sysstat package

The sysstat package contains many utilities which are a collection of performance monitoring tools for Linux system. Sar is a tool which belong to sysstat package.

Installing sysstat package on RHEL and CentOS

$ sudo yum install sysstat

Once the sysstat has been installed successfully, verifying the installation by command:

$ which sar
/usr/bin/sar

By default, sysstat will store the server statistics in the file '/var/log/sa/saDD' (DD is day of month, for example: /var/log/sa/sa08) and will keep the data for 28 days. If you want to keep the data more that, you need to modify the sysstat configuration file '/etc/sysconfig/sysstat'.

$ sudo cat /etc/sysconfig/sysstat

HISTORY=28

Modify the value of HISTORY with the number of days you want to keep the logs.

Installing sysstat package on Ubuntu and Debian

$ sudo apt install sysstat

Verifying the installation by command:

$ which sar
/usr/bin/sar

On Ubuntu, we need to enable it while CentOS is not needed.

Using your favorite editor (vim, nano or gedit) to edit the following file:

$ sudo nano /etc/default/sysstat

And set ENABLED to true:

ENABLED="true"

Restart the service

$ sudo service sysstat restart

On Ubuntu, the configuration file is '/etc/sysstat/sysstat'  and the data will be kept in 7 days by default.

$ cat /etc/sysstat/sysstat

HISTORY=7

From Fedora 21, sysstat comes by default and all you need to do is running below commands:

$ sudo systemctl enable sysstat.service
$ sudo systemctl start sysstat.service

There is a cron job for sysstat which runs in every 10 minutes to collect data and will store it in the corresponding file /var/log/sa/saDD. If you want to change the interval of this cron run, you can modify it in '/etc/cron.d/sysstat'.

For systemd, the timer unit file calls sysstat-collect.service (/usr/lib/systemd/system/sysstat-collect.service) every 10 minutes to collect the stats.

$ cat /etc/cron.d/sysstat

# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

Sar command usage

The basic syntax for sar command usage is as follows:

sar [option] [-o filename] [-f filename] [interval] [count]

Where:

interval: the period in seconds.

count: number of times to make output statistics.

1. Find CPU activity

By default (with no other options given) sar command will report the CPU activity of the server. Also, the option -u can be used to get the CPU utilization report.

If you want to get the CPU activity report in every 2 seconds for 3 times, you can use the following sar command:

$ sar -u 2 3

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	09/06/2020 	_x86_64_	(2 CPU)

22:26:54        CPU     %user     %nice   %system   %iowait    %steal     %idle
22:26:56        all      4,03      0,00      0,50      0,00      0,00     95,47
22:26:58        all      8,08      0,00      0,51      0,00      0,00     91,41
22:27:00        all     12,50      0,00      1,00      0,00      0,00     86,50
Average:        all      8,21      0,00      0,67      0,00      0,00     91,11

Where:

%user denotes the percentage of CPU utilization that occurred while executing at the user level (user space)

%system denotes the percentage of CPU utilization that occurred while executing at the system level (kernel space)

%idle is the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

If you want to get the output in a file in binary form, you need to append -o filename with the sar command. For example:

$ sar -u 2 3 -o log-file

Also, you can get the CPU activity report of a particular day, say: June 09th, 2020. You can find it as:

$ sudo sar -u -f /var/log/sa/sa09

2. Find CPU usage for each processor

The -u option provides the CPU usage on all processors. If you want to find the CPU activity on all processors separately, you need to use the -P option.

$ sar -P ALL 1 1

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

05:38:18        CPU     %user     %nice   %system   %iowait    %steal     %idle
05:38:19        all      3,03      0,00      0,00      0,00      0,00     96,97
05:38:19          0      3,96      0,00      0,99      0,00      0,00     95,05
05:38:19          1      3,00      0,00      0,00      0,00      0,00     97,00

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      3,03      0,00      0,00      0,00      0,00     96,97
Average:          0      3,96      0,00      0,99      0,00      0,00     95,05
Average:          1      3,00      0,00      0,00      0,00      0,00     97,00

3. Find Memory usage

You can find the memory usage (used and free memory of the server) over time using the -r switch.

$ sar -r 1 3

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

05:41:04    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
05:41:05       855300   3025220     77,96         0   1495276   2426864     30,55   1265008   1158172       100
05:41:06       854800   3025720     77,97         0   1495276   2426864     30,55   1265132   1158172       100
05:41:07       855032   3025488     77,97         0   1495276   2426864     30,55   1265012   1158172       100
Average:       855044   3025476     77,97         0   1495276   2426864     30,55   1265051   1158172       100

Where:

kbmemfree indicates the amount of free memory available in kilobytes.

kbmemused indicates the amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.

If you want to gather the memory usage for a particular day say: Sept 05, you need to run it as below command:

$ sar -r -f /var/log/sa/sa05

4. Find Swapping activity

You can check the swap usage of the machine using the -W option.

$ sar -W 1 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:31:12 PM pswpin/s pswpout/s
03:31:13 PM 16.16 0.00
03:31:14 PM 16.00 0.00
03:31:15 PM 15.84 0.00
Average: 16.00 0.00

Where:

pswpin/s indicates the total number of swap pages the system brought in per second.

pswpout/s indicates the total number of swap pages the system brought out per second.

If you want to gather the swap usages for a particular day say Sept 05, you need to run it as following sar command:

$ sar -W -f /var/log/sa/sa05

5. Find load averages over time

You can find the load averages overtime using the -q option.

$ sar -q 1 3

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

06:16:13      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
06:16:14            0       329      0,00      0,04      0,09         0
06:16:15            0       329      0,00      0,04      0,09         0
06:16:16            0       329      0,00      0,04      0,09         0
Average:            0       329      0,00      0,04      0,09         0

Where:

runq-sz: It shows the run queue length (number of tasks waiting for run time).

plist-sz: It shows the number of tasks in the task list.

ldavg-1/5/15: System load average for the last minute/ past 5 minutes/ past 15 minutes.

If you want to gather the load averages for a particular day, say Sept 05, you need to run it following sar command:

$ sar -q -f /var/log/sa/sa05

6. Report of mounted filesystems

You can get the statistics for currently filesystems using the -F option

$ sar -F 2 3

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

06:30:18     MBfsfree  MBfsused   %fsused  %ufsused     Ifree     Iused    %Iused FILESYSTEM
06:30:20        30410      5532     15,39     15,39  18245113    166407      0,90 /dev/mapper/centos-root
06:30:20          796       218     21,52     21,52    523947       341      0,07 /dev/sda1

06:30:20     MBfsfree  MBfsused   %fsused  %ufsused     Ifree     Iused    %Iused FILESYSTEM
06:30:22        30410      5532     15,39     15,39  18245113    166407      0,90 /dev/mapper/centos-root
06:30:22          796       218     21,52     21,52    523947       341      0,07 /dev/sda1

06:30:22     MBfsfree  MBfsused   %fsused  %ufsused     Ifree     Iused    %Iused FILESYSTEM
06:30:24        30410      5532     15,39     15,39  18245113    166407      0,90 /dev/mapper/centos-root
06:30:24          796       218     21,52     21,52    523947       341      0,07 /dev/sda1

Summary:     MBfsfree  MBfsused   %fsused  %ufsused     Ifree     Iused    %Iused FILESYSTEM
Summary         30410      5532     15,39     15,39  18245113    166407      0,90 /dev/mapper/centos-root
Summary           796       218     21,52     21,52    523947       341      0,07 /dev/sda1

If you want to collect the mounted filesystems for a particular day, say June 10th.

$ sar -F -f /var/log/sa/sa10

7. Report details of inode, kernel tables and file tables

You can get the details of inode, processes, kernel threads and file tables used by the system by using sar command with -v option.

$ sar -v 2 3

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

06:57:23    dentunusd   file-nr  inode-nr    pty-nr
06:57:25       160833      2400    132081         3
06:57:27       160833      2400    132081         3
06:57:29       160833      2400    132081         3
Average:       160833      2400    132081         3

Where:

dentunusd: number of unused cache entries in the directory cache

file-nr: numbers of in-use file handles

inode-nr: inode handlers

pty-nr: pseudo terminals

8. Check CPU bottleneck

You can use sar command to find out which interrupt number possible causing CPU bottleneck.

In the below example #9 was the highest excluding system interrupt #0.

$ sar -I XALL 2 10
02:07:10 AM INTR intr/s
02:07:12 AM 0 992.57
02:07:12 AM 1 0.00
02:07:12 AM 2 0.00
02:07:12 AM 3 0.00
02:07:12 AM 4 0.00
02:07:12 AM 5 0.00
02:07:12 AM 6 0.00
02:07:12 AM 7 0.00
02:07:12 AM 8 0.00
02:07:12 AM 9 350.50

Details in '/proc/interrupts' file will also find helpful for you. Interrupt halts processing momentarily so that I/O or other operations can occur. Processing resumes after the specific operation takes place. So it is very important that each device installed in machine must be provided with an interrupt setting that does not conflict with the settings used by the hardware and other peripherals.

$ sudo cat /proc/interrupts

           CPU0       CPU1       
  0:         48          0   IO-APIC-edge      timer
  1:         54          0   IO-APIC-edge      i8042
  8:          1          0   IO-APIC-edge      rtc0
  9:          0          0   IO-APIC-fasteoi   acpi
 12:         35        116   IO-APIC-edge      i8042
 14:          0          0   IO-APIC-edge      ata_piix
 15:          0          0   IO-APIC-edge      ata_piix
 16:        118       2513   IO-APIC-fasteoi   vmwgfx
 24:          0          0   PCI-MSI-edge      PCIe PME, pciehp
 25:          0          0   PCI-MSI-edge      PCIe PME, pciehp
 26:          0          0   PCI-MSI-edge      PCIe PME, pciehp
 27:          0          0   PCI-MSI-edge      PCIe PME, pciehp
NMI:          0          0   Non-maskable interrupts
LOC:   35392807   14792833   Local timer interrupts
SPU:          0          0   Spurious interrupts
PMI:          0          0   Performance monitoring interrupts
IWI:    2677624     215297   IRQ work interrupts
...

The first column refers to the IRQ number. The next column reports the type of interrupt and the last column contains the name of the device that is located to IRQ.

9. Report network statistics

In order to get the report of the network statistics, using sar command with -n option. For example:

$ sar -n DEV

Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 	10/06/2020 	_x86_64_	(2 CPU)

19:20:23        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
19:20:24       ens192     11,00      4,00      1,80      2,75      0,00      0,00      0,00
19:20:24           lo     79,00     79,00     43,62     43,62      0,00      0,00      0,00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:       ens192     11,00      4,00      1,80      2,75      0,00      0,00      0,00
Average:           lo     79,00     79,00     43,62     43,62      0,00      0,00      0,00

Where:

DEV: statistics from the network devices are reported

IFACE: name of the interface (In the above example: ens192)

rxpck/s / txpck/s: total number of packets received / transmitted per second

There are some other options than DEV, such as:

NFS: show NFS client activities

SOCK: indicate the sockets in use

TCP: indicate TCP v4 network traffic

UDP: indicate UDP v4 network traffic

If you want to gather the network statistics for a particular day say June 09, you need to run it as follows:

$ sar -n DEV -f /var/log/sa/sa09

Sar graph

The above sections explained the way to use sar from the terminal. kSar is a java-based GUI application that visualizes your sar data.

Installing ksar:

$ sudo apt install openjdk-11-jdk

$ sudo yum install epel-release
$ sudo yum install java-openjdk

Download ksar and unpack it:

$ wget https://sourceforge.net/projects/ksar/files/latest/download
$ mv download kSar.zip
$ unzip kSar.zip

Now cd into dir

$ cd kSar-*

Now make run.sh file executable and run it as root.

$ chmod +x run.sh
$ sudo su
./run.sh

Now we can test by running local command. We go to Data > Run Local Command and there we can use default command.

Next, we can look at the graphs by clicking on the left panel and looking in the right side the displayed graph:

It is also possible to load graphs from file. First, we need to run sar:

LC_ALL=C sar -A > /tmp/sar.data.txt

And then in menus, we select Data > Load from text file and find the file in /tmp or wherever we have saved it.

Conclusion

In this tutorial, we learned how sar command can be used to help you notice the usage of system resources and improve performance while working with your operating system.

From sar examples, it's clear that it is a great tool to collect, report, or save system activity information.

Thanks for reading and please leave your suggestion in the below comment section.

Related Read: 10 Linux iostat Command to Report CPU and I/O Statistics