How to Monitor Linux Performance Using Sar Command

sar command linux

Sar command is one of the very useful tools for any Linux administrator to monitor the system performance and to investigate the bottlenecks. This tool is provided by the sysstat package and can be used to find a variety of resource usages including CPU, memory, swap, load averages over time.

In this article, we will go through the various ways of ‘sar’ command usages. This command collects the system data and stores in a file “/var/log/sa/sadd” (where dd is the corresponding day of month). The file contains binary data and hence you won’t be able to directly read the file. You can read the data from ‘/var/log/sa/sadd’ file using the ‘sar’ command.

Installing Sysstat utility

On Centos

You can get the ‘sar’ command, by installing the sysstat package. You can install it using the yum package manager as follows.

$ sudo yum install sysstat

By default, sysstat will store the server statistics in the file ‘/var/log/sa/sadd’ and will keep the data for 7 days. If you want to keep the data more that, you need to modify the sysstat configuration file ‘/etc/sysconfig/sysstat’.

# cat /etc/sysconfig/sysstat
# How long to keep log files (days), maximum is a month
HISTORY=7

Modify the value of “HISTORY” with the number of days you want to keep the logs.

On Ubuntu

Installing the command line tool sar is easy, it is part of the package sysstat. So in Ubuntu type following:

$ sudo apt install sysstat

Next, we need to enable it.

Edit following file on Ubuntu. On centos is not needed.

$ sudo nano /etc/default/sysstat

And set ENABLED to true

ENABLED="true"

On Fedora

From Fedora 21 sysstat comes by default and all you need to do is run below commands

$sudo systemctl enable sysstat.service
$sudo systemctl start sysstat.service

There is a cron job for sysstat which runs in every 10 minutes to collect data and will store it in the corresponding file (/var/log/sa/sadd). If you want to change the interval of this cron run, you can modify it in “/etc/cron.d/sysstat”. For systemd, the timer unit file calls sysstat-collect.service (/usr/lib/systemd/system/sysstat-collect.service) every 10 minutes to collect the stats.

# cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 –A

sar command usage

The basic syntax for ‘sar’ command usage is as follows.

sar [option] [-o filename] [-f filename] [interval] [count]

1) To find CPU activity

By default (with no other options given) sar command will report the CPU activity of the server. Also, the option ‘-u’ can be used to get the CPU utilization report. If you want to get the CPU activity report in every 2 seconds for 3 times, you can use the following command.

# sar -u 2 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:15:45 PM CPU %user %nice %system %iowait %steal %idle
03:15:47 PM all 7.75 0.00 0.81 0.44 0.00 91.01
03:15:49 PM all 12.23 0.00 3.81 5.49 0.00 78.48
03:15:51 PM all 1.19 0.00 0.56 0.31 0.00 97.94
Average: all 7.06 0.00 1.73 2.08 0.00 89.13

%user denotes percentage of CPU utilization that occurred while executing at the user level (application) and %system denotes percentage of CPU utilization that occurred while executing at the system level (kernel). %idle is the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

If you want to get the output in a file, you need to append –o filename with the command.
Also, you can get the CPU activity report of a particular day, say 09/05/2012, you can find it as,

sar –u –f /var/log/sa/sa05

2) CPU usage by a particular CPU

The –u switch provides the CPU usage on all processors. If you want to find the CPU activity on all processors separately, you need to use the –P switch.

[email protected] [~]# sar -P ALL 1 1
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:22:52 PM CPU %user %nice %system %iowait %steal %idle
03:22:53 PM all 0.88 0.00 0.50 0.75 0.00 97.88
03:22:53 PM 0 0.00 0.00 0.00 0.00 0.00 100.00
03:22:53 PM 1 0.00 0.00 0.00 0.00 0.00 100.00
03:22:53 PM 2 0.99 0.00 0.00 0.99 0.00 98.02
03:22:53 PM 3 1.00 0.00 1.00 2.00 0.00 96.00
……

3) To find Memory usage

You can find the memory usage (used and free memory of the server) over time using the ‘-r’ switch.

# sar -r 1 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:26:28 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
03:26:29 PM 472260 3666564 88.59 440624 1976800 1565920 530520 25.31 276336
03:26:30 PM 471268 3667556 88.61 440648 1976560 1566100 530340 25.30 276240
03:26:31 PM 470524 3668300 88.63 440672 1976824 1566212 530228 25.29 276216
Average: 471351 3667473 88.61 440648 1976728 1566077 530363 25.30 276264

Kbmemfree shows the amount of free memory available in kilobytes and kbmemused shows the amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.

If you want to gather the memory usage for a particular day say sep 05, you need to run it as,

sar –r –f /var/log/sa/sa05

4) To find Swapping activity

You can check the swapping statistics of the machine using the “-W” option.

# sar -W 1 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012
03:31:12 PM pswpin/s pswpout/s
03:31:13 PM 16.16 0.00
03:31:14 PM 16.00 0.00
03:31:15 PM 15.84 0.00
Average: 16.00 0.00

pswpin/s denotes the total number of swap pages the system brought in per second and pswpout/s shows the total number of swap pages the system brought out per second.

If you want to gather the swap usages for a particular day say sep 05, you need to run it as,

sar –W –f /var/log/sa/sa05

5) To find load averages over time

You can find the load averages overtime using the –q switch.

sar -q 1 3
Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012

03:35:49 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
03:35:50 PM 0 587 0.70 0.85 0.94
03:35:51 PM 0 587 0.70 0.85 0.94
03:35:52 PM 0 587 0.70 0.85 0.94
Average: 0 587 0.70 0.85 0.94

runq-sz : It showsrRun queue length (number of tasks waiting for run time).
plist-sz : It shows the number of tasks in the task list.
ldavg-1/5/15 : System load average for the last minute/ pst 5 minutes/past 15 minutes.

If you want to gather the load averages for a particular day say sep 05, you need to run it as,

sar –q –f /var/log/sa/sa05

6) Check CPU bottleneck

You can use sar command to find out which interrupt number possible causing CPU bottleneck. In the below example #9 was the highest excluding system interrupt #0.

# sar -I XALL 2 10
02:07:10 AM INTR intr/s
02:07:12 AM 0 992.57
02:07:12 AM 1 0.00
02:07:12 AM 2 0.00
02:07:12 AM 3 0.00
02:07:12 AM 4 0.00
02:07:12 AM 5 0.00
02:07:12 AM 6 0.00
02:07:12 AM 7 0.00
02:07:12 AM 8 0.00
02:07:12 AM 9 350.50

Details in /proc/interrupts file will also find helpful for you. Interrupt halts processing momentarily so that I/O or other operations can occur. Processing resumes after the specific operation takes place. So it is very important that each device installed in machine must be provided with an interrupt setting that does not conflict with the settings used by the hardware and other peripherals.

# cat /proc/interrupts
CPU0
0: 3882853 XT-PIC timer
1: 134 XT-PIC i8042
2: 0 XT-PIC cascade
8: 1 XT-PIC rtc
9: 0 XT-PIC uhci_hcd
10: 4397 XT-PIC eth0
11: 46 XT-PIC ioc0
12: 1707 XT-PIC i8042
14: 340596 XT-PIC ide0
15: 63972 XT-PIC ide1
NMI: 0
LOC: 3883128
ERR: 0
MIS: 0

First column refers to the IRQ number. The next column reports the type of interrupt and the last column contains the name of the device that is located to IRQ. Good to read on linux vmstat Command - Tool to Report Virtual Memory Statistics as well.

Sar graph - Installing ksar (a Java-based GUI)

Installing the ksar on top of it is also simple. We need java as the dependency.

sudo apt install openjdk-11-jdk

Or on CentOS

sudo yum install epel-release
sudo yum install java-openjdk

Download ksar and unpack it:

wget https://sourceforge.net/projects/ksar/files/latest/download

mv download kSar.zip

unzip kSar.zip

Now cd into dir

cd kSar-*

Now make run.sh file executable and run it as root.

chmod +x run.sh

sudo su

./run.sh

Now we can test by running local command. We go to Data > Run Local Command and there we can use default command.

ksar run command

Next, we can look at the graphs by clicking on the left panel and looking in the right side the displayed graph:

ksar see graphs

It is also possible to load graphs from file. First we need to run sar:

LC_ALL=C sar -A > /tmp/sar.data.txt

And then in menus, we select Data > Load from text file and find the file in /tmp or wherever we have saved it.

There is also a way to run remote commands over SSH on other computers. Remote computer have to be configured with sar, but ksar is not needed. It is enough to have kSar on local computer. For running SSH command in the menus select Data > Launch SSH command. Then enter [email protected] where 'address' is the FQDN or IP of your computer. By default is localhost.

Read Also:

Bobbin Zachariah 8:53 pm

About Bobbin Zachariah

Founder of LinOxide, passionate lover of Linux and technology writer. Started his career in Linux / Opensource from 2000. Love traveling, blogging and listening music. Reach Bobbin Zachariah about me page and google plus page.

Author's All Posts
Like to become part of Linoxide Team and contribute tips? Contact us here.

Comments

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.