CPU Bottleneck - How To Use Vmstat, Mpstat and SAR Commands

March 14, 2011 | By
| Reply More

Linux CPU

One of the main functions the kernel or Operating System is Memory Management. When process try to access physical memory, if physical memory not free, than kernel move the inactive processes to virtual memory then it is called Paging. To know which process is using physical memory and virtual memory, we use vmstat, mpstat, sar commands.

1) Vmstat (virtual memory statistics) is a computer monitoring tool that collects and displays system memory, processes, interrupts, paging and block I/O. The vmstat command is used for summarizing the total active virtual memory used by all processes in the system, as well as the number of real memory pages.

Example: 1 

[root@mailserver /]# vmstat 2 10 | head -4
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 0 240596 97040 242000 0 0 106 9884 1115 280 69 20 2 9
2 4 0 168404 97116 308744 0 0 0 17944 1085 141 79 21 0 0

Where
Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.
Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)
Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

Example: 2

[root@localhost ~]# vmstat -f
2524 forks

Above command displays all the fork system calls made by the kernel since last reboot. This shows all fork, vfork and clone system call counts.

Example: 3

[root@localhost ~]# vmstat -m | head -5
Cache Num Total Size Pages
cifs_small_rq 30 36 448 9
cifs_request 4 4 16512 1
cifs_oplock_structs 0 0 32 113
cifs_mpx_ids 3 59 64 59

Above command displays the slabinfo.

Example: 4

[root@localhost ~]# vmstat -s
978504 total memory
375712 used memory
80572 active memory
275132 inactive memory
602792 free memory
32836 buffer memory
286500 swap cache
1020116 total swap

Above command displays a table of various event counters and memory statistics. This display does not repeat

2) About mpstat
Mpstat is command which is used for monitoring the CPU resource utilization in linux.

Exampe: 1

[root@localhost ~]# mpstat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011

10:25:32 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
10:25:32 PM all 5.68 0.00 0.49 2.03 0.01 0.02 0.00 91.77 146.55

Above command displays basic information about the CPU.

Example: 2

[root@localhost ~]# mpstat -P ALL
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU)

10:28:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:28:04 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99
10:28:04 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98
10:28:04 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
10:28:04 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
10:28:04 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

The above command displays the statistics of individual as well as hole CPU performance.

Example: 3

[root@localhost ~]# mpstat -P 0
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (8 CPU)

10:28:53 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:28:53 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98

Above –p options displays the statistic of the individual CPU performance.

3) About sar
sar is a performance monitoring utility, comes with sysstat package. Make sure that sysstat is installed on your system. sar collects reports on an ongoing basis and saves the performance data. Using sar command, you can collect the following information in the system.

Collective CPU usage
Individual CPU statistics
Memory used and available
Swap space used and available
Overall I/O activities of the system
Individual device I/O activities
Context switch statistics
Run queue and load average data
Network statistics

Example: 1

[root@localhost ~]# sar 1 2
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

01:27:32 PM CPU %user %nice %system %iowait %steal %idle
01:27:33 PM all 0.00 0.00 0.00 0.00 0.00 100.00
01:27:34 PM all 0.25 0.00 0.25 0.00 0.00 99.50
Average: all 0.33 0.00 0.17 0.00 0.00 99.50

Above command displays System CPU statistics 2 times with 1 second interval.

Example: 2

[root@localhost ~]# sar -u 1 2
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

01:27:32 PM CPU %user %nice %system %iowait %steal %idle
01:27:33 PM all 0.00 0.00 0.00 0.00 0.00 100.00
01:27:34 PM all 0.25 0.00 0.25 0.00 0.00 99.50
Average: all 0.33 0.00 0.17 0.00 0.00 99.50

Above command displays cumulative real time CPU, i.e. usage of all CPUs. It reports every 1 second total of 2 times.

Example: 3

[root@localhost ~]# sar -P ALL 1 1

Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

01:34:12 PM CPU %user %nice %system %iowait %steal %idle
01:34:13 PM all 11.69 0.00 4.71 0.69 0.00 82.90
01:34:13 PM 0 35.00 0.00 6.00 0.00 0.00 59.00
01:34:13 PM 1 22.00 0.00 5.00 0.00 0.00 73.00
01:34:13 PM 2 3.00 0.00 1.00 0.00 0.00 96.00
01:34:13 PM 3 0.00 0.00 0.00 0.00 0.00 100.00

Above command displays statistic of all individual CPUs with interval of 1 second and 1 time.

Example: 4
[root@localhost ~]# sar -P 0 1 1

Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

01:34:12 PM CPU %user %nice %system %iowait %steal %idle
01:34:13 PM 0 35.00 0.00 6.00 0.00 0.00 59.00

Above command displays statistics of first core with interval of 1 second and with single report.

Example: 5

[root@localhost ~]# sar -r 1 2

Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

07:28:06 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact
07:28:07 AM 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204
07:28:08 AM 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204
Average: 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204

Above commands displays statistic of memory of every 1 second and total 2 times.

Example: 6

[root@localhost ~]# sar -S 1 2
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

07:31:06 AM kbswpfree kbswpused %swpused kbswpcad %swpcad
07:31:07 AM 8385920 0 0.00 0 0.00
07:31:08 AM 8385920 0 0.00 0 0.00
Average: 8385920 0 0.00 0 0.00

Above commands displays statistic about swap memory 0f every 1 second and 2 times.

How to check CPU bottleneck on Linux

When the user + system, CPU values approach 100 percent. This just means that the CPUs are working to their full potential. The only metric that identifies a CPU bottleneck is when the run queue (r value) exceeds the number of CPUs on the server.

[root@localhost ~]# vmstat 5 1
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
101 1 5204 204512 148420 11213136 1 1 0 1 3 1 0 3 1 2

Mpstat can be also used to identify CPU bottleneck. Below example you can notice high usage of interrupts.

[root@localhost ~]# mpstat 2
Linux 2.4.21-50.ELsmp (testserver01) 01/28/10

04:51:52 CPU %user %nice %system %iowait %irq %soft %idle intr/s
04:51:54 all 48.06 0.00 51.94 0.00 0.00 0.00 0.00 105.53
04:51:56 all 47.25 0.00 52.75 0.00 0.00 0.00 0.00 120.00
04:51:58 all 46.47 0.00 53.53 0.00 0.00 0.00 0.00 103.00
04:52:00 all 47.25 0.00 52.75 0.00 0.00 0.00 0.00 104.50
04:52:02 all 46.69 0.00 53.31 0.00 0.00 0.00 0.00 106.50
04:52:04 all 46.62 0.00 53.38 0.00 0.00 0.00 0.00 103.50
04:52:06 all 46.56 0.00 53.44 0.00 0.00 0.00 0.00 106.00
04:52:08 all 47.50 0.00 52.50 0.00 0.00 0.00 0.00 104.50
04:52:10 all 45.00 0.00 55.00 0.00 0.00 0.00 0.00 106.00
04:52:12 all 46.50 0.00 53.50 0.00 0.00 0.00 0.00 102.50
04:52:14 all 47.00 0.00 53.00 0.00 0.00 0.00 0.00 103.00

You can use sar to find out which interrupt number was culprit. #9 was the highest excluding system interrupt #0.

[root@localhost ~]# sar -I XALL 2 10
02:07:10 AM INTR intr/s
02:07:12 AM 0 992.57
02:07:12 AM 1 0.00
02:07:12 AM 2 0.00
02:07:12 AM 3 0.00
02:07:12 AM 4 0.00
02:07:12 AM 5 0.00
02:07:12 AM 6 0.00
02:07:12 AM 7 0.00
02:07:12 AM 8 0.00
02:07:12 AM 9 350.50

Details in /proc/interrupts file will also find helpful for you. Interrupt halts processing momentarily so that I/O or other operations can occur. Processing resumes after the specific operation takes place. So it is very important that each device installed in machine must be provided with an interrupt setting that does not conflict with the settings used by the hardware and other peripherals.

[root@mailserver /]# cat /proc/interrupts
CPU0
0: 3882853 XT-PIC timer
1: 134 XT-PIC i8042
2: 0 XT-PIC cascade
8: 1 XT-PIC rtc
9: 0 XT-PIC uhci_hcd
10: 4397 XT-PIC eth0
11: 46 XT-PIC ioc0
12: 1707 XT-PIC i8042
14: 340596 XT-PIC ide0
15: 63972 XT-PIC ide1
NMI: 0
LOC: 3883128
ERR: 0
MIS: 0

First column refers to the IRQ number. The next column reports the type of interrupt and the last column contains the name of the device that is located to IRQ.

XT-PIC This is the old AT computer interrupts.

IO-APIC-edge Voltage signal on this interrupt transitions from low to high creates an edge, where the interrupt occurs and is only signaled once. This kind of interrupt as well as the IO-APIC-level interrupt, is only seen on systems with processors from the 586 family and higher.

IO-APIC-level Generates interrupts when its voltage signal is high until the signal is low again.

Filed Under : HOWTOS, LINUX HOWTO, MONITORING

Tagged With : ,

Free Linux Ebook to Download

Leave a Reply

Commenting Policy:
Promotion of your products ? Comment gets deleted.
All comments are subject to moderation.