14 Command Line Tools to Check CPU Usage in Linux

May 9, 2017 | By in LINUX HOWTO
| 4 Replies More

cpu performance toolsCPU performance is one aspect of measuring the performance of a system. Monitoring the performance of CPU is essential to debug processes inside any system, manage system resources, making system decisions, and evaluating and examining systems in real time. There are a lot of tools available to monitor and display CPU performance and these tools rely on system calls that are built into the operating system to extract the performance readings. This tool can also unveil huge amount of performance related data of any system to mitigate the performance related issues quickly. In this article, we will explore few CPU/Performance monitoring tool and their usages in any Linux based distros.

1. Top

The top command displays real time view of performance related data of all running processes in a system. By default, the top command updates data in every 5 seconds. Type 'top' from the terminal to view the statistical data related to the performance of a system.

 # top

To modify the output of top command, type 'i' to remove the idle processes from the display. To revert back type 'i' again. To sort the data by memory usage type 'M', to sort by how long the processes have been running type 'S' and type 'P' to sort by CPU usage again. It is also possible to modify the processes from within the top command. Type 'u' to view processes owned by a specific user, 'k' to kill processes and 'r' to renice them.

2. Iostat

The iostat command list CPU utilization, device utilization and network file system utilization. This utility display system’s average CPU utilization since the last reboot. Running the following command without any options displays all the three reports.

 # iostat

To list the individual report use -c, -d and -h switch for CPU utilization, device utilization and network file system utilization. The following command will break the CPU utilization into user processes, system processes, I/O wait and idle time.

 # iostat -c

The following command will display extended statistics (-x option ) along with number of time ( -t option  ) each report should be displayed for CPU utilization ( -c option ).

 # iostat -xtc 5 2

3. Vmstat

The vmstat command will display statistics about system processes, memory, swap, I/O, and the CPU performance. These statistics are generated using data from the last time the command was run to the present. In case the command never being run, the data will be from the last reboot to the current time. On multiple CPU systems, vmstat averages the number of CPUs into the output. The vmstat command updates its output like top command and can be executed like below. The following command will update its report in every 5 seconds until there is an interrupt.

 # vmstat 5

The following command will display the data that will update in every 5 seconds and the values will be re-measured and reported every 1 second.

 # vmstat 1 5

The first line of the report will contain the average values since last time the computer was rebooted. The rest of the lines will display their respective current values. Vmstat can be run as a normal user.

4. Mpstat

The mpstat command writes to standard output about activities for each available processor, processor 0 being the first one.  Global average activities among all processors are also reported.  The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed. If no activity has been selected, then the default report is the CPU utilization report.

mpstat without any option will report Global Average Activities of all CPUs

 # mpstat

mpstat with -p option and 'ALL' will list statistics about all CPUs one by one starting from 0.

 # mpstat -P ALL

The following command will list average CPU usage for 4 times, each after 2 seconds.

 # mpstat -P ALL 2 4

5. Sar

The sar is performance monitoring tool for collecting, viewing and recording performance data and can list what a system is doing all the time. Sar can generate report and email them to system admin. The statistics reported by sar includes I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities and TTY statistics, among others. Run sar without any option and check the output. The default output list statistics for every 10 minutes and a final average.

 # sar

Display CPU statistics 3 times with 2 second interval.

 # sar 2 3

The following command displays cumulative real-time CPU usage of all CPU for every 2 seconds a total of 3 times.

 # sar -u 2 3

The '-P ALL' option displays statistics for ALL the individual Cores. If your system has 4 cores then the 'CPU column' will contain the number 0, 1, 2, 3 indicating the corresponding CPU core numbers. The '2 3' option displays the statistics every 2 seconds for 3 times.

 # sar -P ALL 2 3

The '-P 2' option displays statistics for third core every 2 seconds for 3 times.

 # sar -P 2 2 3

6. CoreFreq

CoreFreq is a CPU performance monitoring software designed for 64-bits Processors w/ architectures Intel Atom, Core2, Nehalem, SandyBridge and superior, AMD Family. The CoreFreq provides a framework for retrieving CPU data with a high degree of precision. CoreFreq is made of three parts. A kernel module, a daemon, and a userland command line interface (CLI). The kernel module will have full access to the host. It is its responsibility to gather the low-level data.  Data are then collected by an userland daemon. The daemon must be run as root. Finally, CoreFreq comes with a terminal interface to report data in a nice human-readable format. That CLI tool is an ordinary process and may be used by the end user.

Prerequisite

i) CoreFreq uses the low-level counter and hence you should first disable NMI watchdog through /etc/default/grub. Edit /etc/default/grub and add the 'nmi_watchdog=0' kernel command line argument and subsequently run update-grub.

  # vi /etc/default/grub
  ....................
  ....................
  GRUB_CMDLINE_LINUX="nmi_watchdog=0"
  ....................
  ....................
  # update-grub && reboot

ii) No Virtualization. VMs don't provide access to the registers that the CoreFreq driver employs.

Install CoreFreq

 # sudo apt-get install git dkms build-essential libc6-dev libpthread-stubs0-dev
 # git clone https://github.com/cyring/CoreFreq.git
 # cd CoreFreq
 # make

Install the kernel module with insmod

 # sudo insmod corefreqk.ko

List the CoreFreq module with lsmod

 # lsmod | grep corefreq
 corefreqk              49152  0

Check if the CoreFreq has been recognized by the processor by examining the output of dmesg

 # sudo dmesg | grep CoreFreq
 [  111.825745] CoreFreq: Processor [06_3F] Architecture [Haswell/Mobile] CPU [1/1]

Start the CoreFreq daemon

 # sudo ./corefreqd -i &

Now start the corefreq client as a user.

 # ./corefreq-cli

7. Htop

Htop is a process viewer and a text mode application for system monitoring in real-time like top. Htop displays a complete list of the processes that are running and is easy to use. It is based on ncurses for viewing the processes in a GUI-like environment running in the terminal. Htop displays usage per CPU along with memory and swap usage with a significant text graph printed at the top. Htop is very much convenient to use and easy to understand. Once you start using HTOP, you might not return to TOP.

Install htop using the following command in the terminal

 # sudo apt-get install htop

Run htop using the following command.

 # htop

The output of Htop consists of three parts. The upper part displays system information in the form of three bars that can be changed using HTOP setup menu. This part shows CPU Usage, Memory Usage and Swap Usage. The Middle part list all the processes arranged with respect to their CPU usage. The bottom part shows HTOP menu commands.

8. Nmon

nmon (Nigel’s performance Monitor for Linux & AIX) has been developed by IBM employee Nigel Griffiths. This tool is used to monitor system resources such as CPU, memory, network, disks, file systems, NFS, top processes in the terminal. NMON supports various architectures like POWER, x86, x86_64, Mainframe and ARM (Raspberry Pi). This tool helps system administrator to tune, benchmark performance information to troubleshoot the CPU/system performance when there is some issues. For real-time monitoring, it uses curses library for low CPU impact and displays statistics on the screen and update in every two seconds. nmon consumes less CPU memory as compared with other tools since it uses curses library.

Installation

 # sudo apt-get install nmon (Ubuntu)
 # sudo yum install nmon (CentOS)

CentOS users need to install/enable EPEL Repository in order to install nmon.

Type nmon in the terminal and press enter, nmon will display a welcome screen with all the options to use it further.  If you want to display CPU utilization, press c and to hide the CPU utilization widgets statistics, press 'c' again.

CPU performance monitor with nmon

9. Glances

Glances is a cross-platform curses-based monitoring tool written in Python that uses the psutil library to fetch data from the system. Glance monitor CPU, Load Average, Memory, Network Interfaces, Disk I/O, Processes and File System spaces utilization.

Install glances in your system with the following command.

In Ubuntu:

 # sudo apt-get install glances

In Centos:

 # sudo yum install glances

You can run Glances in 3 modes-

1. Standalone

If you want to monitor your local machine(Standalone), simply run-

 # glances

2. Client/Server

If you want to remotely monitor a machine then execute the following on the server.

 # glances -s

and then from the client, execute the following command.

 # glances -c @server

where @server is the IP address or hostname of the server.

3.  Web server

Run the glances in web server mode by executing following command in the terminal.

 # glances -w 
 Glances web server started on http://0.0.0.0:61208/

To change the refresh rate of the page, just add the period in seconds at the end of the URL. For example, to refresh the page every 30 seconds, append 10 to the glances server URL from the browser.

 http://@server:61208/30

Glances in Android

10. Cpustat

Cpustat is like a fancy sort of top that does different things. Most performance tools average CPU usage over a few seconds or even a minute. This results in a fantasy of excess capacity because of an abrupt increase in resource usage that is blended in with less busy periods. On the other hand, cpustat takes higher frequency samples of each process and summarizes these samples at a lower frequency. For example, it can measure a process in every 200ms and summarize the samples in every 5 seconds, including min/average/max values for some metrics.

Install Cpustat

Cpustat is written in Go language. Therefore, you need GO in your system. If it is not there then install it using following commands in the terminal.

In Ubuntu

 # sudo add-apt-repository ppa:longsleep/golang-backports
 # sudo apt-get update
 # sudo apt-get install golang-go

In CentOS

 # yum install golang

Once Golang is installed in your system then create a directory for Cpustat and make this directory available to the GOPATH.

 # mkdir cpustat
 # export GOPATH=$HOME/cpustat

Install Cpustat and verify that the directories are copied over to the folder cpustat.

 # go get github.com/uber-common/cpustat
 # cd cpustat && ls

Add bin folder to the PATH environment variable. You can also add the following line to ~/.bashrc or /etc/profile depending on what you are using.

 # export PATH=$PATH:/root/cpustat/bin

There are two ways of displaying this data. Either a pure text list of the summary interval or a colorful scrolling dashboard of each sample. To display data in fancy terminal mode, execute the following command in the terminal.

 # cpustat -t

To run in pure text mode, use the following command. In text mode, few system-wide summary metrics come from /proc/stat.

 # cpustat

The following command will take a sample of all processes in every 500ms and summarize this data after 10 samples, which is every 5 seconds.

 # cpustat -s 500 -s 10 -n 20

Use the following command to measure processes that are owned by either user root or user ubuntu. The overall system stats will still be measured.

 # cpustat -u root, ubuntu

Only measure processes of  "apache" or "mysql".  The -p option of cpustat takes a list of process ids returned by pgrep. pgrep is used to get the process ids ( -d option ).

 # cpustat -p $(pgrep -d, apache2\|mysqld)

Use the following command to write memory profile in a file.

 # cpustat -memprofile test.txt

11. perf

Perf can measure CPU performance counters, tracepoints, kprobes, and uprobes that is included in the Linux kernel, under tools/perf. perf began as a tool for using the performance counters subsystem in Linux, and has had various enhancements to add tracing capabilities. The perf tools are integrated into the Linux kernel since the 2.6 version and are based on the perf events subsystem. The perf profiler uses hardware counters to profile the application. The result of this profiler is explicit and fast. The perf utility can be found in the linux-tools package.

Measure entire system for 10 seconds using perf.
 # perf stat -a -- sleep 10

System wide perf stat

If you are interested in finding CPU performance for a particular 'command' like cp. The following perf command will profile cp while it is copying the entire directory Documents to new location (Docs).

 # perf stat cp -r Documents/ Docs

If you want to find CPU counter statistics for a specified PID then use the following command until CTRL+C is pressed.

 # perf stat -p 2087

Find the basic CPU statistics, system wide, for 10 seconds using the following perf command. You can also omit any one of the options under -e option to be more specific.

 # perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10

Collect Sample on-CPU user instructions, for 5 seconds and then use perf report to view the report.

 # perf record -e cycles:u -a -- sleep 5
 # perf report

Collect sample CPU stack traces, once every 10,000 Level 1 data cache misses, for 5 seconds and then use perf report to view the report.

 # perf record -e L1-dcache-load-misses -c 10000 -ag -- sleep 5
 # perf report

12. tiptop

Tiptop reads hardware performance counters and displays statistics about running Linux processes, such as IPC, or cache misses. It provides a dynamic real-time view of the tasks running in the system. It is very similar to top, but the information displayed comes from hardware counters. It has two running modes: live-mode and batch-mode. In both modes, the system is periodically queried for the values of hardware counters, and various ratios are printed for each task.

# tiptop -b
tiptop - up 1 days, 15:36, load average: 0.00, 0.01, 0.05
Fri May 19 08:38:25 UTC 2017
delay: 2.00 idle: 0 threads: 0
Screen 0: default

PID [ %CPU] %SYS P Mcycle Minstr IPC %MISS %BMIS %BUS COMMAND 
 830 0.5 0.0 0 0.04 0.00 0.02 19.45 5.91 1.9 rpcbind

13. ps command

We can use ps command to find cpu usage for each process or users. Given few examples to undestand

To display highest CPU & Memory utilization by processes run by root.

# ps u | sort -k 1 -r | head -5
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 473 0.0 0.1 110036 836 tty1 Ss+ Apr21 0:00 /sbin/agetty --noclear tty1 linux
root 10290 0.0 0.1 107904 664 pts/0 S+ 17:31 0:00 head -5
root 10289 0.0 0.1 116444 900 pts/0 S+ 17:31 0:00 sort -k 1 -r
root 10288 0.0 0.3 151056 1824 pts/0 R+ 17:31 0:00 ps u

To display CPU usage for specific command

# ps -eo pcpu,args | sort -k 1 -r | head -8
%CPU COMMAND
 0.8 sshd: root [priv]
 0.8 sshd: root [priv]
 0.6 sshd: root [priv]
 0.0 [writeback]
 0.0 [watchdog/0]
 0.0 [virtscsi-scan]
 0.0 [vballoon]

To show processes for all users and not attached to a terminal in full format listing with a sort

# ps -auxf | sort -nr -k 3 | head -10
root 10513 0.5 1.0 144476 5300 ? Ss 17:48 0:00 \_ sshd: root [priv]
root 10511 0.5 1.0 144476 5308 ? Ss 17:48 0:00 \_ sshd: root [priv]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
sshd 10514 0.0 0.4 88228 2496 ? S 17:48 0:00 \_ sshd: root [net]
sshd 10512 0.0 0.4 88228 2500 ? S 17:48 0:00 | \_ sshd: root [net]
rpcuser 2492 0.0 0.3 42348 1720 ? Ss Apr21 0:00 /usr/sbin/rpc.statd --no-notify
rpc 2334 0.0 0.2 64948 1416 ? Ss Apr21 0:01 /sbin/rpcbind -w
root 9344 0.0 0.0 0 0 ? S 16:01 0:00 \_ [kworker/0:0]
root 92 0.0 0.0 0 0 ? S Apr21 0:01 \_ [kauditd]
root 9 0.0 0.0 0 0 ? R Apr21 0:05 \_ [rcu_sched]

14. dstat command

dstat command is a versatile tool for generating system resource statistics and it would also show cpu stats. You need to install it in order to use it

On Centos

# yum install -y dstat

On Ubuntu

# apt-get install dstat

To monitor program that is using the most CPU and consuming the most amount of memory.

# dstat -c --top-cpu -dn --top-mem

Conclusion

In this article, we have explored few CPU performance monitoring tool along with their usages. Be sure to read man pages for each of them, as it also documents the usages in details. Remember that using these tools also takes a CPU slice. For example, top takes 3%-4% of CPU while glances take 15% to 20% of CPU. So while using any one of these, you need to consider this aspect as well.  In a nutshell, you will be able to find which process is taking more CPU time, are there enough free memory, whether processes are stalled while waiting for I/O viz. disk or network access to complete read/write and much more by using a combination of few of them.

Filed Under : LINUX HOWTO, MONITORING

Tagged With :

Free Linux Ebook to Download

Comments (4)

Trackback URL | Comments RSS Feed

  1. CyrIng says:

    Nice review, thank you.
    Can you measure the CPU slice taken by CoreFreq ?

  2. Dwijadas Dey says:

    Hi
    Yes, you can use top command to find the CPU slice taken by CoreFreq under %CPU column.

  3. CyrIng says:

    CoreFreq's CPU overhead is 0% with a max of 0.5% periodically.

    Do you get the same results ?

  4. Dwijadas Dey says:

    You are probably checking the memory usage of CoreFreq client. You can find the find the CPU slice of CoreFreq daemon using CPUSTAT

    # cpustat -p $(pgrep -d, corefreqd)

Leave a Reply

All comments are subject to moderation.