How to View Progress of Commands using Linux Pipe

view progress commands pipeWhen you work on some processes, sometimes you need to visualize the progression of file operations. Some commands have a parameter to help you to see the progression but for those which don't have it, you need to use another method. To mitigate this problem, you can use a pipe viewer which indicates the progression of the different processes that you launch.

What is pipe viewer

Pipe viewer (pv) is a CLI tool which allows you to monitor the progress of data through a pipeline by giving information such as percentage completed, ETA, total data transferred, time elapsed, etc. If you have some processes, you can insert the pipe viewer as a normal pipe between the different processes to have a visual indication of the progression.

In this article, I will show you the Unix pipe viewer utility in Node.js proposed by Rocco Musolino on his Github page. It only uses two parameters:

  • -s, --size <size>: Assume the total amount of data to be transferred is SIZE. You can provide a size in bytes or using units (b, kb, mb, gb, tb).
  • -N, --name <name>: Prefix the output information with NAME.

1) Installation

The Unix pv of Rocco can be installed through node.js as below:

# npm install -g pv
/usr/bin/pv -> /usr/lib/node_modules/pv/bin/pv.js
+ [email protected]
added 15 packages in 0.953s

Examples of pipe viewer usage

In this part, I will show you some different examples which can help you to use the pv tool

a) Display a dd copy progression

You can display the progression of a file copying when using the dd command

# dd if=ubuntu-gnome-17.04-desktop-amd64.iso | pv | dd of=dest/ubuntu17
 0.00% | 0 ETA | 1.36GB Transferred | 95.71MB/s2964416+0 records in
2964416+0 records out
1517780992 bytes (1.5 GB) copied, 14.7174 s, 103 MB/s
 100.00% | 0 ETA | 1.41GB Transferred | 96.5MB/s2964416+0 records in
2964416+0 records out
1517780992 bytes (1.5 GB) copied, 14.7301 s, 103 MB/s

b) A cat command progression

You can decide to display the progression of the cat command when displaying the content of message log file for example

# cat /var/log/messages | pv
Oct 13 03:36:02 localhost journal: Runtime journal is using 6.1M (max allowed 49.4M, trying to leave 74.1M free of 488.4M available → current limit 49.4M).
Oct 13 03:36:02 localhost kernel: Linux version 4.9.36-x86_64-linode85 ([email protected]) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Thu Jul 6 15:31:23 UTC 2017
........
........
100.00% | 0 ETA | 114.83KB Transferred | 114.83KB/s

c) Display the progression while compressing

During some operations of compression, we can need to display the progression. It is possible as below

# tar -cf - /var/log/ | pv -N backlog | tar -C . -x
tar: Removing leading `/' from member names
backlog: 100.00% | 0 ETA | 660KB Transferred | 660KB/s

d) Visualize a live ssh network

You can connect to a host via ssh and displays the live transfer speed. In our case, we will direct all transferred data to /dev/null with a local test

# yes | pv | ssh [email protected] "cat > /dev/null"
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:lqes2c9Obi9lCSepwdT8uovcsmhcboUAsylF69MENDE.
ECDSA key fingerprint is MD5:38:67:a1:25:96:7b:69:02:f6:b8:ff:5d:ce:03:fd:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
[email protected]'s password: 
 0.00% | 0 ETA | 800.84MB Transferred | 17.04MB/s

The Unix pipe viewer of Rocco Musolino doesn't display the percentage completed with the progress bar as the "normal" pipe viewer installed through the package. We can see that the pipe viewer is interesting when we face commands which don't allow to visualize the progression of the processes.

About Alain Francois

IT Linux administrator passionate of free and open source software, I work on Linux Systems since some years on installations and deployments of certain solutions. I like to share my experiences with a wider audience by training and conferences.

Author Archive Page

Have anything to say?

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.