Run Command Parallel on Multiple Hosts using PDSH Tool

PDSH is a very smart little tool that enables you to issue the same command on multiple hosts at once, and see the output. It is a high-performance and parallel remote shell utility. It can run multiple remote commands in parallel and uses a "sliding window" (or fanout) of threads to conserve resources on the initiating host while allowing some connections to time out. In this article we explain how to install pdsh tool and show few examples.

How to Install PDSH

Installing pdsh is really simple, please follow blow steps

1) Download latest version of PDSH from google code website.

2) Extract and Decompress a pdsh file

[shaha@oc8535558703 Downloads]$ bzip2 -dc pdsh-2.26.tar.bz2 | tar xvf -

3) Now install pdsh with below commands

[shaha@oc8535558703 Downloads]$ cd pdsh-2.26
[shaha@oc8535558703 pdsh-2.26]$ ./configure --with-ssh --without-rsh
checking metadata... yes
checking build system type... x86_64-redhat-linux-gnu
checking host system type... x86_64-redhat-linux-gnu
checking target system type... x86_64-redhat-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... gcc
[shaha@oc8535558703 pdsh-2.26]$ make
[shaha@oc8535558703 pdsh-2.26]$ make install

4) After complete make install command , please verify pdsh installed version with below command

[shaha@oc8535558703 pdsh-2.29]$ pdsh -V
rcmd modules: ssh,exec (default: ssh)
misc modules: (none)
[shaha@oc8535558703 pdsh-2.29]$

Note :- Puts the binaries into /usr/local/, which is fine for testing purposes. For production work, I would put it in /opt or something like that – just be sure it's in your path.

You might notice that I used the --without-rsh option in the configure command. By default, pdsh uses rsh, which is not really secure, so I choose to exclude it from the configuration. To override rsh and make ssh the default, you just add the following line to your .bashrc file:

[shaha@oc8535558703 ~]$ cat .bashrc
# .bashrc
export PDSH_RCMD_TYPE=ssh
[shaha@oc8535558703 ~]$

Be sure to "source" your .bashrc file (i.e., source .bashrc) to set the environment variable. You can also log out and log back in. If, for some reason, you see the following when you try running pdsh,

Requirment for executing pdsh command on multiple node

a) SSH keys

In this short series of blog posts I’m going to take a look at a few very useful tools that can make your life as the sysadmin of a cluster of Linux machines easier. This may be a Hadoop cluster, or just a plain simple set of ‘normal’ machines on which you want to run the same commands and monitoring.

To start with, we’re going to use the ever-awesome ssh keys to manage security on the cluster. After that we’ll look at executing the same command across multiple machines at the same time using PDSH.

In a nutshell, ssh keys enable us to do password-less authentication in a secure way. Working with SSH keys involves taking the public key from a pair, and adding that to another machine in order to allow the owner of the pair’s private key to access that machine. What we’re going to do here is generate a unique key pair that will be used as the identity across the cluster. So each node will have a copy of the private key, in order to be able to authenticate to any other node, which will be holding a copy of the public key (as well as, in turn, the same private key).

We’ve several ways we could implement the SSH keys. Because it’s a purely sandbox cluster, I could use the same SSH key pair that I generate for the cluster on my machine too, so the same public/private key pair is distributed thus:

First, we need to generate the key using ssh-keygen command

[shaha@oc8535558703 ]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/shaha/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/shaha/.ssh/id_rsa.
Your public key has been saved in /home/shaha/.ssh/
The key fingerprint is:
The key's randomart image is:
+--[ RSA 2048]----+
| .E. |
| o o |
| . + |
| . .. |
| o.S o |
| ..o.oo.o . |
| .o .=+= |
| o o+o. |
| . ... .. |
[shaha@oc8535558703 ]$

This generates in the .ssh folder two files – the private and public (.pub) keys of the pair:

[shaha@oc8535558703 ~]$ cd .ssh/
[shaha@oc8535558703 .ssh]$ ls -lrt
-rw-------. 1 shaha shaha 1675 Aug 4 01:03 id_rsa
-rw-r--r--. 1 shaha shaha 408 Aug 4 01:03
[shaha@oc8535558703 .ssh]$

b) Preparing the authorized_keys file

Now we’ll prepare the authorized_keys file which is where the public SSH key of any identity permitted to access the machine is stored. Note that each user on a machine has their own authorized_keys file, in ~/.ssh/. So for example, the root user has the file in /root/.ssh/authorized_keys and any public key listed in that file will be able to connect to the server as the root user. Be aware the American [mis-]spelling of “authorized” – spell it [correctly] as “authorised” and you’ll not get any obvious errors, but the ssh key login won’t work either.

Distributing the SSH artifacts

Now we’re going to push this set of SSH files out to the .ssh folder of the target user on each node, which in this case is the root user. From a security point of view it’s probably better to use a non-root user for login and then sudo as required, but we’re keeping things simple (and less secure) to start with here. So the files in our folder are:

id_rsa – the private key of the key pair – the public key of the key pair.

Strictly speaking this doesn’t need distributing to all nodes, but it’s conventional and handy to hold it alongside the private key.

authorized_keys – this is the file that the sshd daemon on each node will look at to validate an incoming login request’s offered private key, and so needs to hold the public key of anyone who is allowed to access the machine as this user.

[shaha@oc8535558703 ~]$ cat ~/.ssh/ |ssh 'cat >> .ssh/authorized_keys''s password:
[shaha@oc8535558703 ~]$

At this point you’ll need to enter the password for the target user, but rejoice! This is the last time you’ll need to enter it as subsequent logins will be authenticated using the ssh keys that you’re now configuring.

c) Confirm Password less connectivity

If you are getting below error during password less connectivity

[shaha@oc8535558703 .ssh]$ ssh
Agent admitted failure to sign using the key.'s password:

Then run this command

[shaha@oc8535558703 .ssh]$ export SSH_AUTH_SOCK=0
[shaha@oc8535558703 .ssh]$ ssh
[shaha@oc8535558703 .ssh]$ ssh

PDSH Command Example

To begin, I'll try to get the kernel version of a node by using its IP address:

[shaha@oc8535558703 ~]$ pdsh -w uname -r
ec2-52-59-121-138: 4.4.11-23.53.amzn1.x86_64
[shaha@oc8535558703 ~]$

The -w option means I am specifying the node(s) that will run the command. In this case, I specified the IP address of the node (ec2-52-59-121-138). After the list of nodes, I add the command I want to run, which is uname -r in this case. Notice that pdsh starts the output line by identifying the node name.

1) Single Node Command Execution

To combine commands together that you send to each host or single host you can use the standard bash operator semicolon ;

[shaha@oc8535558703 ~]$ pdsh -w uname -a;hostname ; who
ec2-52-28-150-229: Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
shaha tty1 2016-07-31 20:08 (:0)
shaha pts/0 2016-08-02 11:11 (:0.0)
shaha pts/1 2016-08-02 11:13 (:0.0)
shaha pts/2 2016-08-02 12:02 (:0.0)
shaha pts/3 2016-08-02 20:38 (:0.0)
shaha pts/4 2016-08-03 10:52 (:0.0)
shaha pts/5 2016-08-03 15:19 (:0.0)
shaha pts/6 2016-08-03 10:33 (:0.0)
shaha pts/7 2016-08-03 17:09 (:0.0)
shaha pts/8 2016-08-03 18:48 (:0.0)
shaha pts/9 2016-08-03 22:49 (:0.0)
[shaha@oc8535558703 ~]$

2) Multiple Node command execution

A very common way of using pdsh is to set the environment variable WCOLL to point to the file that contains the list of hosts you want to use in the pdsh command. For example, I created a subdirectory PDSH where I create a file hosts that lists the hosts I want to use:

[shaha@oc8535558703 ~]$ mkdir PDSH
[shaha@oc8535558703 ~]$ cd PDSH
[shaha@oc8535558703 PDSH]$ vi hosts

Insert all node in hostfile

[shaha@oc8535558703 PDSH]$ more hosts
[shaha@oc8535558703 PDSH]$

I'm only using two nodes: and . The first is my test system (like a cluster head node), and the second is my test compute node. You can put hosts in the file as you would on the command line separated by commas. Be sure not to put a blank line at the end of the file because pdsh will try to connect to it. You can put the environment variable WCOLL in your .bashrc file:

export WCOLL=/home/shaha/PDSH/hosts

As before, you can source your .bashrc file, or you can log out and log back in.

3) Specifying Hosts Command Execution

I won't list all the several other ways to specify a list of nodes. The simplest way is to specify the nodes on the command line is to use the -w option:

[shaha@oc8535558703 ~]$ pdsh -w, uname -r
ec2-52-59-121-138: 4.4.11-23.53.amzn1.x86_64
ec2-52-58-254-227: 3.13.0-92-generic
[shaha@oc8535558703 ~]$

In this case, I specified the node names separated by commas. You can also use a range of hosts as follows:

[shaha@oc8535558703 PDSH]$ pdsh -w ^hosts uptime
ubuntu@ec2-52-58-254-227: 10:00:52 up 2 days, 16:48, 0 users, load average: 0.05, 0.04, 0.05
ec2-user@ec2-52-59-121-138: 10:00:50 up 2 days, 16:51, 0 users, load average: 0.00, 0.01, 0.05
[shaha@oc8535558703 PDSH]$

4) More Useful pdsh Commands

Now I can shift into second gear and try some fancier pdsh tricks. First, I want to run a more complicated command on all of the nodes . Notice that I put the entire command in quotes. This means the entire command is run on each node, including the first (cat /proc/cpuinfo) and second (grep bogomips , model ,cpu) parts.

[shaha@oc8535558703 PDSH]$ pdsh 'cat /proc/cpuinfo' | egrep 'bogomips|model|cpu'
ubuntu@ec2-52-58-254-227: cpu family : 6
ubuntu@ec2-52-58-254-227: model : 63
ubuntu@ec2-52-58-254-227: model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
ubuntu@ec2-52-58-254-227: cpu MHz : 2400.070
ubuntu@ec2-52-58-254-227: cpu cores : 1
ubuntu@ec2-52-58-254-227: cpuid level : 13
ubuntu@ec2-52-58-254-227: bogomips : 4800.14
ec2-user@ec2-52-59-121-138: cpu family : 6
ec2-user@ec2-52-59-121-138: model : 62
ec2-user@ec2-52-59-121-138: model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
ec2-user@ec2-52-59-121-138: cpu MHz : 2500.036
ec2-user@ec2-52-59-121-138: cpu cores : 1
ec2-user@ec2-52-59-121-138: cpuid level : 13
ec2-user@ec2-52-59-121-138: bogomips : 5000.07
[shaha@oc8535558703 PDSH]$
[shaha@oc8535558703 PDSH]$ pdsh vmstat 1 2
ec2-user@ec2-52-59-121-138: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
ec2-user@ec2-52-59-121-138: r b swpd free buff cache si so bi bo in cs us sy id wa st
ec2-user@ec2-52-59-121-138: 2 0 0 287424 122872 529200 0 0 1 5 10 20 0 0 100 0 0
ubuntu@ec2-52-58-254-227: procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
ubuntu@ec2-52-58-254-227: r b swpd free buff cache si so bi bo in cs us sy id wa st
ubuntu@ec2-52-58-254-227: 1 0 0 262924 153712 505960 0 0 1 5 13 19 0 0 100 0 0
ec2-user@ec2-52-59-121-138: 0 0 0 287504 122872 529232 0 0 0 0 34 74 0 0 100 0 0
ubuntu@ec2-52-58-254-227: 0 0 0 262924 153712 505992 0 0 0 0 18 20 0 0 100 0 0
[shaha@oc8535558703 PDSH]$
[shaha@oc8535558703 ~]$ pdsh -w ubuntu@ec2-52-59-121-138-node0[1-4] "date;sleep 5;date"
ec2-52-59-121-138-node01: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node03: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node04: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node02: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node01: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node03: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node04: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node02: Sat Nov 30 20:57:11 GMT 2016

Note the use of the quotation marks to enclose the entire command string. Without them the bash interpreter will take the ; as the delimiter of the local commands, and try to run the subsequent commands locally:

[shaha@oc8535558703 ~]$ pdsh -w ubuntu@ec2-52-59-121-138-node0[1-4] date;sleep 5;date
ec2-52-59-121-138-node03: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node04: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node02: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node01: Sat Aug 30 20:57:53 GMT 2016
Sat 30 Aug 2016 20:58:00 GMT

5) Piping and File Redirects

Similar to combining commands above, you can pipe the output of commands, and you need to use quotation marks to enclose the whole command string.

[shaha@oc8535558703 ~]$ pdsh -w root@ec2-52-59-121-138-node[01-4] "chkconfig|grep collectl"
ec2-52-59-121-138-node03: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node01: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node04: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node02: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off


Even experienced admins use parallel shell tools to understand the states of their systems. These tools are easily scriptable, so you can store the data in a flat file or in a database.

Leave a Comment