Run Command Parallel on Multiple Hosts using PDSH Tool

PDSH is a very smart little tool that enables you to issue the same command on multiple hosts at once, and see the output. It is a high-performance and parallel remote shell utility. It can run multiple remote commands in parallel and uses a "sliding window" (or fanout) of threads to conserve resources on the initiating host while allowing some connections to time out. In this article we explain how to install pdsh tool and show few examples.

How to Install PDSH

Installing pdsh is really simple, please follow blow steps

1) Download latest version of PDSH from google code website.

2) Extract and Decompress a pdsh file

[[email protected] Downloads]$ bzip2 -dc pdsh-2.26.tar.bz2 | tar xvf -
pdsh-2.26/
pdsh-2.26/config.h.in
pdsh-2.26/configure.ac
pdsh-2.26/configure
pdsh-2.26/bootstrap
pdsh-2.26/scripts/
pdsh-2.26/scripts/Makefile.am

3) Now install pdsh with below commands

[[email protected] Downloads]$ cd pdsh-2.26
[[email protected] pdsh-2.26]$ ./configure --with-ssh --without-rsh
checking metadata... yes
checking build system type... x86_64-redhat-linux-gnu
checking host system type... x86_64-redhat-linux-gnu
checking target system type... x86_64-redhat-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... gcc

[[email protected] pdsh-2.26]$ make

[[email protected] pdsh-2.26]$ make install

4) After complete make install command , please verify pdsh installed version with below command

[[email protected] pdsh-2.29]$ pdsh -V
pdsh-2.26
rcmd modules: ssh,exec (default: ssh)
misc modules: (none)
[[email protected] pdsh-2.29]$

Note :- Puts the binaries into /usr/local/, which is fine for testing purposes. For production work, I would put it in /opt or something like that – just be sure it's in your path.

You might notice that I used the --without-rsh option in the configure command. By default, pdsh uses rsh, which is not really secure, so I choose to exclude it from the configuration. To override rsh and make ssh the default, you just add the following line to your .bashrc file:

[[email protected] ~]$ cat .bashrc
# .bashrc
export PDSH_RCMD_TYPE=ssh
[[email protected] ~]$

Be sure to "source" your .bashrc file (i.e., source .bashrc) to set the environment variable. You can also log out and log back in. If, for some reason, you see the following when you try running pdsh,

Requirment for executing pdsh command on multiple node

a) SSH keys

In this short series of blog posts I’m going to take a look at a few very useful tools that can make your life as the sysadmin of a cluster of Linux machines easier. This may be a Hadoop cluster, or just a plain simple set of ‘normal’ machines on which you want to run the same commands and monitoring.

To start with, we’re going to use the ever-awesome ssh keys to manage security on the cluster. After that we’ll look at executing the same command across multiple machines at the same time using PDSH.

In a nutshell, ssh keys enable us to do password-less authentication in a secure way. Working with SSH keys involves taking the public key from a pair, and adding that to another machine in order to allow the owner of the pair’s private key to access that machine. What we’re going to do here is generate a unique key pair that will be used as the identity across the cluster. So each node will have a copy of the private key, in order to be able to authenticate to any other node, which will be holding a copy of the public key (as well as, in turn, the same private key).

We’ve several ways we could implement the SSH keys. Because it’s a purely sandbox cluster, I could use the same SSH key pair that I generate for the cluster on my machine too, so the same public/private key pair is distributed thus:

First, we need to generate the key using ssh-keygen command

[[email protected] ]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/shaha/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/shaha/.ssh/id_rsa.
Your public key has been saved in /home/shaha/.ssh/id_rsa.pub.
The key fingerprint is:
42:eb:75:b6:67:f3:86:71:e2:30:af:62:24:15:27:1d [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| .E. |
| o o |
| . + |
| . .. |
| o.S o |
| ..o.oo.o . |
| .o .=+= |
| o o+o. |
| . ... .. |
+-----------------+
[[email protected] ]$

This generates in the .ssh folder two files – the private and public (.pub) keys of the pair:

[[email protected] ~]$ cd .ssh/
[[email protected] .ssh]$ ls -lrt
-rw-------. 1 shaha shaha 1675 Aug 4 01:03 id_rsa
-rw-r--r--. 1 shaha shaha 408 Aug 4 01:03 id_rsa.pub
[[email protected] .ssh]$

b) Preparing the authorized_keys file

Now we’ll prepare the authorized_keys file which is where the public SSH key of any identity permitted to access the machine is stored. Note that each user on a machine has their own authorized_keys file, in ~/.ssh/. So for example, the root user has the file in /root/.ssh/authorized_keys and any public key listed in that file will be able to connect to the server as the root user. Be aware the American [mis-]spelling of “authorized” – spell it [correctly] as “authorised” and you’ll not get any obvious errors, but the ssh key login won’t work either.

Distributing the SSH artifacts

Now we’re going to push this set of SSH files out to the .ssh folder of the target user on each node, which in this case is the root user. From a security point of view it’s probably better to use a non-root user for login and then sudo as required, but we’re keeping things simple (and less secure) to start with here. So the files in our folder are:

id_rsa – the private key of the key pair
id_rsa.pub – the public key of the key pair.

Strictly speaking this doesn’t need distributing to all nodes, but it’s conventional and handy to hold it alongside the private key.

authorized_keys – this is the file that the sshd daemon on each node will look at to validate an incoming login request’s offered private key, and so needs to hold the public key of anyone who is allowed to access the machine as this user.

[[email protected] ~]$ cat ~/.ssh/id_rsa.pub |ssh [email protected] 'cat >> .ssh/authorized_keys'
[email protected]'s password:
[[email protected] ~]$

At this point you’ll need to enter the password for the target user, but rejoice! This is the last time you’ll need to enter it as subsequent logins will be authenticated using the ssh keys that you’re now configuring.

c) Confirm Password less connectivity

If you are getting below error during password less connectivity

[[email protected] .ssh]$ ssh [email protected]
Agent admitted failure to sign using the key.
[email protected]'s password:

Then run this command

[[email protected] .ssh]$ export SSH_AUTH_SOCK=0

[[email protected] .ssh]$ ssh [email protected]
[[email protected] .ssh]$ ssh [email protected]

PDSH Command Example

To begin, I'll try to get the kernel version of a node by using its IP address:

[[email protected] ~]$ pdsh -w [email protected] uname -r
ec2-52-59-121-138: 4.4.11-23.53.amzn1.x86_64
[[email protected] ~]$

The -w option means I am specifying the node(s) that will run the command. In this case, I specified the IP address of the node (ec2-52-59-121-138). After the list of nodes, I add the command I want to run, which is uname -r in this case. Notice that pdsh starts the output line by identifying the node name.

1) Single Node Command Execution

To combine commands together that you send to each host or single host you can use the standard bash operator semicolon ;

[[email protected] ~]$ pdsh -w [email protected] uname -a;hostname ; who
ec2-52-28-150-229: Linux ip-172-31-16-42.eu-central-1.compute.internal 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
LinuxRedhatCluster
shaha tty1 2016-07-31 20:08 (:0)
shaha pts/0 2016-08-02 11:11 (:0.0)
shaha pts/1 2016-08-02 11:13 (:0.0)
shaha pts/2 2016-08-02 12:02 (:0.0)
shaha pts/3 2016-08-02 20:38 (:0.0)
shaha pts/4 2016-08-03 10:52 (:0.0)
shaha pts/5 2016-08-03 15:19 (:0.0)
shaha pts/6 2016-08-03 10:33 (:0.0)
shaha pts/7 2016-08-03 17:09 (:0.0)
shaha pts/8 2016-08-03 18:48 (:0.0)
shaha pts/9 2016-08-03 22:49 (:0.0)
[[email protected] ~]$

2) Multiple Node command execution

A very common way of using pdsh is to set the environment variable WCOLL to point to the file that contains the list of hosts you want to use in the pdsh command. For example, I created a subdirectory PDSH where I create a file hosts that lists the hosts I want to use:

[[email protected] ~]$ mkdir PDSH
[[email protected] ~]$ cd PDSH
[[email protected] PDSH]$ vi hosts

Insert all node in hostfile

[[email protected] PDSH]$ more hosts
[email protected]
[email protected]
[[email protected] PDSH]$

I'm only using two nodes: ec2-52-58-254-227.eu-central-1.compute.amazonaws.com and ec2-52-59-121-138.eu-central-1.compute.amazonaws.com . The first is my test system (like a cluster head node), and the second is my test compute node. You can put hosts in the file as you would on the command line separated by commas. Be sure not to put a blank line at the end of the file because pdsh will try to connect to it. You can put the environment variable WCOLL in your .bashrc file:

export WCOLL=/home/shaha/PDSH/hosts

As before, you can source your .bashrc file, or you can log out and log back in.

3) Specifying Hosts Command Execution

I won't list all the several other ways to specify a list of nodes. The simplest way is to specify the nodes on the command line is to use the -w option:

[[email protected] ~]$ pdsh -w [email protected],[email protected] uname -r
ec2-52-59-121-138: 4.4.11-23.53.amzn1.x86_64
ec2-52-58-254-227: 3.13.0-92-generic
[[email protected] ~]$

In this case, I specified the node names separated by commas. You can also use a range of hosts as follows:

[[email protected] PDSH]$ pdsh -w ^hosts uptime
[email protected]: 10:00:52 up 2 days, 16:48, 0 users, load average: 0.05, 0.04, 0.05
[email protected]: 10:00:50 up 2 days, 16:51, 0 users, load average: 0.00, 0.01, 0.05
[[email protected] PDSH]$

4) More Useful pdsh Commands

Now I can shift into second gear and try some fancier pdsh tricks. First, I want to run a more complicated command on all of the nodes . Notice that I put the entire command in quotes. This means the entire command is run on each node, including the first (cat /proc/cpuinfo) and second (grep bogomips , model ,cpu) parts.

[[email protected] PDSH]$ pdsh 'cat /proc/cpuinfo' | egrep 'bogomips|model|cpu'
[email protected]: cpu family : 6
[email protected]: model : 63
[email protected]: model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
[email protected]: cpu MHz : 2400.070
[email protected]: cpu cores : 1
[email protected]: cpuid level : 13
[email protected]: bogomips : 4800.14
[email protected]: cpu family : 6
[email protected]: model : 62
[email protected]: model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
[email protected]: cpu MHz : 2500.036
[email protected]: cpu cores : 1
[email protected]: cpuid level : 13
[email protected]: bogomips : 5000.07
[[email protected] PDSH]$

[[email protected] PDSH]$ pdsh vmstat 1 2
[email protected]: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
[email protected]: r b swpd free buff cache si so bi bo in cs us sy id wa st
[email protected]: 2 0 0 287424 122872 529200 0 0 1 5 10 20 0 0 100 0 0
[email protected]: procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
[email protected]: r b swpd free buff cache si so bi bo in cs us sy id wa st
[email protected]: 1 0 0 262924 153712 505960 0 0 1 5 13 19 0 0 100 0 0
[email protected]: 0 0 0 287504 122872 529232 0 0 0 0 34 74 0 0 100 0 0
[email protected]: 0 0 0 262924 153712 505992 0 0 0 0 18 20 0 0 100 0 0
[[email protected] PDSH]$

[[email protected] ~]$ pdsh -w [email protected][1-4] "date;sleep 5;date"
ec2-52-59-121-138-node01: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node03: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node04: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node02: Sat Nov 30 20:57:06 GMT 2016
ec2-52-59-121-138-node01: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node03: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node04: Sat Nov 30 20:57:11 GMT 2016
ec2-52-59-121-138-node02: Sat Nov 30 20:57:11 GMT 2016

Note the use of the quotation marks to enclose the entire command string. Without them the bash interpreter will take the ; as the delimiter of the local commands, and try to run the subsequent commands locally:

[[email protected] ~]$ pdsh -w [email protected][1-4] date;sleep 5;date
ec2-52-59-121-138-node03: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node04: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node02: Sat Aug 30 20:57:53 GMT 2016
ec2-52-59-121-138-node01: Sat Aug 30 20:57:53 GMT 2016
Sat 30 Aug 2016 20:58:00 GMT

5) Piping and File Redirects

Similar to combining commands above, you can pipe the output of commands, and you need to use quotation marks to enclose the whole command string.

[[email protected] ~]$ pdsh -w [email protected][01-4] "chkconfig|grep collectl"
ec2-52-59-121-138-node03: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node01: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node04: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ec2-52-59-121-138-node02: collectl 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Conclusion

Even experienced admins use parallel shell tools to understand the states of their systems. These tools are easily scriptable, so you can store the data in a flat file or in a database.

About Shaha Alam

I have Over 4+ years Experience in Telecom domain as Software Engineer with application Support Engineer & AIX / Solaris / Unix / Linux System Admin , Shell Scripting, SQL,Oracle Database , ETL Tools Affirm (Connectiva ) , Mediation IME . Experience on Linux/Unix, Shell Scripting, TCP/IP, Networking Protocol, HTTPS, SMPP, Monitoring Basics. Experience in Vmware , DNS, MAIL, LDAP, NIS , Raid ,Zone , Tomcat , Apache ,AWS.

Author Archive Page

Have anything to say?

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.