First Step Guide to Learn Shell Scripting

myfirstshellscript screenshot

Usually when people say "shell scripting" they have on mind bash, ksh, sh, ash or similar Linux/UNIX scripting language. Scripting is another way to communicate with the computer. Using graphical windows interface (not matter windows or Linux) the user can move a mouse and click on the various objects like buttons, lists, checkboxes and so on. But it is a very inconvenient way which requires user participation and accuracy each time he would like to ask computer/server to do the same tasks - (let's say to convert photos or download new movies, mp3 etc). To make all these things easily accessible and automated we could use shell scripts.

Some programming languages like Pascal, FoxPro, C, Java needs to be compiled before they could be executed. They need an appropriate compiler to make our code to do some job.

Another programming languages like PHP, JavaScript, VisualBasic do not need a compiler. So they need interpreters and we could run our program without compiling the code.

The shell scripts are also like interpreted, but it is usually used to call external compiled programs. Then captures the outputs, exit codes and act accordingly.

One of the most popular shell scripting languages in the Linux world is the bash. And I think (this is my own opinion) this is because bash shell allows the user to easily navigate through the history commands (previously executed) by default, in opposite ksh which requires some tuning in .profile or remember some "magic" key combination to walk through history and amend commands.

OK, I think this is enough for introduction and I leave for you to judge which environment is most comfortable for you. This tutorial will concentrate on bash scripting. In the following examples, I will use the CentOS 6.6 and bash-4.1.2. Just make sure you have the same or greater version.

Shell Script Streams

The shell scripting is something similar to the conversation of several persons. Just imagine that all command is like the persons,  who able to do something if you properly ask them. Let's say you would like to write a document. First of all, you need the paper, then you need to say the content to someone to write it, and finally, you would like to store it somewhere. Or you would like to build a house, so you will ask appropriate persons to clean up space. After they say "its done" then other engineers could build for you the walls. And finally, when engineers also tell "Its done" you can ask the painters to color your house. And what would happen if you ask the painters coloring your walls before they are built? I think they will start to complain. Almost all commands like the persons could speak and if they did its job without any issues they speak to "standard output". If they can't do what you are asking - they speaking to the "standard error". So finally all commands listening for you through "standard input".

Quick example- when you opening Linux terminal and writing some text - you speaking to bash through "standard input". So ask the bash shell who am i

Now let's ask something that bash will not understand us:
[[email protected] ~]# blablabla           <--- and again, you speaking through standard input
-bash: blablabla: command not found     <--- bash complaining through standard error
The first word before ":" usually is the command which complaining to you. Actually, each of these streams has their own index number:
  • standard input (stdin) - 0
  • standard output (stdout) - 1
  • standard error (stderr) - 2

If you really would like to know how to switch output command said something - you need to redirect (to use "greater than ">" symbol after command and stream index) that speech to file:

[[email protected] ~]# blablabla 1> output.txt
-bash: blablabla: command not found
In this example, we tried to redirect 1 (stdout) stream to file named output.txt. Let's look does to the content of that file. We use the command cat for that:
[[email protected] ~]# cat output.txt
[[email protected] ~]#
Seams that is empty. Ok now let's try to redirect 2 (stderr) stream:
[[email protected] ~]# blablabla 2> error.txt
[[email protected] ~]#
Ok, we see that complains gone. Let's check the file:
[[email protected] ~]# cat error.txt
-bash: blablabla: command not found
[[email protected] ~]#
Exactly! We see that all complaints were recorded the errors.txt file.

Sometimes commands produce stdout and stderr simultaneously. To redirect them to separate files we can use the following syntax:

command 1>out.txt 2>err.txt
To shorten this syntax a bit we can skip the "1" as by default the stdout stream will be redirected:
command >out.txt 2>err.txt

OK, let's try to do something "bad". let's remove the file1 and folder1 with the rm command:

[[email protected] ~]# rm -vf folder1 file1 > out.txt 2>err.txt

Now check our output files:

[[email protected] ~]# cat out.txt
removed `file1'
[[email protected] ~]# cat err.txt
rm: cannot remove `folder1': Is a directory
[[email protected] ~]#

As we see the streams were separated into different files. Sometimes it is not handy as usually, we want to see the sequence when the errors appeared - before or after some actions. For that we can redirect both streams to the same file:

command >>out_err.txt 2>>out_err.txt

Note: Please notice that i use ">>" instead of ">". It allows us to append file instead of overwriting.

We can redirect one stream to another:

command >out_err.txt 2>&1

Let me explain. All stdout of the command will be redirected to the out_err.txt. The errout will be redirected to the 1-st stream which (as I already explained above) will be redirected to the same file. Let see the example:

[[email protected] ~]# rm -fv folder2 file2 >out_err.txt 2>&1
[[email protected] ~]# cat out_err.txt
rm: cannot remove `folder2': Is a directory
removed `file2'
[[email protected] ~]#

Looking at the combined output we can state that first of all rm command tried to remove the folder2 and it was not a success as Linux require the -r key for rm command to allow remove folders. At the second the file2 was removed. By providing the -v (verbose) key for the rm command we asking rm command to inform us about each removed file or folder.

This is almost all you need to know about redirection. I say almost because there is one more very important redirection which called "piping". By using | (pipe) symbol we usually redirecting stdout stream.

Let's say we have the text file:

[[email protected] ~]# cat text_file.txt
This line does not contain H e l l o  word
This lilne contains Hello
This also containd Hello
This one no due to HELLO all capital
Hello bash world!

and we need to find the lines in it with the words "Hello". Linux has the grep command for that:

[[email protected] ~]# grep Hello text_file.txt
This lilne contains Hello
This also containd Hello
Hello bash world!
[[email protected] ~]#

This is OK when we have file and would like to search for it. But what if we need to find something in the output of another command? Yes, of course, we can redirect the output to the file and then look in it:

[[email protected] ~]# fdisk -l>fdisk.out
[[email protected] ~]# grep "Disk /dev" fdisk.out
Disk /dev/sda: 8589 MB, 8589934592 bytes
Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes
Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes
[[email protected] ~]#

If you going to grep something with white spaces embrace that with " quotes!

Note:  fdisk command shows information about Linux OS disk drives

As we see this way is not very handy as soon we will miss the space with temporary files. For that, we can use the pipes. They allow us to redirect one command stdout to another command stdin streams:

[[email protected] ~]# fdisk -l | grep "Disk /dev"
Disk /dev/sda: 8589 MB, 8589934592 bytes
Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes
Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes
[[email protected] ~]#

As we see, we get the same result without any temporary files. We have redirected frisk stdout to the grep stdin.

Note: Pipe redirection is always from left to right.

There are several other redirections but we will speak about them later.

Displaying custom messages in the shell

As we already know usually communication with and within the shell is going as dialog. So let's create some real script which also will speak with us. It will allow you to learn some simple commands and better understand the scripting concept.

Imagine we are working in some company as help desk manager and we would like to create some shell script to register the call information: phone number, Username and a brief description of the issue. We going to store it in the plain text file data.txt for future statistics. The script itself should work in dialog way to make life easy for help desk workers. So, first of all, we need to display the questions. For displaying any messages there are echo and printf commands. Both of them displaying messages, but printf is more powerful as we can nicely form output to align it to the right, left or leave dedicated space for a message. Lets start from simple one. For file creation please use your favorite text editor (kate, nano, vi, ...) and create the file named note.sh with the command inside:

echo "Phone number ?"
or
echo -n "Phone number?"

Example

#!/bin/bash
 echo -n "Phone number?"
 read phone
 echo -n "Name?"
 read -n name
 echo -n "Issue?"
 read issue

How to run/execute a script

After you have saved the file we can run it with bash command by providing our file as an argument:

[[email protected] ~]# bash note.sh
Phone number ?

Actually to use this way for script execution is not handy. It would be more comfortable just execute the script without any bash command as a prefix. To make it executable we can use chmod command:

[[email protected] ~]# ls -la note.sh
-rw-r--r--. 1 root root 22 Apr 23 20:52 note.sh
[[email protected] ~]# chmod +x note.sh
[[email protected] ~]# ls -la note.sh
-rwxr-xr-x. 1 root root 22 Apr 23 20:52 note.sh
[[email protected] ~]#

set permission script file

Note: ls command displays the files in the current folder. By adding the keys -la it will display a bit more information about files.

As we see, before chmod command execution, the script has only read (r) and write (w) permissions. After chmod +x it got execute (x) permissions. (More details about permissions I am going to describe in next article.) Now we can simply run it:

[[email protected] ~]# ./note.sh
Phone number ?

Before script name I have added ./ combination. (dot) in the Unix world means current position (current folder), the / (slash) is the folder separator. (In Windows OS we use \ (backslash) for the same). So whole this combination means: "from the current folder execute the note.sh script". I think it will be more clear for you if I run this script with full path:

[[email protected] ~]# /root/note.sh
Phone number ?
[[email protected] ~]#

It also works.

Everything would be OK if all Linux users would have the same default shell. If we simply execute this script default user shell will be used to parse script content and run the commands. Different shells have a bit different syntax, internal commands, etc. So to guarantee the bash will be used for our script we should add #!/bin/bash as the first line. In this way default user shell will call /bin/bash and only then will execute following shell commands in the script:

[[email protected] ~]# cat note.sh
#!/bin/bash
echo "Phone number ?"

Only now we will be 100% sure that bash will be used to parse our script content. Let's move on.

Reading the inputs

After we have displayed the message script should wait for an answer from the user. There is the command read:

#!/bin/bash
echo "Phone number ?"
read phone

After execution script will wait for the user input until he presses the [ENTER] key:

[[email protected] ~]# ./note.sh
Phone number ?
12345                               <--- here is my input
[[email protected] ~]#

Everything you have input will be stored in the variable phone. To display the value of the variable we can use the same echo command:

[[email protected] ~]# cat note.sh
#!/bin/bash
echo "Phone number ?"
read phone
echo "You have entered $phone as a phone number"
[[email protected] ~]# ./note.sh
Phone number ?
123456
You have entered 123456 as a phone number
[[email protected] ~]#

In bash shell, we using $ (dollar) sign as a variable indication, except when reading into a variable and few other moments (will describe later).

Ok, now we are ready to add the rest questions:

#!/bin/bash
echo "Phone number?"
read phone
echo "Name?"
read name
echo "Issue?"
read issue

[[email protected] ~]# ./note.sh
Phone number?
123
Name?
Jim
Issue?
script is not working.
[[email protected] ~]#

Using stream redirection

Perfect! There is left to redirect everything to the file data.txt. As a field separator, we going to use / (slash) symbol.

Note: You can choose any which you think is the best, bat be sure that content will not have these symbols inside. It will cause extra fields in the line.

Do not forget to use ">>" instead of ">" as we would like to append the output to the end of file!

[[email protected] ~]# tail -2 note.sh
read issue
echo "$phone/$name/$issue">>data.txt
[[email protected] ~]# ./note.sh
Phone number?
987
Name?
Jimmy
Issue?
Keybord issue.
[[email protected] ~]# cat data.txt
987/Jimmy/Keybord issue.
[[email protected] ~]#
Note: The command tail displays the last -n lines of the file.

Bingo. Let's run once again:

[[email protected] ~]# ./note.sh
Phone number?
556
Name?
Janine
Issue?
Mouse was broken.
[[email protected] ~]# cat data.txt
987/Jimmy/Keybord issue.
556/Janine/Mouse was broken.
[[email protected] ~]#

Our file is growing. Let's add the date in the front of each line. This will be useful later when playing with data while calculating the statistic. For that we can use command date and give it some format as I do not like default one:

[[email protected]ost ~]# date
Thu Apr 23 21:33:14 EEST 2015                      <---- default output of date command
[[email protected] ~]# date "+%Y.%m.%d %H:%M:%S"
2015.04.23 21:33:18                               <---- formated output

There are several ways to read the command output to the variable. In this simple situation, we will use ` (backquotes). Instead of backquotes we can also use $( ). Say for example now = $(date)

[[email protected] ~]# cat note.sh
#!/bin/bash
now=`date "+%Y.%m.%d %H:%M:%S"`
echo "Phone number?"
read phone
echo "Name?"
read name
echo "Issue?"
read issue
echo "$now/$phone/$name/$issue">>data.txt
[[email protected] ~]# ./note.sh
Phone number?
123
Name?
Jim
Issue?
Script hanging.
[[email protected] ~]# cat data.txt
2015.04.23 21:38:56/123/Jim/Script hanging.
[[email protected] ~]#

Hmmm... Our script looks a bit ugly. Let's prettify it a bit. If you would read manual about read command you would find that read command also could display some messages. For this we should use -p key and message:

[[email protected] ~]# cat note.sh
#!/bin/bash
now=`date "+%Y.%m.%d %H:%M:%S"`
read -p "Phone number: " phone
read -p "Name: " name
read -p "Issue: " issue
echo "$now/$phone/$name/$issue">>data.txt
You can find a lots of interesting about each command directly from the console. Just type: man read, man echo, mandate, man...

Agree it looks much better!

[[email protected] ~]# ./note.sh
Phone number: 321
Name: Susane
Issue: Mouse was stolen
[[email protected] ~]# cat data.txt
2015.04.23 21:38:56/123/Jim/Script hanging.
2015.04.23 21:43:50/321/Susane/Mouse was stolen
[[email protected] ~]#

And the cursor is right after the message (not in new line) what makes a bit sense.

Loop

It's time to improve our script. If the user works all day with the calls it is not very handy to run it each time. Let's add all these actions in the never-ending loop:

[[email protected] ~]# cat note.sh
#!/bin/bash
while true
do
        read -p "Phone number: " phone
        now=`date "+%Y.%m.%d %H:%M:%S"`
        read -p "Name: " name
        read -p "Issue: " issue
        echo "$now/$phone/$name/$issue">>data.txt
done

I have swapped read phone and now=`date lines. This is because I would like to get the time right after the phone number will be entered. If I would leave it as the first line in the loop - the now variable will get the time right after the data was stored in the file. And it is not good as the next call could be after 20 mins or so.

[[email protected] ~]# ./note.sh
Phone number: 123
Name: Jim
Issue: Script still not works.
Phone number: 777
Name: Daniel
Issue: I broke my monitor
Phone number: ^C
[[email protected] ~]# cat data.txt
2015.04.23 21:38:56/123/Jim/Script hanging.
2015.04.23 21:43:50/321/Susane/Mouse was stolen
2015.04.23 21:47:55/123/Jim/Script still not works.
2015.04.23 21:48:16/777/Daniel/I broke my monitor
[[email protected] ~]#

NOTE: To exit from the never-ending loop you can by pressing [Ctrl]+[C] keys. Shell will display ^ as the Ctrl key.

Using pipe redirection

Let's add more functionality to our "Frankenstein" I would like the script will display some statistic after each call. Lets say we want to see the how many times each number called us. For that we should cat the data.txt file:

[[email protected] ~]# cat data.txt
2015.04.23 21:38:56/123/Jim/Script hanging.
2015.04.23 21:43:50/321/Susane/Mouse was stolen
2015.04.23 21:47:55/123/Jim/Script still not works.
2015.04.23 21:48:16/777/Daniel/I broke my monitor
2015.04.23 22:02:14/123/Jimmy/New script also not working!!!
[[email protected] ~]#

Now all this output we can redirect to the cut command to cut each line into the chunks (our delimiter "/") and print the second field:

[[email protected] ~]# cat data.txt | cut -d"/" -f2
123
321
123
777
123
[[email protected] ~]#

Now, this, output we can redirect to another command to sort:

[[email protected] ~]# cat data.txt | cut -d"/" -f2|sort
123
123
123
321
777
[[email protected] ~]#
and leave only unique lines.  To count unique entries just add -c key for uniq command:
[[email protected] ~]# cat data.txt | cut -d"/" -f2 | sort | uniq -c
    3 123
    1 321
    1 777
[[email protected] ~]#
Just add this to end of our loop:
#!/bin/bash
while true
do
        read -p "Phone number: " phone
        now=`date "+%Y.%m.%d %H:%M:%S"`
        read -p "Name: " name
        read -p "Issue: " issue
        echo "$now/$phone/$name/$issue">>data.txt
        echo "===== We got calls from ====="
        cat data.txt | cut -d"/" -f2 | sort | uniq -c
        echo "--------------------------------"
done

Run it:

[[email protected] ~]# ./note.sh
Phone number: 454
Name: Malini
Issue: Windows license expired.
===== We got calls from =====
    3 123
    1 321
    1 454
    1 777
--------------------------------
Phone number: ^C

running script

The current scenario is going through well-known steps like:

  • Display message
  • Get user input
  • Store values in the file
  • Do something with stored data

But what if a user has several responsibilities and he needs sometimes to input data, sometimes to do statistic calculations, or might be to find something in stored data? For that, we need to implement switches/cases. In next article, i will show you how to use them and how to nicely form the output. It is useful while "drawing" the tables in the shell.

Petras Liumparas 9:28 am

Have anything to say?

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.

1 Comment