Usually when people say "shell scripting" they have on mind bash, ksh, sh, ash or similar Linux/UNIX scripting language. Scripting is another way to communicate with the computer. Using graphical windows interface (not matter windows or Linux) the user can move a mouse and click on the various objects like buttons, lists, checkboxes and so on. But it is a very inconvenient way which requires user participation and accuracy each time he would like to ask computer/server to do the same tasks - (let's say to convert photos or download new movies, mp3 etc). To make all these things easily accessible and automated we could use shell scripts.
Some programming languages like Pascal, FoxPro, C, Java needs to be compiled before they could be executed. They need an appropriate compiler to make our code to do some job.
The shell scripts are also like interpreted, but it is usually used to call external compiled programs. Then captures the outputs, exit codes and act accordingly.
One of the most popular shell scripting languages in the Linux world is the bash. And I think (this is my own opinion) this is because bash shell allows the user to easily navigate through the history commands (previously executed) by default, in opposite ksh which requires some tuning in .profile or remember some "magic" key combination to walk through history and amend commands.
OK, I think this is enough for introduction and I leave for you to judge which environment is most comfortable for you. This tutorial will concentrate on bash scripting. In the following examples, I will use the CentOS 6.6 and bash-4.1.2. Just make sure you have the same or greater version.
Shell Script Streams
The shell scripting is something similar to the conversation of several persons. Just imagine that all command is like the persons, who able to do something if you properly ask them. Let's say you would like to write a document. First of all, you need the paper, then you need to say the content to someone to write it, and finally, you would like to store it somewhere. Or you would like to build a house, so you will ask appropriate persons to clean up space. After they say "its done" then other engineers could build for you the walls. And finally, when engineers also tell "Its done" you can ask the painters to color your house. And what would happen if you ask the painters coloring your walls before they are built? I think they will start to complain. Almost all commands like the persons could speak and if they did its job without any issues they speak to "standard output". If they can't do what you are asking - they speaking to the "standard error". So finally all commands listening for you through "standard input".
Quick example- when you opening Linux terminal and writing some text - you speaking to bash through "standard input". So ask the bash shell who am i
[[email protected] ~]# blablabla <--- and again, you speaking through standard input -bash: blablabla: command not found <--- bash complaining through standard error
- standard input (stdin) - 0
- standard output (stdout) - 1
- standard error (stderr) - 2
If you really would like to know how to switch output command said something - you need to redirect (to use "greater than ">" symbol after command and stream index) that speech to file:
[[email protected]localhost ~]# blablabla 1> output.txt -bash: blablabla: command not found
[[email protected] ~]# cat output.txt [[email protected] ~]#
Sometimes commands produce stdout and stderr simultaneously. To redirect them to separate files we can use the following syntax:
command 1>out.txt 2>err.txt
command >out.txt 2>err.txt
OK, let's try to do something "bad". let's remove the file1 and folder1 with the rm command:
[[email protected] ~]# rm -vf folder1 file1 > out.txt 2>err.txt
Now check our output files:
As we see the streams were separated into different files. Sometimes it is not handy as usually, we want to see the sequence when the errors appeared - before or after some actions. For that we can redirect both streams to the same file:
command >>out_err.txt 2>>out_err.txt
Note: Please notice that i use ">>" instead of ">". It allows us to append file instead of overwriting.
We can redirect one stream to another:
command >out_err.txt 2>&1
Let me explain. All stdout of the command will be redirected to the out_err.txt. The errout will be redirected to the 1-st stream which (as I already explained above) will be redirected to the same file. Let see the example:
Looking at the combined output we can state that first of all rm command tried to remove the folder2 and it was not a success as Linux require the -r key for rm command to allow remove folders. At the second the file2 was removed. By providing the -v (verbose) key for the rm command we asking rm command to inform us about each removed file or folder.
This is almost all you need to know about redirection. I say almost because there is one more very important redirection which called "piping". By using | (pipe) symbol we usually redirecting stdout stream.
Let's say we have the text file:
[[email protected] ~]# cat text_file.txt This line does not contain H e l l o word This lilne contains Hello This also containd Hello This one no due to HELLO all capital Hello bash world!
and we need to find the lines in it with the words "Hello". Linux has the grep command for that:
[[email protected] ~]# grep Hello text_file.txt This lilne contains Hello This also containd Hello Hello bash world! [[email protected] ~]#
This is OK when we have file and would like to search for it. But what if we need to find something in the output of another command? Yes, of course, we can redirect the output to the file and then look in it:
[[email protected] ~]# fdisk -l>fdisk.out [[email protected] ~]# grep "Disk /dev" fdisk.out Disk /dev/sda: 8589 MB, 8589934592 bytes Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes [[email protected] ~]#
If you going to grep something with white spaces embrace that with " quotes!
Note: fdisk command shows information about Linux OS disk drives
As we see this way is not very handy as soon we will miss the space with temporary files. For that, we can use the pipes. They allow us to redirect one command stdout to another command stdin streams:
[[email protected] ~]# fdisk -l | grep "Disk /dev" Disk /dev/sda: 8589 MB, 8589934592 bytes Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes [[email protected] ~]#
As we see, we get the same result without any temporary files. We have redirected frisk stdout to the grep stdin.
Note: Pipe redirection is always from left to right.
There are several other redirections but we will speak about them later.
Displaying custom messages in the shell
As we already know usually communication with and within the shell is going as dialog. So let's create some real script which also will speak with us. It will allow you to learn some simple commands and better understand the scripting concept.
Imagine we are working in some company as help desk manager and we would like to create some shell script to register the call information: phone number, Username and a brief description of the issue. We going to store it in the plain text file data.txt for future statistics. The script itself should work in dialog way to make life easy for help desk workers. So, first of all, we need to display the questions. For displaying any messages there are echo and printf commands. Both of them displaying messages, but printf is more powerful as we can nicely form output to align it to the right, left or leave dedicated space for a message. Lets start from simple one. For file creation please use your favorite text editor (kate, nano, vi, ...) and create the file named note.sh with the command inside:
echo "Phone number ?"
echo -n "Phone number?"
#!/bin/bash echo -n "Phone number?" read phone echo -n "Name?" read -n name echo -n "Issue?" read issue
How to run/execute a script
After you have saved the file we can run it with bash command by providing our file as an argument:
[[email protected] ~]# bash note.sh Phone number ?
Actually to use this way for script execution is not handy. It would be more comfortable just execute the script without any bash command as a prefix. To make it executable we can use chmod command:
[[email protected] ~]# ls -la note.sh -rw-r--r--. 1 root root 22 Apr 23 20:52 note.sh [[email protected] ~]# chmod +x note.sh [[email protected] ~]# ls -la note.sh -rwxr-xr-x. 1 root root 22 Apr 23 20:52 note.sh [[email protected] ~]#
Note: ls command displays the files in the current folder. By adding the keys -la it will display a bit more information about files.
As we see, before chmod command execution, the script has only read (r) and write (w) permissions. After chmod +x it got execute (x) permissions. (More details about permissions I am going to describe in next article.) Now we can simply run it:
[[email protected] ~]# ./note.sh Phone number ?
Before script name I have added ./ combination. (dot) in the Unix world means current position (current folder), the / (slash) is the folder separator. (In Windows OS we use \ (backslash) for the same). So whole this combination means: "from the current folder execute the note.sh script". I think it will be more clear for you if I run this script with full path:
[[email protected] ~]# /root/note.sh Phone number ? [[email protected] ~]#
It also works.
Everything would be OK if all Linux users would have the same default shell. If we simply execute this script default user shell will be used to parse script content and run the commands. Different shells have a bit different syntax, internal commands, etc. So to guarantee the bash will be used for our script we should add #!/bin/bash as the first line. In this way default user shell will call /bin/bash and only then will execute following shell commands in the script:
[[email protected] ~]# cat note.sh #!/bin/bash echo "Phone number ?"
Only now we will be 100% sure that bash will be used to parse our script content. Let's move on.
Reading the inputs
After we have displayed the message script should wait for an answer from the user. There is the command read:
#!/bin/bash echo "Phone number ?" read phone
After execution script will wait for the user input until he presses the [ENTER] key:
Everything you have input will be stored in the variable phone. To display the value of the variable we can use the same echo command:
In bash shell, we using $ (dollar) sign as a variable indication, except when reading into a variable and few other moments (will describe later).
Ok, now we are ready to add the rest questions:
Using stream redirection
Perfect! There is left to redirect everything to the file data.txt. As a field separator, we going to use / (slash) symbol.
Note: You can choose any which you think is the best, bat be sure that content will not have these symbols inside. It will cause extra fields in the line.
Do not forget to use ">>" instead of ">" as we would like to append the output to the end of file!
[[email protected] ~]# tail -2 note.sh read issue echo "$phone/$name/$issue">>data.txt [[email protected] ~]# ./note.sh Phone number? 987 Name? Jimmy Issue? Keybord issue. [[email protected] ~]# cat data.txt 987/Jimmy/Keybord issue. [[email protected] ~]#
Bingo. Let's run once again:
Our file is growing. Let's add the date in the front of each line. This will be useful later when playing with data while calculating the statistic. For that we can use command date and give it some format as I do not like default one:
There are several ways to read the command output to the variable. In this simple situation, we will use ` (backquotes). Instead of backquotes we can also use $( ). Say for example now = $(date)
[[email protected] ~]# cat note.sh #!/bin/bash now=`date "+%Y.%m.%d %H:%M:%S"` echo "Phone number?" read phone echo "Name?" read name echo "Issue?" read issue echo "$now/$phone/$name/$issue">>data.txt [[email protected] ~]# ./note.sh Phone number? 123 Name? Jim Issue? Script hanging. [[email protected] ~]# cat data.txt 2015.04.23 21:38:56/123/Jim/Script hanging. [[email protected] ~]#
Hmmm... Our script looks a bit ugly. Let's prettify it a bit. If you would read manual about read command you would find that read command also could display some messages. For this we should use -p key and message:
[[email protected] ~]# cat note.sh #!/bin/bash now=`date "+%Y.%m.%d %H:%M:%S"` read -p "Phone number: " phone read -p "Name: " name read -p "Issue: " issue echo "$now/$phone/$name/$issue">>data.txt
Agree it looks much better!
[[email protected] ~]# ./note.sh Phone number: 321 Name: Susane Issue: Mouse was stolen [[email protected] ~]# cat data.txt 2015.04.23 21:38:56/123/Jim/Script hanging. 2015.04.23 21:43:50/321/Susane/Mouse was stolen [[email protected] ~]#
And the cursor is right after the message (not in new line) what makes a bit sense.
It's time to improve our script. If the user works all day with the calls it is not very handy to run it each time. Let's add all these actions in the never-ending loop:
[[email protected] ~]# cat note.sh #!/bin/bash while true do read -p "Phone number: " phone now=`date "+%Y.%m.%d %H:%M:%S"` read -p "Name: " name read -p "Issue: " issue echo "$now/$phone/$name/$issue">>data.txt done
I have swapped read phone and now=`date lines. This is because I would like to get the time right after the phone number will be entered. If I would leave it as the first line in the loop - the now variable will get the time right after the data was stored in the file. And it is not good as the next call could be after 20 mins or so.
[[email protected] ~]# ./note.sh Phone number: 123 Name: Jim Issue: Script still not works. Phone number: 777 Name: Daniel Issue: I broke my monitor Phone number: ^C [[email protected] ~]# cat data.txt 2015.04.23 21:38:56/123/Jim/Script hanging. 2015.04.23 21:43:50/321/Susane/Mouse was stolen 2015.04.23 21:47:55/123/Jim/Script still not works. 2015.04.23 21:48:16/777/Daniel/I broke my monitor [[email protected] ~]#
NOTE: To exit from the never-ending loop you can by pressing [Ctrl]+[C] keys. Shell will display ^ as the Ctrl key.
Using pipe redirection
Let's add more functionality to our "Frankenstein" I would like the script will display some statistic after each call. Lets say we want to see the how many times each number called us. For that we should cat the data.txt file:
[[email protected] ~]# cat data.txt 2015.04.23 21:38:56/123/Jim/Script hanging. 2015.04.23 21:43:50/321/Susane/Mouse was stolen 2015.04.23 21:47:55/123/Jim/Script still not works. 2015.04.23 21:48:16/777/Daniel/I broke my monitor 2015.04.23 22:02:14/123/Jimmy/New script also not working!!! [[email protected] ~]#
Now all this output we can redirect to the cut command to cut each line into the chunks (our delimiter "/") and print the second field:
[[email protected] ~]# cat data.txt | cut -d"/" -f2 123 321 123 777 123 [[email protected] ~]#
Now, this, output we can redirect to another command to sort:
[[email protected] ~]# cat data.txt | cut -d"/" -f2 | sort | uniq -c 3 123 1 321 1 777 [[email protected] ~]#
#!/bin/bash while true do read -p "Phone number: " phone now=`date "+%Y.%m.%d %H:%M:%S"` read -p "Name: " name read -p "Issue: " issue echo "$now/$phone/$name/$issue">>data.txt echo "===== We got calls from =====" cat data.txt | cut -d"/" -f2 | sort | uniq -c echo "--------------------------------" done
[[email protected] ~]# ./note.sh Phone number: 454 Name: Malini Issue: Windows license expired. ===== We got calls from ===== 3 123 1 321 1 454 1 777 -------------------------------- Phone number: ^C
The current scenario is going through well-known steps like:
- Display message
- Get user input
- Store values in the file
- Do something with stored data
But what if a user has several responsibilities and he needs sometimes to input data, sometimes to do statistic calculations, or might be to find something in stored data? For that, we need to implement switches/cases. In next article, i will show you how to use them and how to nicely form the output. It is useful while "drawing" the tables in the shell.