Awk Command in Linux

awk command Linux

AWK is one of the most powerful command in Linux. You can manage data and generate reports using the awk command. It also allows us to use logical operation, variables, print functions and many more. AWK stands for "Aho, Weinberger and Kernighan" and is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches the specified pattern and then perform associated actions. It reads from a file or from its standard input and outputs to its standard output. For each line, it matches with given pattern in the given order, if matches perform the corresponding action.

Features

• Views a text file as records and fields
• It has variables, conditionals and loops
• Has arithmetic and string operators
• Can generate formatted reports
• Read and edit text from a string or file

In this tutorial, we will take a look into the AWK Linux command with examples and see what it can do.

Basix Syntax of AWK

The basic syntax of AWK command is shown below:

awk options program input-file

A brief explanation of each option is shown below:

• -F fs : Used to specify a file separator.
• -f file : Used to specify a file that contains awk script.
• -v var=value : USed to declare a variable.

We will use the following text file as the input file for all examples in this article:

cat > contents.txt

hitesh engineer sales 30000
jayesh director account 25000
vyom manager purchase 20000
bhavesh engineer sales 30000
rajesh directory sales 40000
niraj clerk account 20000
jay peon purchase 23000
deep clerk sales 20000

Now, let's check the content of the file named 'contents.txt' with the AWK command:

awk '{print}' contents.txt

This will print the content of the file as shown below:

hitesh engineer sales 30000
jayesh director account 25000
vyom manager purchase 20000
bhavesh engineer sales 30000
rajesh directory sales 40000
niraj clerk account 20000
jay peon purchase 23000
deep clerk sales 20000

In the above example, no pattern is given so it will print the whole file.

Now, print all lines which match with the pattern "sales":

awk '/sales/ {print}' contents.txt

This will print all lines that contains the word "sales" as shown below:

hitesh engineer sales 30000
bhavesh engineer sales 30000
rajesh directory sales 40000
deep clerk sales 20000

Variables in AWK

AWK comes with some built-in variables that are used to break a line of text into individual words or pieces called fields. Some of them are shown below:

• $0 : Used for the entire line.
• $1 : Used for the first field.
• $2 : Used for the second field.
• $n : Used for the nth field.
• NR : Used to specify the total number of current records.
• NF : Used to specify the total number of fields in the record.
• FS : It contains the field separator character and is used to divide fields on the input line.
• RS : It stores the current record separator character.
• OFS : It stores the output field separator and is used to separates the fields when Awk prints them.
• ORS : It stores the output record separator and is used to separates the output lines when Awk prints them.

Now, print the field no 1 and 3 from the file contents.txt use the following syntax:

awk '{print $1,$3}' contents.txt

You should see only first and third field of the file contents.txt:

hitesh sales
jayesh account
vyom purchase
bhavesh sales
rajesh sales
niraj account
jay purchase
deep sales

You can use NR with the AWK command to print all the lines along with the line number:

awk '{print NR,$0}' contents.txt

Output:

1 hitesh engineer sales 30000
2 jayesh director account 25000
3 vyom manager purchase 20000
4 bhavesh engineer sales 30000
5 rajesh directory sales 40000
6 niraj clerk account 20000
7 jay peon purchase 23000
8 deep clerk sales 20000

You can use NF to display the last field and $1 to display the first field:

awk '{print $1,$NF}' contents.txt

This will print the first and last field of the file contents.txt:

hitesh 30000
jayesh 25000
vyom 20000
bhavesh 30000
rajesh 40000
niraj 20000
jay 23000
deep 20000

To print the line number from 2 to 5 use the NR variable as shown below:

awk 'NR==2, NR==5 {print NR,$0}' contents.txt

Output:

2 jayesh director account 25000
3 vyom manager purchase 20000
4 bhavesh engineer sales 30000
5 rajesh directory sales 40000

To count the number of the lines in file contents.txt using the NR:

awk 'END { print NR } ' contents.txt

You should see the following output:

8

BEGIN and END Blocks

There are also optional BEGIN and END blocks that can contain commands to execute before and after the file processing, respectively. The BEGIN block is used to perform actions before records are processed while END block is used to perform actions after records are processed.

The basic syntax to use BEGIN and END blocks with AWK command is shown below:

awk 'BEGIN { action; } /search/ { action; } END { action; }' input_file

You can print information about the fields you are printing with BEGIN and END blocks.
The following example will print the message before and after processing the second field of each record in file contents.txt:

awk 'BEGIN { print "Start Process." }; { print $2 }; END { print "End Process." }' contents.txt

Output:

Start Process.
engineer
director
manager
engineer
directory
clerk
peon
clerk
End Process.

You can also use the BEGIN and END blocks to transform the data from the file and convert into a table. The following example will convert the file /etc/passwd in to table:

awk 'BEGIN { FS=":"; print "User\t\tUID\t\tGID\t\tHome\t\tShell\n--------------"; } {print $1,"\t\t",$3,"\t\t",$4,"\t\t",$6,"\t\t",$7;} END { print "---------\nFile Complete" }' /etc/passwd

Output:

User		UID		GID		Home		Shell
--------------
root 		 0 		 0 		 /root 		 /bin/bash
daemon 		 1 		 1 		 /usr/sbin 		 /usr/sbin/nologin
bin 		 2 		 2 		 /bin 		 /usr/sbin/nologin
sys 		 3 		 3 		 /dev 		 /usr/sbin/nologin
sync 		 4 		 65534 		 /bin 		 /bin/sync
games 		 5 		 60 		 /usr/games 		 /usr/sbin/nologin
man 		 6 		 12 		 /var/cache/man 		 /usr/sbin/nologin
lp 		 7 		 7 		 /var/spool/lpd 		 /usr/sbin/nologin
mail 		 8 		 8 		 /var/mail 		 /usr/sbin/nologin
news 		 9 		 9 		 /var/spool/news 		 /usr/sbin/nologin
uucp 		 10 		 10 		 /var/spool/uucp 		 /usr/sbin/nologin
proxy 		 13 		 13 		 /bin 		 /usr/sbin/nologin
www-data 		 33 		 33 		 /var/www 		 /usr/sbin/nologin
---------
File Complete

Conditional Search

The AWK command also supports several conditional statements including, if, while loop, for loop and many more. This will helps you to fetch lines that matches a specific condition.

The following example will use "if" condition to print all lines that contain "sales" in the third field:

awk '{ if ($3 ~ /sales/) print}' contents.txt

Output:

hitesh engineer sales 30000
bhavesh engineer sales 30000
rajesh directory sales 40000
deep clerk sales 20000

The following example will use the "for" loop to prints the first three fields of each record, one per line.

awk '{ for (i = 1; i <= 3; i++) print $i }' contents.txt

Output:

hitesh
engineer
sales
jayesh
director
account
vyom
manager
purchase
bhavesh
engineer
sales
rajesh
directory
sales
niraj
clerk
account
jay
peon
purchase
deep
clerk
sales

The following example will use the "while" loop to prints the first two fields of each record, one per line.

awk '{ i = 1;  while ( i <= 2 ) {  print $i i++ } }' contents.txt

Output:

hitesh1
engineer2
jayesh1
director2
vyom1
manager2
bhavesh1
engineer2
rajesh1
directory2
niraj1
clerk2
jay1
peon2
deep1
clerk2

Processing Output from Other Commands

You can also use AWK command to parse the output from the other command instead of specifying a filename. The "ip a" command print the information about system IP, Mac address, and other network-related information as shown below:

ip a s wlan0

Output:

2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:bb:58:9c:f5:55 brd ff:ff:ff:ff:ff:ff
    inet 172.20.10.3/28 brd 172.20.10.15 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 2401:4900:1d71:ef26:8846:95b2:4ca8:aa7d/64 scope global temporary dynamic 
       valid_lft 600574sec preferred_lft 81574sec
    inet6 2401:4900:1d71:ef26:4ebb:58ff:fe9c:f555/64 scope global dynamic 
       valid_lft forever preferred_lft forever
    inet6 fe80::4ebb:58ff:fe9c:f555/64 scope link 
       valid_lft forever preferred_lft forever

Now, you can use AWK command to print only the IP address of the system as shown below:

ip a s wlan0 | awk -F '[\/ ]+' '/inet / {print $3}

You should see the following output:

172.20.10.3

AWK Command Advance Examples

Example 1: The AWK command allows you to print the lines with some specified count of characters. For example, print lines with more than 27 characters, use the below command:

awk 'length($0) > 27' contents.txt

Output:

jayesh director account 25000
bhavesh engineer sales 30000
rajesh directory sales 40000

Example 2: Check the cube of the given number

To print the cube of the given number till 5, run the following command:

awk 'BEGIN { for(i=1; i<=5; i++) print "Cube of",i,"is",i*i*i; }'

Output:

Cube of 1 is 1
Cube of 2 is 8
Cube of 3 is 27
Cube of 4 is 64
Cube of 5 is 125

Example 3: Count the number of lines in the specified file

You can check the number of lines in the specified file and print it using the following command:

awk 'END { print NR }' contents.txt

Output:

8

Example 4: Find the longest line in the given file and print the character

You can find the longest line in the given file and print the character of that line using the following command:

awk '{ if (length($0) > max) max = length($0) } END { print max }' contents.txt

Output

29

Example 5: Sort the first column of a given file

To sort and print the first column of the file contents.txt, run the following command:

awk -F: '{ print $1 }' contents.txt | sort

Output:

bhavesh engineer sales 30000
deep clerk sales 20000
hitesh engineer sales 30000
jayesh director account 25000
jay peon purchase 23000
niraj clerk account 20000
rajesh directory sales 40000
vyom manager purchase 20000

Example 6: Print the even-numbered lines

To print only even-numbered lines in the file contents.txt, run the following command:

awk 'NR % 2 == 0' contents.txt

Output:

jayesh director account 25000
bhavesh engineer sales 30000
niraj clerk account 20000
deep clerk sales 20000

Example 7: Change the field separator

You can change the filed separator from space to | and print it with the following command:

awk 'BEGIN{OFS="|"}{print $1,$2,$3,$4}' contents.txt

Output:

hitesh|engineer|sales|30000
jayesh|director|account|25000
vyom|manager|purchase|20000
bhavesh|engineer|sales|30000
rajesh|directory|sales|40000
niraj|clerk|account|20000
jay|peon|purchase|23000
deep|clerk|sales|20000

Conclusion

In this tutorial, you learned how to use the AWK command to match the specified patterns and then perform the associated actions. I hope you have a clear idea of how to use AWK command to manipulate, format, and selectively print text files.

Hitesh Jethva 2:50 pm

Comments

Your email address will not be published. Required fields are marked *