Methods to Use Uniq Command in Linux with Examples

uniq command linux
If you are a Linux user and your work involves with working with and manipulating text files and strings, then you should be already familiar with the uniq command, as it is most commonly used in that area.

For those who are not familiar with uniq command, it is a command line tool which is used to report or omit repeated strings or lines. This basically filter adjacent matching lines from INPUT (or standard input) and write to OUTPUT (or standard output). With no options, matching lines are merged to the first occurrence.

Below are few examples of usage of the uniq command

1) Omit duplicates

Executing the uniq commands without specifying any parameters simply omits duplicates and displays a unique string output.

[email protected]:~/Documents/files$cat file1 
Hello
Hello
How are you?
How are you?
Thank you
Thank you
[email protected]:~/Documents/files$ uniq file1 
Hello
How are you?
Thank you

2) Display number of repeated lines

With the -c parameter, it is possible to view the duplicate line count in a file

[email protected]:~/Documents/files$ cat file1 
Hello
Hello
How are you?
How are you?
Thank you
Thank you
[email protected]:~/Documents/files$ uniq -c file1 
      2 Hello
      2 How are you?
      2 Thank you

3) Print only the duplicates

By using -d parameter, we can select only the lines which have been duplicated inside a file

[email protected]:~/Documents/files$ cat file1 
Hello
Hello
Good morning
How are you?
How are you?
Thank you
Thank you
Bye
[email protected]:~/Documents/files$ uniq -d file1 
Hello
How are you?
Thank you

4) Ignore case when comparing

Normally when you use the uniq command it take the case of letters into consideration. But if you want to ignore the case, you can use -i parameter

[email protected]:~/Documents/files$ cat file1 
Hello
hello
How are you?
How are you?
Thank you
thank you
[email protected]:~/Documents/files$ uniq file1 
Hello
hello
How are you?
Thank you
thank you
[email protected]:~/Documents/files$ uniq -i file1 
Hello
How are you?
Thank you

5) Only print unique lines

If you only want to see the unique lines in a file, you can use -u parameter

[email protected]:~/Documents/files$ cat file1 
Hello
Hello
Good morning
How are you?
How are you?
Thank you
Thank you
Bye
[email protected]:~/Documents/files$ uniq -u file1 
Good morning
Bye

6) Sort and find duplicates

Sometimes duplicate entries may contain in different places of a files. In that case if we simply use the uniq command, it will not detect these duplicate entries in different lines. In that case we first need to sort the file and then we can find duplicates

[email protected]:~/Documents/files$ cat file1 
Adam
Sara
Frank
John
Ann
Matt
Harry
Ann
Frank
John
[email protected]:~/Documents/files$ sort file1 | uniq -c
      1 Adam
      2 Ann
      2 Frank
      1 Harry
      2 John
      1 Matt
      1 Sara

7) Save the output in another file

The output of our uniq command can be simply saved in another file as below

[email protected]:~/Documents/files$ cat file1 
Hello
Hello
How are you?
Good morning
Good morning
Thank you
[email protected]:~/Documents/files$ uniq -u file1 
How are you?
Thank you
[email protected]:~/Documents/files$ uniq -u file1 output 
[email protected]:~/Documents/files$ cat output 
How are you?
Thank you

8) Ignore characters

In order to ignore few characters at the beginning you can use -s parameter, but you need to specify the number of characters you need to ignore

[email protected]:~/Documents/files$ cat file1 
1apple
2apple
3pears
4banana
5banana
[email protected]:~/Documents/files$ uniq -s 1 file1 
1apple
3pears
4banana

If you have any questions or thoughts to share on this topic, use the feedback form

Read Also:

Nishali Wijesinghe 7:01 am

About Nishali Wijesinghe

Nishali is a Linux enthusiastic who works as a Linux System Engineer with 4+ years experience in this field. She is interested in Information Security and Linux Administration and also learning about new technologies.

Author's All Posts
Like to become part of Linoxide Team and contribute tips? Contact us here.

Comments

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.