DISCLAIMER : Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility.If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.

Uniq command in Unix

Uniq command in unix or linux system is used to suppress the duplicate lines from a file. It discards all the successive identical lines except one from the input and writes the output.

The syntax of uniq command is 
uniq [option] filename

The options of uniq command are:
  • c : Count of occurrence of each line.
  • d : Prints only duplicate lines.
  • D : Print all duplicate lines
  • f : Avoid comparing first N fields.
  • i : Ignore case when comparing.
  • s : Avoid comparing first N characters.
  • u : Prints only unique lines.
  • w : Compare no more than N characters in lines
Uniq Command Examples:

First create the following example.txt file in your unix or linux operating system. 
# cat example.txt
Unix operating system
unix operating system
unix dedicated server
linux dedicated server

1. Suppress duplicate lines

The default behavior of the uniq command is to suppress the duplicate line. Note that, you have to pass sorted input to the uniq, as it compares only successive lines. 
# uniq example.txt
unix operating system
unix dedicated server
linux dedicated server

If the lines in the file are not in sorted order, then use the sort command and then pipe the output to the uniq command. 
# sort example.txt | uniq

2. Count of lines.

The -c option is used to find how many times each line occurs in the file. It prefixes each line with the count. 
# uniq -c example.txt
      2 unix operating system
      1 unix dedicated server
      1 linux dedicated server

3. Display only duplicate lines.

You can print only the lines that occur more than once in a file using the -d option. 
# uniq -d example.txt
unix operating system

# uniq -D example.txt
unix operating system
unix operating system

The -D option prints all the duplicate lines.

4. Skip first N fields in comparison.

The -f option is used to skip the first N columns in comparison. Here the fields are delimited by the space character. 
# uniq -f2 example.txt
unix operating system
unix dedicated server

In the above example the uniq command, just compares the last fields. For the first two lines, the last field contains the string "system". Uniq prints the first line and skips the second. Similarly it prints the third line and skips the fourth line.

5. Print only unique lines.

You can skip the duplicate lines and print only unique lines using the -u option 
# uniq -u example.txt
unix dedicated server
linux dedicated server
 
By No comments: