Cloud DevOps Admin Guide

First create the following example.txt file in your unix or linux operating system.

# cat example.txt
Unix operating system
unix operating system
unix dedicated server
linux dedicated server

1. Suppress duplicate lines

The default behavior of the uniq command is to suppress the duplicate line. Note that, you have to pass sorted input to the uniq, as it compares only successive lines.

# uniq example.txt
unix operating system
unix dedicated server
linux dedicated server

If the lines in the file are not in sorted order, then use the sort command and then pipe the output to the uniq command.

# sort example.txt | uniq

2. Count of lines.

The -c option is used to find how many times each line occurs in the file. It prefixes each line with the count.

# uniq -c example.txt
      2 unix operating system
      1 unix dedicated server
      1 linux dedicated server

3. Display only duplicate lines.

You can print only the lines that occur more than once in a file using the -d option.

# uniq -d example.txt
unix operating system

# uniq -D example.txt
unix operating system
unix operating system

The -D option prints all the duplicate lines.

4. Skip first N fields in comparison.

The -f option is used to skip the first N columns in comparison. Here the fields are delimited by the space character.

# uniq -f2 example.txt
unix operating system
unix dedicated server

In the above example the uniq command, just compares the last fields. For the first two lines, the last field contains the string "system". Uniq prints the first line and skips the second. Similarly it prints the third line and skips the fourth line.

5. Print only unique lines.

You can skip the duplicate lines and print only unique lines using the -u option

# uniq -u example.txt
unix dedicated server
linux dedicated server