Linux Shell Script To Delete Duplicate Files

May 19, 2013 | By
| 2 Replies More

Shell Script is looking for duplicated names in files withing sub-directories, and after check md5sum. If md5sum of the files same then we conclude its duplicated. This helps system administrator to delete unnecessary copy to reduce used space. Script also ask user to enter directory where to search, and check if input is empty.

Linux Shell Script


#file, where we will store full list of files.

#we ask user to enter directory where search for duplicated files
echo -n "Please enter directory where to search for duplicated files: "

#we read user input
while read dir


#we check if user input is not empty
test -z "$dir" && {

#if user input empty we ask once more to enter directory
echo -n "Please enter directory: "



#if directory entered, exit from while loop


#getting list of files inside entered directory
find $dir -type f -print > $ListOfFiles

#writing list of files to variable
FileList=`cat $ListOfFiles`

#we get number of files
count=`wc -l $ListOfFiles| awk '{print $1}'`


#we get files one by one
for file in $FileList


#just make this variable empty for every loop

#we need to get all non-proceeded files
let tailvalue=$count-$i

#we get only filename, without path
filename=$(basename $file)

#getting list of un-proceeded files, and we check if there is file with same filename
samefiles=`tail -${tailvalue} $ListOfFiles | grep $filename`

#starting loop for all same files
for samefile in $samefiles


#we get md5sum of filename with same name
msf=`md5sum $samefile | awk '{print $1}'`

#we get md5sum of original file
ms=`md5sum $file | awk '{print $1}'`

#we compare md5sums
if [ "$msf" = "$ms" ]; then

#if md5sums equal, we tell user about duplicated files
echo "File $file duplicated to $samefile"

#end of if loop

#end of while loop

#increase counter by 1
let i=$i+1


Script Output


Please enter directory where to search for duplicated files: /tmp

File /tmp/1/user.list duplicated to /tmp/user.list

shell script delete duplicate files


Free Linux Ebook to Download

Comments (2)

Trackback URL | Comments RSS Feed

  1. bardy says:

    Good tips! I also know another tool which is very usefull is cases like this, DuplicateFilesDeleter. Thanks! :)

Leave a Reply

All comments are subject to moderation.