文章作者:Tyan 博客:noahsnail.com | CSDN | 简书
在Linux系统处理数据时,经常会遇到删除重复文件的问题。例如,在进行图片分类任务时,希望删除训练数据中的重复图片。在Linux系统中,存在一个fdupes
命令可以查找并删除重复文件。
Fdupes是Adrian Lopez用C语言编写的Linux实用程序,它能够在给定的目录和子目录集中找到重复文件,Fdupes通过比较文件的MD5签名然后进行字节比较来识别重复文件。其比较顺序为:
大小比较 > 部分MD5签名比较 > 完整MD5签名比较 > 字节比较
以CentOS系统为例,fdupes
的安装命令为:
sudo yum install -y fdupes
删除重复文件,并且不需要询问用户:
$ fdupes -dN [folder_name]
其中,-d
参数表示保留一个文件,并删除其它重复文件,-N
与-d
一起使用,表示保留第一个重复文件并删除其它重复文件,不需要提示用户。
使用说明:
$ fdupes -h
Usage: fdupes [options] DIRECTORY...
-r --recurse for every directory given follow subdirectories
encountered within
-R --recurse: for each directory given after this option follow
subdirectories encountered within (note the ':' at
the end of the option, manpage for more details)
-s --symlinks follow symlinks
-H --hardlinks normally, when two or more files point to the same
disk area they are treated as non-duplicates; this
option will change this behavior
-n --noempty exclude zero-length files from consideration
-A --nohidden exclude hidden files from consideration
-f --omitfirst omit the first file in each set of matches
-1 --sameline list each set of matches on a single line
-S --size show size of duplicate files
-m --summarize summarize dupe information
-q --quiet hide progress indicator
-d --delete prompt user for files to preserve and delete all
others; important: under particular circumstances,
data may be lost when using this option together
with -s or --symlinks, or when specifying a
particular directory more than once; refer to the
fdupes documentation for additional information
-N --noprompt together with --delete, preserve the first file in
each set of duplicates and delete the rest without
prompting the user
-I --immediate delete duplicates as they are encountered, without
grouping into sets; implies --noprompt
-p --permissions don't consider files with different owner/group or
permission bits as duplicates
-o --order=BY select sort order for output and deleting; by file
modification time (BY='time'; default), status
change time (BY='ctime'), or filename (BY='name')
-i --reverse reverse order while sorting
-v --version display fdupes version
-h --help display this help message