{"slug": "h2oai-jdupes-h2o-ai-fork-of-jdupes", "title": "H2oai/jdupes: H2O.ai fork of jdupes", "summary": "H2O.ai has published a fork of jdupes, the duplicate-file scanner that identifies exact matches and can delete, hard link, symlink or block-level deduplicate them. The upstream jdupes documentation states the tool is over 7 times faster than fdupes-1.51 on average across tested data sets, ships a native Windows port, and prioritizes data loss avoidance over aggressive matching, with dangerous partial-match options such as -Q and -T requiring explicit expert use. The fork gives H2O.ai its own maintained copy of the utility rather than changing the underlying deduplication behavior.", "body_md": "jdupes is a program for identifying and taking actions upon duplicate files such as deleting, hard linking, symlinking, and block-level deduplication (also known as \"dedupe\" or \"reflink\"). It is faster than most other duplicate scanners. It prioritizes data safety over performance while also giving expert users access to advanced (and sometimes dangerous) features.\n\nPlease consider financially supporting continued development of jdupes using the links on my home page (Ko-fi, PayPal, SubscribeStar, etc.):\n\nThe biggest reason is raw speed. In testing on various data sets, jdupes is over 7 times faster than fdupes-1.51 on average.\n\njdupes provides a native Windows port. Most duplicate scanners built on Linux and other UNIX-like systems do not compile for Windows out-of-the-box and even if they do, they don't support Unicode and other Windows-specific quirks and features.\n\njdupes is generally stable. All releases of jdupes are compared against a known working reference versions of fdupes or jdupes to be certain that output does not change. You get the benefits of an aggressive development process without putting your data at increased risk.\n\nCode in jdupes is written with data loss avoidance as the highest priority. If a choice must be made between being aggressive or careful, the careful way is always chosen.\n\njdupes includes features that are not always found elsewhere. Examples of such\nfeatures include block-level data deduplication and control over which file is\nkept when a match set is automatically deleted. jdupes is not afraid of\ndropping features of low value; a prime example is the `-1` switch which\noutputs all matches in a set on one line, a feature which was found to be\nuseless in real-world tests and therefore thrown out.\n\nWhile jdupes maintains some degree of compatibility with fdupes from which it was originally derived, there is no guarantee that it will continue to maintain such compatibility in the future. However, compatibility will be retained between minor versions, i.e. jdupes-1.6 and jdupes-1.6.1 should not have any significant differences in results with identical command lines.\n\nIf the program eats your dog or sets fire to your lawn, the authors cannot be held responsible. If you notice a bug, please report it.\n\nPlease note that jdupes ONLY works on 100% exact matches. It does not have any sort of \"similarity\" matching, nor does it know anything about any specific file formats such as images or sounds. Something as simple as a change in embedded metadata such as the ID3 tags in an MP3 file or the EXIF information in a JPEG image will not change the sound or image presented to the user when opened, but technically it makes the file no longer identical to the original.\n\nPlenty of excellent tools already exist to \"fuzzy match\" specific file types using knowledge of their file formats to help. There are no plans to add this type of matching to jdupes.\n\nThere are some match options available in jdupes that enable dangerous file\nmatching based on partial or likely but not 100% certain matching. These are\nconsidered expert options for special situations and are clearly and loudly\ndocumented as being dangerous. The `-Q` and `-T` options are notable examples,\nand the extreme danger of the `-T` option is safeguarded by a requirement to\nspecify it twice so it can't be used accidentally.\n\nThe standard output format of jdupes is extremely simple. Match sets are presented with one file path per line, and match sets are separated by a blank line. This is easy to process with fairly simple shell scripts. You can find example shell scripts in the \"example scripts\" directory in the jdupes source code. The main example script, \"example.sh\", is easy to modify to take basic actions on each file in a match set. These scripts are used by piping the standard jdupes output to them:\n\njdupes dir1 dir2 dir3 | example.sh scriptparameters\n\n```\nUsage: jdupes [options] DIRECTORY...\n```\n\nDuplicate file sets will be printed by default unless a different action option is specified (delete, summarize, link, dedupe, etc.)\n\n```\n -@ --loud              output annoying low-level debug info while running\n -0 --print-null        output nulls instead of CR/LF (like 'find -print0')\n -1 --one-file-system   do not match files on different filesystems/devices\n -A --no-hidden         exclude hidden files from consideration\n -B --dedupe            do a copy-on-write (reflink/clone) deduplication\n -C --chunk-size=#      override I/O chunk size in KiB (min 4, max 262144)\n -d --delete            prompt user for files to preserve and delete all\n                        others; important: under particular circumstances,\n                        data may be lost when using this option together\n                        with -s or --symlinks, or when specifying a\n                        particular directory more than once; refer to the\n                        documentation for additional information\n -D --debug             output debug statistics after completion\n -e --error-on-dupe     exit on any duplicate found with status code 255\n -f --omit-first        omit the first file in each set of matches\n -h --help              display this help message\n -H --hard-links        treat any linked files as duplicate files. Normally\n                        linked files are treated as non-duplicates for safety\n -i --reverse           reverse (invert) the match sort order\n -I --isolate           files in the same specified directory won't match\n -j --json              produce JSON (machine-readable) output\n -l --link-soft         make relative symlinks for duplicates w/o prompting\n -L --link-hard         hard link all duplicate files without prompting\n                        Windows allows a maximum of 1023 hard links per file\n -m --summarize         summarize dupe information\n -M --print-summarize   will print matches and --summarize at the end\n -N --no-prompt         together with --delete, preserve the first file in\n                        each set of duplicates and delete the rest without\n                        prompting the user\n -o --order=BY          select sort order for output, linking and deleting:\n                        by mtime (BY=time) or filename (BY=name, the default)\n -O --param-order       sort output files in order of command line parameter\nsequence\n                        Parameter order is more important than selected -o sort\n                        which applies should several files share the same\nparameter order\n -p --permissions       don't consider files with different owner/group or\n                        permission bits as duplicates\n -P --print=type        print extra info (partial, early, fullhash)\n -q --quiet             hide progress indicator\n -Q --quick             skip byte-by-byte duplicate verification. WARNING:\n                        this may delete non-duplicates! Read the manual first!\n -r --recurse           for every directory, process its subdirectories too\n -R --recurse:          for each directory given after this option follow\n                        subdirectories encountered within (note the ':' at\n                        the end of the option, manpage for more details)\n -s --symlinks          follow symlinks\n -S --size              show size of duplicate files\n -t --no-change-check   disable security check for file changes (aka TOCTTOU)\n -T --partial-only      match based on partial hashes only. WARNING:\n                        EXTREMELY DANGEROUS paired with destructive actions!\n                        -T must be specified twice to work. Read the manual!\n -u --print-unique      print only a list of unique (non-matched) files\n -U --no-trav-check     disable double-traversal safety check (BE VERY CAREFUL)\n                        This fixes a Google Drive File Stream recursion issue\n -v --version           display jdupes version and license information\n -X --ext-filter=x:y    filter files based on specified criteria\n                        Use '-X help' for detailed extfilter help\n -y --hash-db=file      use a hash database text file to speed up repeat runs\n                        Passing '-y .' will expand to  '-y jdupes_hashdb.txt'\n -z --zero-match        consider zero-length files to be duplicates\n -Z --soft-abort        If the user aborts (i.e. CTRL-C) act on matches so far\n                        You can send SIGUSR1 to the program to toggle this\n\nDetailed help for jdupes -X/--extfilter options\nGeneral format: jdupes -X filter[:value][size_suffix]\n\nnoext:ext1[,ext2,...]           Exclude files with certain extension(s)\n\nonlyext:ext1[,ext2,...]         Only include files with certain extension(s)\n\nsize[+-=]:size[suffix]          Only Include files matching size criteria\n                                Size specs: + larger, - smaller, = equal to\n                                Specs can be mixed, i.e. size+=:100k will\n                                only include files 100KiB or more in size.\n\nnostr:text_string               Exclude all paths containing the string\nonlystr:text_string             Only allow paths containing the string\n                                HINT: you can use these for directories:\n                                -X nostr:/dir_x/  or  -X onlystr:/dir_x/\nnewer:datetime                  Only include files newer than specified date\nolder:datetime                  Only include files older than specified date\n                                Date/time format: \"YYYY-MM-DD HH:MM:SS\"\n                                Time is optional (remember to escape spaces!)\n\nSome filters take no value or multiple values. Filters that can take\na numeric option generally support the size multipliers K/M/G/T/P/E\nwith or without an added iB or B. Multipliers are binary-style unless\nthe -B suffix is used, which will use decimal multipliers. For example,\n16k or 16kib = 16384; 16kb = 16000. Multipliers are case-insensitive.\n\nFilters have cumulative effects: jdupes -X size+:99 -X size-:101 will\ncause only files of exactly 100 bytes in size to be included.\n\nExtension matching is case-insensitive.\nPath substring matching is case-sensitive.\n```\n\nThe `-U`/`--no-trav-check` option disables the double-traversal protection.\nIn the VAST MAJORITY of circumstances, this SHOULD NOT BE DONE, as it protects\nagainst several dangerous user errors, including specifying the same files or\ndirectories twice causing them to match themselves and potentially be lost or\nirreversibly damaged, or a symbolic link to a directory making an endless loop\nof recursion that will cause the program to hang indefinitely. This option was\nadded because Google Drive File Stream presents directories in the virtual hard\ndrive used by GDFS with identical device:inode pairs despite the directories\nactually being different. This triggers double-traversal prevention against\nevery directory, effectively blocking all recursion. Disabling this check will\nreduce safety, but will allow duplicate scanning inside Google Drive File\nStream drives. This also results in a very minor speed boost during recursion,\nbut the boost is unlikely to be noticeable.\n\nThe `-t`/`--no-change-check` option disables file change checks during/after\nscanning. This opens a security vulnerability that is called a TOCTTOU (time of\ncheck to time of use) vulnerability. The program normally runs checks\nimmediately before scanning or taking action upon a file to see if the file has\nchanged in some way since it was last checked. With this option enabled, the\nprogram will not run any of these checks, making the algorithm slightly faster,\nbut also increasing the risk that the program scans a file, the file is changed\nafter the scan, and the program still acts like the file was in its previous\nstate. This is particularly dangerous when considering actions such as linking\nand deleting. In the most extreme case, a file could be deleted during scanning\nbut match other files prior to that deletion; if the file is the first in the\nlist of duplicates and auto-delete is used, all of the remaining matched files\nwill be deleted as well. This option was added due to user reports of some\nfilesystems (particularly network filesystems) changing the reported file\ninformation inappropriately, rendering the entire program unusable on such\nfilesystems.\n\nThe `-n`/`--no-empty` option was removed for safety. Matching zero-length files\nas duplicates now requires explicit use of the `-z`/`--zero-match` option\ninstead.\n\nDuplicate files are listed together in groups with each file displayed on a separate line. The groups are then separated from each other by blank lines.\n\nThe `-s`/`--symlinks` option will treat symlinked files as regular files, but\ndirect symlinks will be treated as if they are hard linked files and the\n-H/--hard-links option will apply to them in the same manner.\n\nWhen using `-d` or `--delete`, care should be taken to insure against\naccidental data loss. While no information will be immediately lost, using this\noption together with `-s` or `--symlink` can lead to confusing information\nbeing presented to the user when prompted for files to preserve. Specifically,\na user could accidentally preserve a symlink while deleting the file it points\nto. A similar problem arises when specifying a particular directory more than\nonce. All files within that directory will be listed as their own duplicates,\nleading to data loss should a user preserve a file without its \"duplicate\" (the\nfile itself!)\n\nUsing `-1` or `--one-file-system` prevents matches that cross filesystems, but\na more relaxed form of this option may be added that allows cross-matching for\nall filesystems that each parameter is present on.\n\n`-Z` or `--soft-abort` used to be `--hard-abort` in jdupes prior to v1.5 and had\nthe opposite behavior. Defaulting to taking action on abort is probably not\nwhat most users would expect. The decision to invert rather than reassign to a\ndifferent option was made because this feature was still fairly new at the time\nof the change.\n\nOn non-Windows platforms that support SIGUSR1, you can toggle the state of the\n`-Z` option by sending a SIGUSR1 to the program. This is handy if you want to\nabort jdupes, didn't specify `-Z`, and changed your mind and don't want to lose\nall the work that was done so far. Just do '`killall -USR1 jdupes`' and you will\nbe able to abort with `-Z`. This works in reverse: if you want to prevent a\n`-Z` from happening, a SIGUSR1 will toggle it back off. That's a lot less\nuseful because you can just stop and kill the program to get the same effect,\nbut it's there if you want it for some reason. Sending the signal twice while\nthe program is stopped will behave as if it was only sent once, as per normal\nPOSIX signal behavior.\n\nThe `-O` or `--param-order` option allows the user greater control over what\nappears in the first position of a match set, specifically for keeping the `-N`\noption from deleting all but one file in a set in a seemingly random way. All\ndirectories specified on the command line will be used as the sorting order of\nresult sets first, followed by the sorting algorithm set by the `-o` or\n`--order` option. This means that the order of all match pairs for a single\ndirectory specification will retain the old sorting behavior even if this\noption is specified.\n\nWhen used together with options `-s` or `--symlink`, a user could accidentally\npreserve a symlink while deleting the file it points to.\n\nThe `-Q` or `--quick` option only reads each file once, hashes it, and performs\ncomparisons based solely on the hashes. There is a small but significant risk\nof a hash collision which is the purpose of the failsafe byte-for-byte\ncomparison that this option explicitly bypasses. Do not use it on ANY data set\nfor which any amount of data loss is unacceptable. You have been warned!\n\nThe `-T` or `--partial-only` option produces results based on a hash of the\nfirst block of file data in each file, ignoring everything else in the file.\nPartial hash checks have always been an important exclusion step in the jdupes\nalgorithm, usually hashing the first 4096 bytes of data and allowing files that\nare different at the start to be rejected early. In certain scenarios it may be\na useful heuristic for a user to see that a set of files has the same size and\nthe same starting data, even if the remaining data does not match; one example\nof this would be comparing files with data blocks that are damaged or missing\nsuch as an incomplete file transfer or checking a data recovery against\nknown-good copies to see what damaged data can be deleted in favor of restoring\nthe known-good copy. This option is meant to be used with informational actions\nand can result in EXTREME DATA LOSS if used with options that delete files,\ncreate hard links, or perform other destructive actions on data based on the\nmatching output. Because of the potential for massive data destruction, this\noption MUST BE SPECIFIED TWICE to take effect and will error out if it is only\nspecified once.\n\nThe `-I`/`--isolate` option attempts to block matches that are contained in the\nsame specified directory parameter on the command line. Due to the underlying\nnature of the jdupes algorithm, a lot of matches will be blocked by this option\nthat probably should not be. This code could use improvement.\n\nThe `-C`/`--chunk-size` option overrides the size of the I/O \"chunk\" used for\nall file operations. Larger numbers will increase the amount of data read at\nonce from each file and may improve performance when scanning lots of files\nthat are larger than the default chunk size by reducing \"thrashing\" of the hard\ndisk heads. Smaller numbers may increase algorithm speed depending on the\ncharacteristics of your CPU but will usually increase I/O and system call\noverhead as well. The number also directly affects memory usage: I/O chunk size\nis used for at least three allocations in the program, so using a chunk size of\n16777216 (16 MiB) will require 48 MiB of RAM. The default is usually between\n32768 and 65536 which results in the fastest raw speed of the algorithm and\ngenerally good all-around performance. Feel free to experiment with the number\non your data set and report your experiences (preferably with benchmarks and\ninfo on your data set.)\n\nUsing `-P`/`--print` will cause the program to print extra information that may\nbe useful but will pollute the output in a way that makes scripted handling\ndifficult. Its current purpose is to reveal more information about the file\nmatching process by printing match pairs that pass certain steps of the process\nprior to full file comparison. This can be useful if you have two files that\nare passing early checks but failing after full checks.\n\nThe `-y`/`--hash-db` feature creates and maintains a text file with a list of\nfile paths, hashes, and other metadata that enables jdupes to \"remember\" file\ndata across runs. Specifying a period '.' as the database file name will use a\nname of \"jdupes_hashdb.txt\" instead; this alias makes it easy to use the hash\ndatabase feature without typing a descriptive name each time. THIS FEATURE IS\nCURRENTLY UNDER DEVELOPMENT AND HAS MANY QUIRKS. USE IT AT YOUR OWN RISK. In\nparticular, one of the biggest problems with this feature is that it stores\nevery path exactly as specified on the command line; if any paths are passed\ninto jdupes on a subsequent run with a different prefix then they will not be\nrecognized and they will be treated as totally different files. For example,\nrunning `jdupes -y . foo/` is not the same as `jdupes -y . ./foo` nor the same\nas (from a sibling directory) `jdupes -y ../foo`. You must run jdupes from the\nsame working directory and with the same path specifications to take advantage\nof the hash database feature. When used correctly, a fully populated hash\ndatabase can reduce subsequent runs with hundreds of thousands of files that\nnormally take a very long time to run down to the directory scanning time plus\na couple of seconds. If the directory data is already in the OS disk cache,\nthis can make subsequent runs with over 100K files finish in under one second.\n\nA set of arrows are used in file linking to show what action was taken on each link candidate. These arrows are as follows:\n\n`---->` File was hard linked to the first file in the duplicate chain\n\n`-@@->` File was symlinked to the first file in the chain\n\n`-##->` File was cloned from the first file in the chain\n\n`-==->` Already a hard link to the first file in the chain\n\n`-//->` File linking failed due to an error during the linking process\n\nIf your data set has linked files and you do not use `-H` to always consider\nthem as duplicates, you may still see linked files appear together in match\nsets. This is caused by a separate file that matches with linked files\nindependently and is the correct behavior. See notes below on the \"triangle\nproblem\" in jdupes for technical details.\n\nWindows has a hard limit of 1024 hard links per file. There is no way to change this. The documentation for CreateHardLink() states: \"The maximum number of hard links that can be created with this function is 1023 per file. If more than 1023 links are created for a file, an error results.\" (The number is actually 1024, but they're ignoring the first file.)\n\nPairs of files are excluded individually based on how the two files compare.\nFor example, if `--hard-links` is not specified then two files which are hard\nlinked will not match one another for duplicate scanning purposes. The problem\nwith only examining files in pairs is that certain circumstances will lead to\nthe exclusion being overridden.\n\nLet's say we have three files with identical contents:\n\n```\na/file1\na/file2\na/file3\n```\n\nand `a/file1` is linked to `a/file3`. Here's how `jdupes a/` sees them:\n\n``` php\n    Are 'a/file1' and 'a/file2' matches? Yes\n    [point a/file1->duplicates to a/file2]\n\n    Are 'a/file1' and 'a/file3' matches? No (hard linked already, `-H` off)\n\n    Are 'a/file2' and 'a/file3' matches? Yes\n    [point a/file2->duplicates to a/file3]\n```\n\nNow you have the following duplicate list:\n\n``` php\na/file1->duplicates ==> a/file2->duplicates ==> a/file3\n```\n\nThe solution is to split match sets into multiple sets, but doing this will also remove the guarantee that files will only ever appear in one match set and could result in data loss if handled improperly. In the future, options for \"greedy\" and \"sparse\" may be introduced to switch between allowing triangle matches to be in the same set vs. splitting sets after matching finishes without the \"only ever appears once\" guarantee.\n\nYes. If you've not read this list of cautions, it is available at\n[http://rmlint.readthedocs.io/en/latest/cautions.html](http://rmlint.readthedocs.io/en/latest/cautions.html)\n\nHere's a breakdown of how jdupes addresses each of the items listed.\n\nThese guidelines are for the user of duplicate scanning software, not the software itself. Back up your files regularly. Use jdupes to print a list of what is found as duplicated and check that list very carefully before automatically deleting the files.\n\nThe only character that poses a concern in jdupes is a newline `\\n` and that is\nonly a problem because the duplicate set printer uses them to separate file\nnames. Actions taken by jdupes are not parsed like a command line, so spaces\nand other weird characters in names aren't a problem. Escaping the names\nproperly if acting on the printed output is a problem for the user's shell\nscript or other external program.\n\nThis is also an exercise for the user.\n\njdupes tracks each directory traversed by dev:inode pair to avoid adding the contents of the same directory twice. This prevents the user from being able to register all of their files twice by duplicating an entry on the command line. Symlinked directories are only followed if they weren't already followed earlier. Files are renamed to a temporary name before any linking is done and if the link operation fails they are renamed back to the original name.\n\njdupes uses xxHash for file data hashing. This hash is extremely fast with a low collision rate, but it still encounters collisions as any hash function will (\"secure\" or otherwise) due to the pigeonhole principle. This is why jdupes performs a full-file verification before declaring a match. It's slower than matching by hash only, but the pigeonhole principle puts all data sets larger than the hash at risk of collision, meaning a false duplicate detection and data loss. The slower completion time is not as important as data integrity. Checking for a match based on hashes alone is irresponsible, and using secure hashes like MD5 or the SHA families is orders of magnitude slower than xxHash while still suffering from the risk brought about by the pigeonholing. An example of this problem is as follows: if you have 365 days in a year and 366 people, the chance of having at least two birthdays on the same day is guaranteed; likewise, even though SHA512 is a 512-bit (64-byte) wide hash, there are guaranteed to be at least 256 pairs of data streams that causes a collision once any of the data streams being hashed for comparison is 65 bytes (520 bits) or larger.\n\njdupes does not protect the user from putting ASCII control characters in their file names; they will mangle the output if printed, but they can still be operated upon by the actions (delete, link, etc.) in jdupes.\n\njdupes uses an I/O chunk size that is optimized for reading as much as possible from disk at once to take advantage of high sequential read speeds in traditional rotating media drives while balancing against the significantly higher rate of CPU cache misses triggered by an excessively large I/O buffer size. Enlarging the I/O buffer further may allow for lots of large files to be read with less head seeking, but the CPU cache misses slow the algorithm down and memory usage increases to hold these large buffers. jdupes is benchmarked periodically to make sure that the chosen I/O chunk size is the best compromise for a wide variety of data sets.\n\nThis is a very subjective concern considering that even a cell phone in someone's pocket has at least 1GB of RAM, however it still applies in the embedded device world where 32MB of RAM might be all that you can have. Even when processing a data set with over a million files, jdupes memory usage (tested on Linux x86-64 with -O3 optimization) doesn't exceed 2GB. A low memory mode can be chosen at compile time to reduce overall memory usage with a small performance penalty.\n\nThe most naive way to look for files that are the same is to compare all files\nto all other files using a tool like `cmp` command on Linux/macOS/BSD or the\n`fc` command on Windows/DOS. This works but is extremely slow and wastes a lot\nof time. For every new file to compare, the number of comparisons increases\nexponentially (the formula is n(n-1)/2 for the discrete math nerds):\n\n| Files | Compares | \n|---|---|\n| 2 | 1 | \n| 3 | 3 | \n| 4 | 6 | \n| 5 | 10 | \n| 10 | 45 | \n| 100 | 4950 | \n| 1000 | 499500 | \n| 5000 | 12497500 | \n| 10000 | 49995000 | \n| 14142 | 99991011 | \n\nLet's say that every file is 1,000 bytes in size and you have 10,000 files for a total size of 10,000,000 bytes (about 9.53 MiB). Using this naive comparison approach means the actual amount of data to compare is around 47,679 MiB. You should be able to see how extreme this can get--especially with larger files.\n\nA slightly smarter approach is to use *file hashes* as a substitute for the\nfull file contents. A *hash* is a number based on the data fed into a *hash\nfunction* and the number is always the same when the same data is fed in.\nIf the hash for two files is different then the contents of those files are\nguaranteed to be different; if the hash is the same then the data **might** be\nthe same, though this is not guaranteed due to the *birthday problem:* the\nsize of the number is much smaller than the size of the data it represents, so\n*there will always be many different inputs that produce the same hash value.*\nFiles with matching hash values must still be compared just to be sure that\nthey are 100% identical.\n\n49,995,000 comparisons can be done much quicker when you're only comparing a single big number every time instead of thousands or millions of bytes. This makes a big difference in performance since the only files being compared are files that look likely to be identical.\n\n**Fast exclusion of non-duplicates is the main purpose of duplicate scanners.**\n\njdupes uses a lot of fast exclusion techniques beyond this. A partial list of these in the order they're performed is as follows:\n\n1. Files that the user asks the program to exclude are skipped entirely\n2. Files with different sizes can't be identical, so they're not compared\n3. The first 4 KiB is hashed and compared which avoids reading full files\n4. Entire files are hashed and compared which avoids comparing data directly\n5. Finally, actual file data is compared to verify that they are duplicates\n\nThe vast majority of non-duplicate file pairs never make it past the partial (4 KiB) hashing step. This reduces the amount of data read from disk and time spent comparing things to the smallest amount possible.\n\nLong options now have consistent hyphenation to separate the words used in the\noption names. Run `jdupes -h` to see the correct usage. Legacy options will\nremain in place until the next major or minor release (v2.0 or v1.21.0) for\ncompatibility purposes. Users should change any scripts using the old options\nto use the new ones...or better yet, stop using long options in your scripts\nin the first place, because it's unnecessarily verbose and wasteful to do so.\n\nIf a file was added through recursion and also added explicitly, that file would end up matching itself. This issue can be seen in v1.14.1 or older versions that support single file addition using a command like this in the jdupes source code directory:\n\n/usr/src/jdupes$ jdupes -rH testdir/isolate/1/ testdir/isolate/1/1.txt testdir/isolate/1/1.txt testdir/isolate/1/1.txt testdir/isolate/1/2.txt\n\nEven worse, using the special dot directory will make it happen without the -H option, which is how I discovered this bug:\n\n/usr/src/jdupes/testdir/isolate/1$ jdupes . 1.txt ./1.txt ./2.txt 1.txt\n\nThis works for any path with a single dot directory anywhere in the path, so it has a good deal of potential for data loss in some use cases. As such, the best option was to shove out a new minor release with this feature turned off until some additional checking can be done, e.g. by making sure the canonical paths aren't identical between any two files.\n\nA future release will fix this safely.\n\nGeneral program information, help, and tech info: [https://www.jdupes.com/](https://www.jdupes.com/)\n\nDevelopment, source code, and releases: [https://codeberg.org/jbruchon/jdupes](https://codeberg.org/jbruchon/jdupes)\n\nHave a bug report or questions? contact Jody Bruchon [jody@jodybruchon.com](mailto:jody@jodybruchon.com)\n\njdupes is Copyright (C) 2015-2023 by Jody Bruchon [jody@jodybruchon.com](mailto:jody@jodybruchon.com)\n\nDerived from the original 'fdupes' 1.51 (C) 1999-2014 by Adrian Lopez\n\nThe MIT License\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.", "url": "https://wpnews.pro/news/h2oai-jdupes-h2o-ai-fork-of-jdupes", "canonical_source": "https://github.com/h2oai/jdupes", "published_at": "2026-10-23 16:35:45+00:00", "updated_at": "2026-10-23 16:43:11.732281+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["H2O.ai", "jdupes", "fdupes"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/h2oai-jdupes-h2o-ai-fork-of-jdupes", "markdown": "https://wpnews.pro/news/h2oai-jdupes-h2o-ai-fork-of-jdupes.md", "text": "https://wpnews.pro/news/h2oai-jdupes-h2o-ai-fork-of-jdupes.txt", "jsonld": "https://wpnews.pro/news/h2oai-jdupes-h2o-ai-fork-of-jdupes.jsonld"}}