Free duplicate file removal on a Mac: verify before deleting

Looking for a free way to delete duplicate files usually means the disk is close to full and the budget for solving it is zero. Free scanners are easy to find. The part that deserves attention is not the price. It is the moment a list of a few thousand file paths appears and something has to decide which half of that list disappears.

That decision is not reversible in the way most people assume. This walks through what a scanner is actually claiming when it calls two files duplicates, why the space estimate on screen can be wrong before anything is deleted, and how to turn a scanner result into a deletion you could explain afterwards.

Four different tests get called the same word

Scanners rarely say which test they ran. The four in common use disagree with each other often enough that the difference matters.

Name match finds report.pdf and report copy.pdf. It is fast and it is wrong constantly. Two files can carry the same name and hold different content, which happens every time a project template gets filled in twice.

Size match compares byte counts. It produces false positives in bulk. Photos from the same camera at the same settings land on identical byte counts regularly, and so do exported PDFs from the same template.

Hash match reads every byte and produces a fixed length digest. Two files with the same SHA-256 digest hold the same content. This is the test worth trusting, and it is also the slow one, because the whole file has to be read from disk.

Byte-for-byte comparison reads both files and stops at the first difference. It proves the same thing a hash does, without producing a value that can be stored and reused later.

Fast scanners usually run name and size first, then hash only the candidates that survive. That is a reasonable design. The problem is when the result screen presents name matches and hash matches in one undifferentiated list, and the checkbox next to each row looks the same.

A scanner that cannot tell you which test produced a given row is asking for trust it has not earned. Before accepting any list, find the setting that controls the comparison depth and read what it says.

The space on the screen may not be on the disk

APFS, the file system on every current Mac, supports cloning. A copy made inside the same volume can share its data blocks with the original until one of them is modified. Both files show a full size in the Finder and in du, but the disk holds one copy of the data.

The gap is easy to measure. Creating a 200 MB file, then cloning it with cp -c, drops free space on the volume by roughly 1 MB, not 200 MB. Both files report 200 MB in every size column on the system.

dd if=/dev/urandom of=a.bin bs=1m count=200
df -k /            # note the free space
cp -c a.bin b.bin  # clone, sharing blocks
df -k /            # free space barely moved

Hard links behave in a related way. Several directory entries point at one set of data, and the space is released only when the last entry goes. A scanner that adds up the reported size of every file it flagged will overstate the recovery in both cases.

This does not make the duplicates harmless. Two entries for the same content still create the problem of not knowing which one is current. It does mean the number printed on the scanner's summary screen is a claim about file sizes, not a claim about free space. Check the actual free space before and after any large deletion rather than trusting the estimate.

Build the list with commands whose output can be read

A scanner is a black box that produces a list. The same list can be produced with commands where every step is visible, which matters when the next step deletes things.

find ~/Documents -type f -size +1M -exec shasum -a 256 {} + \
  | sort | awk '{ if ($1 == prev) print; prev = $1 }'

This reads every file over 1 MB under ~/Documents, prints a digest and a path, sorts by digest, and prints the rows whose digest matches the previous row. What comes out is a list of second and later copies, grouped by content.

For a single pair, cmp -s fileA fileB returns success only when the two are identical byte for byte. It says nothing and exits quietly when they match, which makes it easy to use inside a loop.

Command	What it proves	Cost
ls -l	Two files report the same byte count	Instant, weak evidence
shasum -a 256	Content is identical	Reads the whole file
cmp -s	Content is identical, for one pair	Reads both files, stops at first difference
du -h	Blocks the file occupies, clones counted at full size	Instant, misleading on APFS
df -k	Free space on the volume, before and after	Instant, the number that actually matters

The point of doing this by hand once is not to replace a scanner permanently. It is to have a reference list to compare a scanner against on a folder small enough to check. If the two lists disagree, the scanner's default settings are not what was assumed.

Where free stops

Free duplicate finders fall into a few shapes, and the shape determines where the limit sits.

Some are free to scan and charge to act. The scan completes, the list appears, and the button that removes anything opens a purchase screen. Nothing is wrong with that model, but it means the free part is only the part that was never the risk.

Some are free with a cap, usually on the number of files removed per run or the total size. The cap is not always visible before the scan finishes.

Some are genuinely free and open source, in which case the limit is usually the interface. Fewer guard rails, no preview, and a delete action that goes straight past the Trash.

Some are free because the scan is the product and the removal is done by a companion app with a subscription.

Read the removal step's terms before running the scan, not after. A list of four thousand paths creates pressure to finish, and that is the worst moment to be reading a pricing page. The Pricing page of any file tool is a faster read than its feature list, because it states plainly which actions are gated.

Files the scan will never reach

A duplicate list is only as complete as the set of files the scanner was allowed to read, and on a current Mac that set is smaller than most people expect.

Files stored in iCloud Drive may not be on the disk at all. When Optimize Mac Storage is on, the local entry can be a placeholder while the content lives on the server. A scanner reading that entry sees a stub, not the bytes, so it either skips the file or reports a size that has nothing to do with the content. Two files that are genuinely identical will not be flagged if one of them is not downloaded. The reverse also happens: forcing a comparison can trigger a download of every file it touches, which is a slow and expensive way to discover a duplicate.

Permissions are the second boundary. macOS gates several locations behind Full Disk Access, including parts of the user Library, the Photos library, and Mail data. A scanner launched without that permission simply returns nothing from those paths, usually without saying so. The absence looks like a clean result. Granting the permission is a real decision, since it hands a third party tool read access to everything on the machine, and it is worth making deliberately rather than by clicking through a prompt mid scan.

Unmounted volumes are the third. External disks, network shares, and disk images that are not attached at scan time contribute nothing to the list, which matters because archive copies usually live exactly there. A duplicate hunt run only against the internal disk will happily flag the working copy of a file whose only other copy is on a drive sitting in a drawer.

None of this is a reason to avoid scanning. It is a reason to treat the resulting list as a partial view. Note which volumes were mounted and whether iCloud Drive was fully downloaded, and record that alongside the list, because a decision made from an incomplete list needs the incompleteness written down next to it.

What the Trash actually guarantees

Deleting through the Trash is recoverable until the Trash is emptied. After that it is not.

Items that were automatically removed from the Trash are permanently deleted and are no longer available. Source: support.apple.com

Two settings change how long that window lasts. The Finder has an option to remove items from the Trash after 30 days, which turns a deliberate action into a scheduled one. And items moved to the Trash from iCloud Drive are removed after 30 days regardless of that setting.

Tools differ here more than anywhere else. Some move flagged files to the Trash, which leaves a real chance to change course. Some move them to a quarantine folder the tool controls, which is recoverable but only through that tool. Some call the delete system call directly, which leaves nothing.

Find out which of the three a tool does before running it, not after. The answer is usually in a preference rather than the marketing copy.

Make the deletion auditable

The useful discipline is to separate finding from deleting, with a file in between.

Write the list of paths to be removed into a plain text file. Read it. Sort it by directory and look at the directories rather than the files, because a wrong directory in the list is far more common than a wrong single file, and far easier to spot. Then run the deletion from that file rather than from a checkbox grid.

sort dupes.txt | awk -F/ '{ $NF=""; print }' | uniq -c | sort -rn | head -20

That prints how many flagged files came from each directory. A directory contributing hundreds of rows is either the answer or the mistake, and either way it deserves a look before anything is removed.

Three directories deserve suspicion whenever they appear in a duplicate list. Application bundles contain identical resource files by design, and removing one breaks the app. Version control directories contain content addressed object files where identical content is expected. Package manager directories hold many copies of the same library on purpose, and the tooling above them assumes those copies exist.

The wider point is that all of this involves moving between a list of paths, a folder view, and a terminal, and the switching is where the time goes. A file manager that keeps the folder view and the shell in one window removes the step where a path gets copied from one place to another and pasted slightly wrong. The comparison page covers how different tools handle that split.

What to change first

Run the hash command above on one folder, compare its output against whatever scanner is being considered, and see whether the two agree. That single check tells you more about the tool than any feature list.

Then decide the deletion route before the scan, not after: Trash, quarantine, or immediate. If a tool will not say which one it uses, that is the answer. Keeping the list, the folder, and the shell in one window makes the check quick enough to actually do, and Atriens is built around that arrangement.