重複ファイルの整理: the setup order that holds up

Most duplicate cleanups do not fail during the scan. They fail about four minutes after it finishes, when a list of several thousand pairs is on screen and there is no rule for deciding which side of each pair survives. The window gets closed. The disk stays full. Three months later the same scan runs again and produces a longer list.

The fix is almost entirely about sequence. Three decisions belong before the scan, not after it, and a deletion is not finished until it has been verified against something other than the tool's own report. What follows is that sequence, with timings measured on a real folder of 31,263 files across 91 GB.

Decision one: which copy survives

A result list is a list of pairs, and every pair needs a winner. Deciding that pair by pair is what kills the session, because the criteria drift somewhere around row 400 and everything decided before that point becomes untrustworthy.

Pick one rule and write it down as a sentence:

• Location wins. The copy inside the designated archive folder survives, regardless of dates.
• Depth wins. The shallower path survives, on the reasoning that deeply nested copies are byproducts of extraction and temporary work.
• Oldest wins. The original survives and everything that appeared later is removed.
• Newest wins. The most recently modified copy survives, for material that is still being edited.

The last two are mutually exclusive and they select different files, because identical content still carries different timestamps. Choosing between them is the single decision that most changes the outcome.

Write the rule with its exception in two lines, something like: the archive folder copy survives; when neither copy is in the archive folder, the shallower path survives. Two lines is enough, and having them written means a row that does not fit the rule can be set aside rather than triggering a redesign of the rule mid list.

Decide in advance what happens to those set-aside rows. Moving them to a holding folder rather than deleting them keeps one ambiguous pair from stopping the entire pass.

Decision two: what is in scope and what is excluded

Pointing a scanner at an entire home folder guarantees that a large fraction of the results are files that must not be touched.

Excluded	Why
The Library folder	Application support data where identical files legitimately coexist
Photo libraries and application bundles	Packages whose internals break when edited from outside
Dependency directories for development projects	Shared components are duplicated across projects by design
Version control internals	Object storage where identical content is meaningful
Undownloaded items in a cloud sync folder	Reading contents triggers the download of the real file

That last row is the one that produces surprises. With storage optimisation enabled, the folder shows filenames whose data lives elsewhere. Content comparison opens files, so a scan can silently begin pulling down tens of gigabytes. Either exclude that folder or materialise it first.

Scope also has an ordering problem of its own. Running across several disks at once surfaces large numbers of laptop-versus-external-drive matches, and most of those are deliberate backups rather than waste. A first pass confined to the working areas of one disk, meaning Downloads, Desktop and Documents, produces a list where nearly every row is a real decision.

Record the folders included and excluded. Without that record, a larger count next quarter cannot be distinguished from a wider scope.

Decision three: how many filter stages

Hashing every file is unnecessary work. Cheaper tests eliminate most candidates before anything is read. Measured across the 31,263 file folder:

Stage	Test	Remaining	Time
1	File sizes match	24,998 files	about 1 second
2	First 64 KB hashes match	23,379 files	3.7 seconds
3	Full content hashes match	5,305 groups	11.3 seconds

Stage one removed 6,252 files while opening nothing, because sizes come from file metadata. Stage three read 17.98 GB, roughly a fifth of the folder. Total elapsed time was about 16 seconds.

Hashing everything in full would mean reading all 91 GB, which lengthens the run by a factor of around five and considerably more than that across a network or an external drive. When a commercial tool advertises speed, staged filtering is usually what is being described.

How much stage two earns depends on the material. Here it removed only 1,619 files, because many documents share their opening bytes: same export format, same template, same header. Collections weighted toward video or disk images benefit far more, since a partial read rejects large files cheaply. Comparing the stage one and stage two counts on a given folder answers this directly, and when the gap is small the stage can simply be dropped.

Stage order is not interchangeable. Size checks touch metadata only. Hash checks read content. Putting the expensive test first removes the point of staging entirely.

One implementation detail causes more failed first attempts than the hashing does: passing filenames between steps. Names containing spaces, parentheses or non-ASCII characters get split when a shell treats whitespace as a separator. Passing paths null-separated removes that whole class of failure on day one.

Write the candidate list to a file before deleting anything

Send the stage three output to a file rather than straight into a deletion. Scrolling a list on screen loses track of position, and there is no record afterwards of what was considered.

Five columns are enough: group number, surviving path, path to remove, file size, bytes recovered for that group. Opened in a spreadsheet, it sorts by size.

Keeping the list as a file has a second benefit that shows up later. It is the only record of what the folder looked like at that moment, so a scan run three months on can be compared against it rather than against a vague impression. Growth between two lists is the number that says whether the cleanup is holding.

Sorting changes what the work looks like. The 5,305 groups from stage three held 9.53 GB of redundant data between them, and that total is heavily concentrated: a handful of installers, disk images and video files at the top of the sorted list account for most of it, while the remaining thousands of rows are small assets from extracted archives. If the goal is free space, the top of the sorted list is the entire job. If the goal is a navigable folder, the long tail is handled at folder level rather than file by file, because thousands of individual decisions is not a task anyone finishes.

Route deletions through the Trash

Move files to the Trash instead of deleting them outright. Something will turn out to have been referenced by a script, a backup exclusion rule or an application's recent items list, and that becomes apparent later rather than immediately.

Empty the Trash after a week of normal work, not the same day. A broken reference usually surfaces the next time the relevant application opens, which may be several days out.

Keep a second copy of the candidate list at this point. The Trash shows filenames with no indication of which pair each one came from, so restoring a file to the right location depends on having the group numbers and both paths written down somewhere.

External drives and network volumes need separate handling. A Trash on an external drive lives on that drive, so space is not returned until it is emptied. Network volumes may bypass the Trash and delete immediately. For both, moving files to a holding folder is safer than deleting.

Verify against free space, not against the report

The final step is refusing to accept the tool's recovered-space figure at face value.

Record free space before and after, and compare. Two things commonly cause a gap. The deleted files may have been APFS clones that never held their own storage, in which case removing them returns almost nothing. Or the freed blocks may still be held by a local snapshot, in which case space returns later on its own. Recording free space on the day and again several days on separates these two.

When the gap is large, stop rather than deleting more. Repeated passes that do not move the free space figure only increase the number of files removed. If a folder turns out to be clone-heavy, the space goal will not be met there and the effort belongs somewhere else.

Stop the folder from refilling

The redundancy measured in that folder did not accumulate over a decade. It came from ordinary weekly activity: files received, archives expanded, project folders copied before a risky change. Running the same cleanup every quarter treats the output and ignores the input.

Three changes reduce the inflow, and none of them require a tool.

Receive in one place. Browser downloads, messaging attachments and transfer clients each have their own default destination, and having three destinations is what produces the same file in three places. Pointing all of them at a single folder means a second copy of the same file lands next to the first, where the filesystem gives it a suffix and the duplication becomes visible immediately rather than months later.

Expand archives somewhere disposable. Extraction is the single largest generator of identical files, because every file inside keeps its original name and the extracted tree is usually a complete second copy of something already stored. Expanding into a scratch folder that gets cleared on a schedule keeps those copies out of the archive entirely.

Reference instead of copying. When the same document is needed in two projects, storing it once and placing an alias in the second location keeps one authoritative copy. This also removes the edit-divergence problem, where two copies of a document both get edited and neither is clearly correct afterwards.

A fourth habit helps with the version-suffix pattern specifically. Files named proposal_v2, proposal_final and proposal_final2 are not duplicates by content, so no scanner will group them, yet they occupy the same conceptual slot and cost the same time to navigate. Deciding that only the current version lives in the working folder, with superseded versions moved to a dated subfolder, keeps that class of clutter out of a problem that content matching cannot solve.

None of this eliminates cleanup. It changes the interval, which is the number that determines whether this is a recurring afternoon or an occasional one.

What to change first

Write the keep rule down before opening any scanner, because that single sentence is what turns a list of thousands into work that can be finished in one sitting. Then confine the first pass to one disk and check the result against free space rather than the summary screen. Keeping the candidate list, the terminal that produced it and the folder itself in one window removes the step where a decision made in a spreadsheet has to be re-found in a file browser, which is the part a setup like Atriens is built around. How different tools match and what they can reach is compared in Compared with other file managers, and scope questions about external drives and sync folders are covered in FAQ.