Spotlight indexing: what the Mac reads while you wait

Spotlight says it is indexing. A search that worked yesterday now returns half of what it should, a progress bar sits at the top of the search window, and there is no estimate of when it will be finished. The obvious question is how long to wait. The more useful question is what the Mac is actually doing, because the answer decides whether waiting is the right move at all.

Indexing is not a scan of filenames. It is a process of opening files, extracting text and attributes from each one, and writing the results into a database that lives on the volume being indexed. Understanding that shape makes the timing predictable and makes the available controls obvious.

What the index is, and what it is not

Apple describes the Spotlight index as a regularly updated search index of the data stored on the device, and states that it is a local, private index that stays on the device and is not shared with Apple or with other devices. There is no account, no upload and no server side component.

Physically, each indexed volume carries its own store in a hidden folder at its root named .Spotlight-V100. The per volume configuration inside it, Store-V1/VolumeConfig.plist, is what records whether indexing is turned on for that disk. This matters more than it sounds: indexing is a property of a volume, not of the Mac. An external drive can be indexed while another is not, and a drive that is indexed on one Mac arrives unindexed on another.

What the store holds is not a copy of the files. It holds extracted text and a set of named attributes per file. Those attributes are visible for any single file:

mdls ~/Documents/report.pdf

The output is a list of keys beginning with kMDItem, covering names, sizes, creation and modification dates, and, depending on the file type, page counts, authors, camera settings, audio bit rates and more. The full vocabulary is larger than most people expect. Asking the importer system to print every attribute it knows about returns several hundred lines:

mdimport -A

The practical consequence is that the index is a queryable database of file properties, not a list of names. A search for every PDF over ten megabytes modified this month is a database query, and it returns immediately because the answer was computed during indexing rather than at the moment of the search.

What gets read out of each file

Extraction is handled by importer plug-ins, one per family of file types. They are ordinary bundles and the built in set can be listed:

ls /System/Library/Spotlight/

That returns importers for applications, archives, audio, fonts, images, MIDI, mail, Office documents, PDF, PostScript, rich text, calendar files, iWork documents and vCards, among others. Applications install their own into /Library/Spotlight, and the mdimport manual page documents -L to list every installed importer, including the third party ones.

This is the single best predictor of how long an indexing run will take. Cost follows extractable text, not file count.

Content	Cost to index	Why
Ten thousand small photographs	Low	Metadata only, no text body to read
Two hundred long PDF files	High	Every page is parsed for text
A large mail archive	High	Each message is a separate item
A folder of video files	Low	Attributes only, no transcript
A source code repository	Moderate	Plain text, but many small files
A virtual machine image	Low but slow	One enormous file, read in full

It also explains a surprise that catches people out. Installing an application that ships a new importer can set off a fresh burst of indexing for a file type that was previously ignored, because every existing file of that type now has something worth extracting. Nothing on the disk changed. The rules for reading it did.

Why the first run is long and the rest are not

Apple lists the triggers directly: indexing happens when a device is first set up, after a software update or upgrade, and as data changes on the device. The same documentation states that depending on the amount of data, indexing can take hours or even days, and that incremental updates after that are much faster.

The difference between the two is the whole story. A first run reads everything. After that, the system is told about individual changes as they happen, so a saved document costs one file's worth of work. The manual page for mdworker, the process that does the reading, describes it as being used to scan and index files as a volume is mounted or a file changes.

Apple also documents three conditions under which indexing completes more quickly: the device is not in use, it is connected to power, and it is connected to the internet by Wi-Fi or Ethernet, in case email or other data needs to be downloaded in order to be indexed. That last condition is easy to overlook and explains why a mail heavy index can stall on a machine that is awake but offline.

The result is a reasonable plan for a long run. Plug in the power adapter, leave the Mac awake and idle, and check back in a few hours rather than every few minutes. Interrupting the run by putting the machine to sleep or by quitting the indexing processes does not save time. It discards work in progress and lengthens the next attempt.

Watching the progress honestly

Three checks give a truthful picture, and none of them requires third party software.

The progress indicator. Apple documents that while indexing is in progress on a Mac, an indexing progress indicator appears at the top of the Spotlight or Siri window that opens when Command and Space bar are pressed. A bar that moves between checks is a run that is working.

The per volume status. This reports whether indexing is enabled for a volume at all:

mdutil -s /

Substituting the path of an external disk answers the same question for that disk. A volume reporting that indexing is disabled will never finish, because it never started.

A rough count. Because the index is a database, it can be asked how many items it currently matches:

mdfind -count -onlyin ~/Documents "kMDItemContentModificationDate >= \$time.this_month"

Running the same query twice an hour apart shows whether the number is climbing. This is not a percentage, and it should not be read as one, but a figure that moves is more informative than a bar that might be idle.

When the index and the disk disagree

A run that has finished does not guarantee that everything is in the store. Three gaps are common and none of them produces an error message.

A file type with no importer installed contributes its filename and filesystem attributes but no text body. Plain formats are usually covered, proprietary ones from applications that are no longer installed often are not, and a search for a phrase inside such a file will find nothing while a search for its name succeeds.

A file the account cannot read is skipped. Apple notes that a reindex can fail with a permissions error when the user account does not have ownership of items in the folder being indexed, and the same limit applies during ordinary indexing.

A synchronisation placeholder is not a file yet. Content that has been offloaded to a cloud service and left as a stub on disk has no text to extract until it is downloaded, which is one reason a folder that looks full can behave as though it is empty.

Testing a single file against the importers settles which case applies, without writing anything to the store:

mdimport -t -d1 ~/Documents/report.pdf

The manual page describes -t as a test import whose attributes are not stored in the index, with -d controlling how much of the result is printed. A file that produces no text content in that test will never be findable by its contents, no matter how many times the index is rebuilt. That distinction saves a great deal of wasted rebuilding.

Setting the scope before the next run

Three controls decide what the index covers, and they work at different levels.

Categories come first. In System Settings, the Spotlight pane, called Siri and Spotlight in some versions, lists the kinds of results Spotlight will return. Apple's own troubleshooting advice for missing results starts here: check whether the category in question is turned on.

Locations come second. The same pane has a Search Privacy button, labelled Spotlight Privacy in some versions, at the bottom. Anything added to that list is excluded from indexing. This is the right lever for content that is large, machine generated and never searched by hand: build output, dependency caches, virtual machine images, video render scratch folders.

Volumes come third, and are the blunt instrument. The mdutil manual page documents -i on | off for setting the indexing status of a volume, and -E for erasing its local store. Switching a scratch disk out of the index entirely is reasonable. Doing the same to a working disk removes more than the search box, as the next section covers.

There is also a marker the system honours inside a directory, .metadata_never_index, which appears in the metadata server itself. It is a developer level mechanism rather than a supported user setting, and the Search Privacy list is the maintained way to achieve the same result.

What depends on the index besides the search box

The cost of excluding something is easy to underestimate, because the search field in the menu bar is only one of the things reading from the store.

Finder search stops returning content matches for excluded locations. Names may still turn up through a live scan of the folder, but a phrase inside a document will not.

Smart Folders stop filling. A saved search is a stored query against the index, so an empty or excluded region produces an empty folder with no message explaining why.

Mail search degrades, and Apple lists a notification from Mail that indexing is incomplete as one of the visible signs of a run in progress.

Command line selection stops working. mdfind consults the same store, and mdls reads the same attributes. Any script that picks files by metadata rather than by name silently returns nothing, which is harder to notice than an error.

That last consequence lands hardest on people who already work through the terminal, where listing files by state and piping the paths into the next command is routine. Keeping the listing and the shell in one place makes the dependency visible instead of hidden, which is the argument for a file manager with a built in terminal, and the comparison of this category sets out which tools query the Spotlight index and which walk the filesystem directly.

What to change first

If a run is under way and something changed recently, connect the power adapter, leave the Mac idle, and check the progress indicator in a few hours rather than every few minutes. If results have been wrong for longer than that, exclude the one folder that is large, generated and never searched, and leave the rest of the disk alone. The answers to the recurring cases are worth keeping to hand, because the next system update will start a run of its own.