Any time a folder has more than a few thousand items in it, the filesystem is going to be a lot slower when working with that folder. Adding a new file, for example, requires that the filesystem compare the new item name to the name of every other file in the folder to check for conflicts, so trivial tasks like that will take progressively longer as the file count increases. Gathering the enormous file list will also take progressively longer as the list gets larger.
[…]
Last week, one of our users found the task as shown above. Upon closer analysis, we determined that the “media” folder had 181,274 files in it. In other words, more than 10% of the files on the whole startup disk were in that “media” folder. In extreme cases like this, the delay to retrieve a file list can be so long (i.e. longer than 10 minutes) that the task aborts with an error[…]
[…]
For a contrasting example, consider how Mail organizes a potentially astronomic list of files. If you navigate to the hidden Library folder in your home folder, then to Mail > V10 > {any UUID} > {any mailbox} > {another UUID} > Data, you’ll see folders named by number, four layers deep, until you finally get to a Messages folder with actual files in it.
Apple should also employ this technique for Core Data external storage and Spotlight temporary files.
Previously: