A 20-year retrospective on reverse-engineering an Apple bug and a very desperate plea for help.

A little bit of History

I thought it’d be helpful to have a retrospective of a little reverse-engineering task I did about 20 years ago, when Mac OS X 10.4.x was at the dawn of the PowerPC to Intel transition. (Mac OS X 10.5 was the first merged release.) I gathered all this information through reverse engineering, including reading PowerPC disassembly and using a debugger. None of the information was acquired by actually examining source code, as I didn’t have access to said source code.

A brief background on the process/application with the bug

In Mac OS X 10.4, a process known as SystemUIServer was responsible for showing menu extras (but not status items at the time), taking screenshots, providing the Spotlight user interface, and handling some iPod/Digital Hub device events. It also held all the code for some of Apple’s own menu extras—the menu extra plugin bundles for those extras were just stubs to enable the menu.

A screenshot of SystemUIServer from Mac OS X 10.4.3 being decompiled in Hopper with the built-in menu extra code.

Most people’s interactions with SystemUIServer were solely with the true menu extras on the menu bar’s right-hand side.

Today’s SystemUIServer with menu extras, status items, Siri, and control center items

At the time, Apple only allowlisted specific menu extras by class name (checked in -[SUISStartupObject _canLoadClass:]). Any attempt to load a menu extra that advertised a different class name in its Info.plist’s NSPrincipalClass entry would fail. Menu extras were first-class citizens. You could hold the Command (⌘) key down to move or quit them. They loaded automatically and needed no backing application. This sent lots of developers looking for workarounds to Apple’s allowlist, as they wanted these features for their own menus.

Some developers would steal one of the allowlisted class names for their menu extra plugin. This was unwise. The Objective-C runtime only allowed one class instance with a specific name to be loaded at a time. When it found duplicates, the one it chose was an implementation detail that could cause unexpected crashes. Favorite choices of class names to hijack included AppleVerizonExtra or IrDAExtra, i.e., something a user isn’t likely to have enabled. In the rare case someone did enable these, or if more than one developer chose to steal the same class name, all hell could break loose.

This was the impetus for Menu Extra Enabler. It was an old-style InputManager¹ plugin that automatically loaded into SystemUIServer when installed and overrode the -[SUISStartupObject _canLoadClass:] instance method to return YES unregardless of what class name was used for the menu extra’s principal class.

¹InputManagers were a cheap and quick way to get into the address space of arbitrary Cocoa runtime applications at a determinate location.

In Mac OS X 10.3.x, this allowlist limitation bit Apple in the calyx. Apple added a Keychain menu extra in /Applications/Utilities/Keychain Access.app/Contents/Resources/Keychain.menu without updating the allowlist to include the new principal class, so Apple created a hacky Enable.menu to allow it. This Enable.menu enabled the Keychain menu extra, and just for a bonus, also disabled all third-party menu extras. I updated Menu Extra Enabler to prevent Enable.menu from loading—Menu Extra Enabler already allowed Keychain.menu to load, so the hack wasn’t necessary.

The Bug in Mac OS X 10.4

Mac OS X 10.4.x introduced a new problem. If the system delayed the launch of SystemUIServer for long enough, the Bluetooth menu extra would not load even if Bluetooth was enabled. In fact, SystemUIServer seemingly "erased" all menu extras to the right of the Bluetooth menu item! At Unsanity, we received many reports blaming missing Menu Extras on ShapeShifter, with a few people blaming it on Cee Pee You.

Neither caused the problem. In fact, the underlying "problem" could have occurred days before you applied a theme or started Cee Pee You, since it happened only when logging into Mac OS X and at no other time. It also occurred whether or not any third-party menu extras were in use (and even if Menu Extra Enabler wasn't installed). I remember browsing forums and seeing a screenshot of the late Dwyane McDuffie’s menu bar, and he had been bitten by the bug—only the Bluetooth menu extra was visible.

After five straight hours of reverse engineering and debugging, I found the sequence of events that triggers the bug:

1. The user enables Bluetooth and the Bluetooth menu extra.
2. At some later time, the user starts a new login session with a bunch of login items (iChat, Extensis Suitcase, TextEdit, Console, Word, Alfred, whatever)—just enough to delay the normal initialization of SystemUIServer.
3. When the system gets to it, it launches SystemUIServer. It reads its preferences from disk, including the list of menu extras installed last time it ran. It then registers for notifications, calling -[[SUISDocklingServer alloc] initWithController:docklingServer] in -[SUISStartupObject init] so it can respond and install future menu extras when they’re opened. Without this, you’d have to log out and back in to see any new menu extras. So far, so good.
4. Like other Cocoa runtime applications that implement NSApp delegate methods, SystemUIServer receives the -[SUISStartupObject applicationDidFinishLaunching:] call when everything’s ready to go. Loading menu extras here causes some kind of problem, so instead, SystemUIServer takes this opportunity to create a timer to call -[SUISStartupObject _loadMenuBarExtras] after a 20-second delay.
5. The Dockling Server decides it’s now time to load the Bluetooth menu extra, so it issues a notification (see step 3) that eventually calls -[SUISStartupObject addMenuExtra:position:reserved:data:] on the Bluetooth menu extra. The 20-second timer has not elapsed.
6. -[SUISStartupObject addMenuExtra:position:reserved:data:] takes the Bluetooth menu extra and passes it to -[SUISStartupObject createMenuExtra:atPosition:write:data:], which eventually calls -[NSMenuExtra initWithBundle:] and the Dockling server thinks it’s installed the menu extra.²
7. Now that the new menu extra is supposedly loaded, SystemUIServer saves its menuBarPlugins array, writing the preferences to disk.

• The 20-second timer has not elapsed, so the array contains one item: the Bluetooth menu extra. This list of installed menu extras is not reread from disk until SystemUIServer is relaunched, usually at the next login.

8. When the 20 seconds noted in step 4 have elapsed, the timer fires and the system finally calls -[SUISStartupObject _loadMenuBarExtras]. It’s time to load the menu extras! First, the method creates the NSMenuToolbar³ where all menu extras live. After that, it looks at the list of previously-installed menu extras (read from disk in step 3), loads each of them, and makes them into NSMenuToolbarItems added to the main NSMenuToolbar. Everything’s coming up Milhouse!

So what’s the problem? There’s no Bluetooth menu extra! Let’s go back to step 5 and see what really happened.

5. a. Since the NSMenuToolbar that hosts all menu extras doesn’t get created until step 8 (after the 20-second timer elapses), everything triggered by -[SUISStartupObject addMenuExtra:position:reserved:data:] fails.
6. a. SystemUIServer does not check for failure—if it can’t load a menu extra, no big whoop; it’ll just keep iterating through the ones it has.
7. a. But it does matter when it saves (to disk) the preferences containing only the Bluetooth menu extra, which isn’t even loaded. At this point, the list of all menu extras to load next time has been erased. The poor user probably doesn’t notice this—SystemUIServer has already loaded the list for this session (step 3) and doesn’t read back the mangled set of preferences until it launches again, usually at the next login.
8. a. If the list of menu extras from last session contains the Bluetooth menu extra, it still doesn’t load. All subsequent calls to -[SUISStartupObject createMenuExtra:atPosition:write:data:] with the Bluetooth menu extra return NULL (fail). That’s because _alreadyHasExtra: returns YES since it thinks it loaded the Bluetooth menu extra in step 5. So when the bug hits, the Bluetooth menu extra never gets loaded.

² This is foreshadowing!
³ Although NSMenuToolbar is named like an AppKit class, it only exists in SystemUIServer in this story.

I figured out all this after 5 hours of straight debugging/reverse engineering.

The Solution

Unsanity customers thought that Menu Extra Enabler was causing the problem since it optionally restarts SystemUIServer when applying a new theme, thereby reading the mangled preferences from disk, not trying to load anything but Bluetooth. Others blamed Cee Pee You since it restarts SystemUIServer during installation (it has to install Menu Extra Enabler, which is only loaded on process launch).

My first fix was to prevent SystemUIServer from saving menu extras if the new list contained only items to the left of the Bluetooth menu extra and the Bluetooth menu extra itself. This failed if the user manually removed menu extras to the right of the Bluetooth menu extra (by holding the command [⌘] key and dragging them off the menu bar in a “poof”), since the list of loaded menu extras was saved on a timer, and the user could remove multiple menu extras before the timer fired and the list was finally saved to disk.

It was also extremely hackish and required a lot of code (about 50 lines). I was not happy with this at all. Not only did it not fix the problem (just the outcome) since the Bluetooth menu extra still wouldn't load in these cases, but it could also break depending on the user's actions. It annoyed me so much that I woke from sleep after creating this horrible hack. So, when I woke up, I tried again, from scratch.

The new fix with this understanding is only 5 lines of code; it is future-proof and doesn't require the user to do anything.

Apple did fix this bug in Mac OS X 10.5, and Menu Extra Enabler already had code to disable my workaround of not allowing the Bluetooth menu extra even to attempt to load (step 5) if the NSToolbarMenu wasn’t set up yet (step 5a) if the user was running Mac OS X 10.5 or later.

When you do things right, people won't be sure you've done anything at all.

Using Xcode to debug plugins

Menu Extra Enabler was a plugin that had to be in a special location (~/Library/InputManagers/) to function, and loaded in an application to which I had no source code. I had to take advantage of a neat Xcode feature: you can set an arbitrary process/application to be a target’s executable, and it’ll automatically enable breakpoints in your plugin. It’s extremely useful for debugging plugins!

Xcode’s Scheme Editor, Run Section

The necessary Run Script build phase to copy the built plugin to the Proper plugins folder 

This was slightly difficult to do with SystemUIServer, as Mac OS X 10.4 automatically launched it if it wasn’t running. So, there was a lot of killall SystemUIServer in Terminal to keep it dead until Xcode could launch it.

For an application to be debugged in today’s macOS versions, it must have the com.apple.security.get-task-allow entitlement. This is the default for development apps, but those don’t usually run on other people’s computers—they won’t have the development certificate used to sign the app. To prevent developers from accidentally notarizing development versions of their apps that have the com.apple.security.get-task-allow entitlement by default, we at Apple made an explicit decision to allow this entitlement if and only if paired with the disable library validation entitlement (required to load third-party plugins if you notarize your macOS application).

This allows application developers (like Adobe) to ship notarized applications (like Photoshop) that run third-party plugins in-process, while still allowing third-party plugin developers to debug their plugins without having to disable any of the newer macOS security features, like System Integrity Protection or the Signed System Volume.

What’s changed in 20 years.

The Rules

Input Managers are now dead, the allowlist for class names for menu extras is no longer needed due to library validation, and you cannot attach to system processes such as SystemUIServer by default. To prevent (or at least discourage) people running Mac OS X on generic PCs, Apple “encrypted” SystemUIServer and a few other Mac OS X 10.5-and-later executables for use with Don’t Steal Mac OS X.kext.

To add to this, many elements that were previously part of SystemUIServer have been either completely removed (iPod support) or extracted from SystemUIServer into components like Control Center, Notification Center, the Screenshot process, and other plugins, app extensions, and other processes, as if there was a fire sale on the components and everyone wanted a souvenir.

Arbitrary in-process plugins, like menu extras, have always been inherent security and stability risks (and a way to do super neat, unexpected things, per my old job). Apple has bet heavily on out-of-process plugins to solve issues caused by the older ways, like XPC/ExtensionKit for Mail plugins, Xcode plugins, QuickTime plugins (yay!), and Kernel Extensions.

Furthermore, the main reason for using third-party menu extras instead of status items was finally addressed in macOS 10.12, when status items became first-class citizens in the menu bar. They still need a backing app, but they can be moved and removed by holding the command key down and dragon-dropping them around.

The Tools

Long before Gen Alpha was born, I would use otool (the variant known as otx) dumps and feed them into BBEdit to view the absolutely huge files it generated, as BBEdit was the only text editor that could/can handle that much text without falling over backwards onto a pike—as long as word-wrapping was disabled.

The Objective-C runtime requires special processing for connecting methods to call sites in reverse engineering tools. With Objective-C, method calls go to a runtime function (like objc_msgSend()), which then calls the correct method on the proper class after some magic. Due to this indirect calling and highly dynamic dynamism, reverse engineering tools can’t just do the usual “double-click to view function call” on Objective-C code; they need an extra step to connect the methods and calls through Objective-C runtime functions.

On the other hand, Swift adds a lot of boilerplate code to applications, even for basic operations on a class or struct. Great reverse engineering tools with decompilers should recognize the boilerplate code and remove it to show what the developer wrote, not just what the compiler wrote. It’s a minor pet peeve of mine when reverse engineering Swift code.

Hopper

Hopper is a first-class citizen on macOS. 

SystemUIServer in Hopper

For Hopper and Objective-C 4 2.0, Hopper kindly does the extra work of connecting runtime functions to the actual class and method being invoked, if those are available through static analysis. It has an SDK and some plugins. Hopper is a $99/year subscription.

National Security Agency’s Ghidra

When you hear that the US government’s spying agency decided to release a free reverse engineering tool, “NSA’s Ghidra," you may fear exactly what I did, getting shivers down your spine and making you wary: “It’s written in Java, isn’t it?”. Yes, it’s written in Java. It’s not a first-class citizen on macOS. It’s an ugly tool. Bitmap graphics are everywhere. It feels like it escaped from the late 1980s. The installation method is a mess.

SystemUIServer in the NSA’s Ghidra. Semi-supports decompiling 32-bit PowerPC apps!

But it’s completely free and has some really, really good plugins written for it to use with Apple platform (iOS, macOS, et fam) binaries, like LaurieWired’s absolutely fabulous Malimite. It also supports the Objective-C runtime well.

IDA Pro

IDA Pro is a very, very expensive reverse engineering tool. In 2022, private equity firms bought it, removed the perpetual licenses, and raised the prices by several thousand dollars. Prices for a secure version of IDA Pro range from $2,999 to $8,599. It’s not cheap.

SystemUIServer from Mac OS X 10.4 in IDA Pro. Look at the iPod SCSI notification! 

IDA Pro does have a vibrant plugin community. I’m using a very old version and I’m not sure if the built-in Objective-C support has been improved, but I’ve found it lacking.

A desperate plea for help.

One year ago, I was “let go” from Apple as I had used up all my disability leave. It was either “come back, or quit”. So I came back, even though my disability wasn’t treating me any better. I was shortly thereafter let go because my disability was incompatible with productivity, especially under a boss who couldn’t understand non-physical disabilities.

Over the past year, I’ve completely burned through all my savings. This month (May 2025) is the last month I’m able to pay rent. I’m completely broke, even in debt after paying May’s rent. I can’t really afford food. I’m living in Sunnyvale—I can’t afford to stay here and can’t afford to move, especially since I’d need a regular income to move to a new, cheaper place. But frankly, I don’t feel safe here. I don’t feel like this is a home.

I’m desperately looking for any assistance that can be provided. Specifically, I need some temporary help to afford the health insurance and rent. I’ve been seeing a therapist for many years, and it was determined that I definitely have ADHD (along with major depressive disorder). While I’m being treated for the major depressive disorder, I can’t afford the actual testing for ADHD, which also takes time I don’t have to get a proper medication.

My ultimate goal is to find some contracting work for macOS/iOS where I can use my reverse engineering and bug fixing/finding/working around skills. I miss figuring out how things work, something I could do in spades while working on macOS Notarization at Apple.

I also miss when people could launch a potato salad fundraiser on Kickstarter or another fundraising site. Since some people have asked, I do have the Apple Cash at my first name (see above) at the computer Apple makes dot com. And PayPal at the same address, but I think it’s misconfigured—it says you’re supposed to receive a product if anything is sent there, even though there are no products. Also, I have that Cash app thing, $<my first name>, I think? As for food, I have an Amazon Wishlist, but I don’t know how useful it is or what can be put on there.

I can help you solve problems with other people’s code. My résumé is available, if that’s worth something to you, please contact me.

Love Tropicana: The Fix for securityd Eating Gobs of RAM When Updating Keychain Entries

A few nights ago I was updating some not-to-be-named software on my laptop. This piece of software had a few passwords stored in the Keychain. Since said application was recently updated and therefore the code was modified, the system asked me if I wanted to give access to the keychain to this updated application. The dialog that it shown to the user is shown below:

Bad things happened when I clicked the "Change All" button to once again allow this updated application to access all the passwords it was allowed to access. Specifically, the securityd process was using 1.3-1.7GBs of ram (the rprvt value is all that matters). This was really, really bad as it caused my machine to page-out and page-in like crazy. Due to the high memory usage, it also caused my boot volume to run out of space because of all of the swap files in /var/vm/. My point is that very, very, very bad things happened. After I cleared a lot of unused crap (Garage Band loops and old iDVD themes) off my boot volume, I rebooted. I then tried launching the updated application again. I got the same dialog and the same problem. However, since I now had enough hard drive space available, I just waited for about 10 minutes. The passwords were accessed successfully. I then relaunched the application and securityd crashed. Lovely. Rebooting just repeated the cycle. Also lovely.

The securityd daemon handles all authorization and marshals keychain access. If securityd  is not running, you cannot authorize any application (although sudo still functions) and all calls from applications to the keychain either immediately return an error without asking for a password or permission or they just freeze indefinitely. This is also very, very bad. It amounts to a denial of service since you're unable to do much with your computer.

So I clearly had a problem. I remember seeing this problem before while browsing the internets. I specifically found it on 43 Folders. I started searching The Google for any solutions. The searches brought up some really amusing "solutions" such as a thread from Mac OS X Hints. People are talking about some crazy voodoo solutions there. Examples include updating the firmware or adding a "keychain" extension to the user's keychain. Some voodoo causes are also mentioned. One such example is that it is cause by syncing keychains between an ICBM and a PowerPC-based Mac.

The Troubleshooting

Ok, so obviously there were no known solutions to the problem that I could find. This kind of pissed me off. Especially since I had to wake up at 5am to make it to a Microsoft Vista launch event. It was 3am at the time. Obviously, there wasn't going to be any sleep for me.

I decided to do my own troubleshooting. The crash of securityd clearly pointed to a corrupted file of some sort. So I had to determine which file was hosed like a bear coming into contact with the strikingly handsome Stephen Colbert. First, I restored all my keychains from known-good backups that another Mac was actively using. I rebooted and tried to reproduce the problem. Bloody 'ell! The problem remained. So it obviously wasn't that the keychains were corrupted. I rebooted again, started up the terminal, ran sudo fs_usage -f filesys securityd to see what files securityd was accessing when it was trying to update the keychain reference for the application. /var/db/SystemKey repeatedly showed up. However, I was very, very tired and getting very, very angry so I decided to go with a machete approach to fix the problem. I ranstrings on securityd and it turned up four "interesting" files and folders:

/var/db/SystemKey
/var/db/TokenCache (this is a folder).
/var/db/SystemEntropyCache
/var/db/CodeEquivalenceDatabase

Since I was so angry, I just "moved aside" three of these. "Moved aside" meaning that I just renamed everyone of them to have a ".old" extension using the Terminal (via mv). The TokenCache folder did not exist on my ICBM so I couldn't move it aside. I rebooted.The problem was fixed!. I then went to the Microsoft event. I emailed the very sexy Merlin Mann with the possible shoot-everything fix. He reported back that it fixed the problem for him, at least. Hallelujah!

The Actual Fix

I was pissed off at the fact I wasn't able to narrow down the problem to one file. So after a very good 14 hour sleep (after being awake for over 30 hours), I set about to confirm the issue. I started by examining the files and comparing them to the files on my PowerPC Mac. The TokenCache folder didn't exist on the ICBM and the SystemKey didn't exist on the PowerPC Mac. So I could rule those two out. I examined theSystemEntrophyCache file and its contents were boring. So I was able to rule that out. I then opened the CodeEquivalenceDatabase in TextMate (an awesomely awesome overnight text editor) and the text strings matched the applications that currently have Keychain access. So I put the "moved aside" CodeEquivalenceDatabase file back to its original location and rebooted. I then opened Transmit (an awesomely awesome FTP application) which I had just updated and the problem reoccurred. YAY! We (the royal We) had just narrowed down the exact corrupted file that was creating the problem.

In order to fix this problem if you are having it, just open the Terminal (/Applications/Utilities/Terminal) and type:

sudo mv /var/db/CodeEquivalenceDatabase /var/db/CodeEquivalenceDatabase.old

or

open /var/db (and then manually move CodeEquivalenceDatabase to the trash, if you can).

Upon rebooting, God should be in His Heaven and all should be well with the keychain.

Note 1: If CodeEquivalenceDatabase is corrupted, then updating Mac OS X will also cause securityd to eat gobs of memory near the very end of the update cycle when the installer updates Apple applications. This may make it seem like the update stalled.

Note 2: The /var/db folder is interesting. It's not required to boot Mac OS X and all the files in there are created as needed by Mac OS X. It's also explicitly ignored by repairing permissions. However, it does hold important account information, so I would not delete the folder or the items inside unless you are trying to revert your Mac to "factory fresh" and do not wish to have any of the same accounts available without recreating them.

Exercises in Futility: Repairing Permissions is Useless

 Update: I've reposted this blog entry here on Squarespace as a test. Some links do not work any more and some information is outdated, but the facts remain facts.

Update 2015-06-16: Common sense has won! Mac OS X 10.11 "El Jefe" removes the Repair Permissions functionality! And it now automatically checks for proper permissions during a Mac OS X Update or Security Update! Quoth the Raven: «System file permissions are automatically protected, and updated during Software Updates. The Repair Permissions function is no longer necessary.»

DEAD DEAD DEAD!

Update 2015-06-16: I've updated this article with some other relevant changes that have been made since it was originally written. Including information on Mac OS X 10.11's new System Integrity Protection (rootless) features. Search for "2015-06" to locate the changes quickly.

This is a rant I've wanted to write for an extremely long time. However, I prefer to let my anger/annoyance with some topics sit in the stew that is my soul and slowly boil until it is like you dropped a tea bag into a cup of very hot water housed in a smooth glass container that you just stuck in the microwave for 10 minutes. Yes, doing that will cause the water to spontaneously explode, leaving horrible burn marks all over your face (just be glad it wasn't maple syrup or something else that could stick to human flesh).

Update: Despite what Apple's Knowledge Base Article says, Repair Permissions does not repair permissions on any third party software (or any Apple software outside of the Base System). I just checked this by changing the permissions on FontAgent Pro and then repairing permissions. The permissions were not set to their correct values. This makes repairing permissions even more useless as it can't be used for any non-apple software. It also explains why installing CHUD or iPhoto from iLife '05 would cause the incorrect warnings to appear.

Now that that pleasantness is over with, the real issue I have is all these websites that suggest repairing permissions will actually fix/prevent problems. Even worse is when otherwise intelligent people are poised with a Mac OS X related troubleshooting problem and immediately suggest the user repair permissions. Repairing permissions won't fix your problem. As Jason Harris said (and I am paraphrasing), "Repairing permissions is zapping the PRAM for the twenty-first century". I couldn't agree more. Both are equally futile attempts to fix a completely unrelated problem. In other words, 99% of the time, neither will fix nor prevent any problem. Especially not the problems they are recommended for. Covering yourself in vaseline and rolling around naked in the dirt and repairing permissions are just as likely to fix your Mac OS X problem. People swear by repairing permissions as often as they save files and present as proof the fact they don't have any problems. That's rather specious reasoning. As with everything in life, The Simpsons has covered this topic well. Yes, I've shamelessly copied this text verbatim.

Homer: Not a bear in sight. The Bear Patrol must be working like a charm.

Lisa:That's specious reasoning, Dad.

Homer: Thank you, dear.

Lisa: By your logic I could claim that this rock keeps tigers away.

Homer: Oh, how does it work?

Lisa: It doesn't work.

Homer: Uh-huh.

Lisa: It's just a stupid rock.

Homer: Uh-huh.

Lisa: But I don't see any tigers around, do you?

[Homer thinks of this, then pulls out some money]

Homer: Lisa, I want to buy your rock.

[Lisa refuses at first, then takes the exchange]

Please ignore the fact that they're talking about tigers and Mac OS X version 10.4 just happens to be called "Tiger". The point stands. Just because something isn't around because you do something every day does not mean it would suddenly come around when you stop doing the something every day. Some people used to smear chicken blood all over their kin to keep the devils away too.

Call Me Lucy

Update 2015-06-16: The location of the Receipts folder moved in Mac OS X 10.5 to /Library/Receipts/db/. The Receipts folder moved again in Mac OS X 10.6 to /var/db/receipts/ and the BOM files are no longer stored inside packages (.pkg). The BOM files are now saved in a flat manner directly in the root of /var/db/receipts/. The information and commands listed below should be adjusted for these changes.

I guess I should explain what repairing permissions is and isn't. Repairing permissions goes through all the Package files (.pkg) in /Library/Receipts/. A receipt package is created when (and only when) you install something using Installer.app (Apple's installer). First it creates a temporary package based on all the files in the package. Then when you install, it creates the actual package that actually contains a listing of all the files you installed from that package. You can see these two steps if you open Installer.app's Log window and choose "Show everything". The package lists the paths for all the files along with the permissions Installer.app set for them when they were installed (those permissions are part of the actual package in which the files were installed from). The items in /Library/Receipts/ are basically just the shell package without any of the actual files inside. You can see the contents of these by using the lsbom utility. The usage is basically lsbom path/to/archive.bom. Like so:

lsbom /Library/Receipts/MacOSX10.4.pkg/Contents/Archive.bom

This will list all the files that were installed by the package (by absolute path, usually), their installed permissions, file size, and some other information. See the man page for lsbom for more information on the output.

Anywho, when repairing permissions, the disk utility goes through the permissions of all the files in the target volume's /Library/Receipts/ folder. Apple has a kbase article on this as well. In order for the "repair permissions" or "verify permissions" button to show up, the target volume must have a version of Mac OS X installed on it. Repairing permissions only works on volumes that have a /Library/Receipts/ folder. Which is only there if OS X is installed on that volume.

Based on this information (and the sheer stupidity of Installer.app) you can correctly assume that Repair Permissions won't touch any files in any of the user's home folders since Installer.app can't target user folders specifically, only any folder or a specific path, and there are no packages in ~/Library/Receipts/. The only way it'd ever touch any files in a user's folder is if you installed something that let you explicity select a folder to install in (there are very few of those, none are available from Apple publically) and you chose a folder inside your user's folder. The receipt would still be installed in /Library/Receipts/ and it would only affect the user that installed it. It also won't fix permissions for any files that were created during the normal (or abnormal) use of OS X. This means it won't touch any cache files, database files, swap files, or settings files not created by the installer. If a file isn't listed in a receipt, it doesn't exist to the repair permissions process. It's really as simple as that. And because it reads from /Library/Receipts/ on the target disk, you can boot from a Mac OS X 10.2 CD and still use it to (correctly) repair permissions on a volume with Mac OS X 10.4.6 installed on it and it will set the permissions to the ones that 10.4.6 requires. There is no need for you to boot off a volume in order to fix its permissions.

A Bit of History

Does anyone actually remember when Apple first offered the ability to repair permissions and why it was needed? I do. Apple introduced the Repair Privileges Utility as a download for machines running Mac OS X 10.1.5. This was back in the long ago time when Macs would still boot Mac OS 9 and the default environment for a lot of Mac users was still Mac OS 9. Mac OS 9 didn't care about permissions at all. Mac OS 9 was Mac OS X's worst enemy in this area. If you booted into Mac OS 9 and ran some common applications, compressed and decompressed files, moved or renamed files, or (worse) ran a disk utility like Norton, they could completely destroy the permissions for many files that OS X needed to boot or run correctly. Since this was a relatively common occurrence and ahuge support issue, Apple introduced the Repair Privileges Utility. When Mac OS X version 10.2 "Jaguar" came around, Apple rolled it into Disk Utility where it belonged. But by this time it wasn't needed nearly as much since many new Macs couldn't even boot Mac OS 9 thus rendering the fear of bad disk utilities that didn't pay attention to the rules set out by the HFS+ Technote ruining the ability to run OS X completely moot. Just because HFS+ didn't use the features when you wrote the disk utility doesn't mean it won't in the future (and the future is now, excluding named forks). None of this applies to using Classic as Classic lives in a mostly happy sandbox with just a trace amount of urine (which is more than I can say for my spacesuit). 

The other number one cause of permissions going wonky were 3rd party installers that asked for root on OS X and changed permissions on some folders that were in the path to the destination. I know that MindVision and Allume (Courtney Cox-Arquette) have long since updated their installers to prevent this kind of weirdness (these would be the same installers that told you to quit all applications when installing software on OS X). I know there is always a chance I could be wrong about things like this so I've been running Repair Permissions after every install using these two installers just to make sure, and I haven't seen it note anything at all. So these two companies' installers are good. So this cause is also deemed completely moot.

The final minor cause for incorrect permissions is, basically, the user. I've said it before, and I'll say it again, you can't account for stupidity. People that change permissions on something because they think it might fix a problem they're having because they know more than the system but it very obviously won't fix the problem. More on this reason later.

Update 2015-06-16: Mac OS X 10.11 "El Jefe" apparently can account for stupidity with a feature called System Integrity Protection (also known as "rootless"). With SIP enabled, even root (the superuser) cannot change the permissions or the content of required system files. This means that it is no longer possible to enter into a situation where the permissions on key system files have the wrong permissions. While SIP is enabled by default, an extremely determined idiot can boot into Recovery Mode to disable SIP. However, disabling SIP removes a lot of security protections in addition to permitting wonton permission changes when root. For example, it becomes possible for malware to hijack system processes with task_for_pid() (in layman's terms, it's a function used at the beginning of a list of steps needed to allow one running application to insert its code into another running application).

The Ugly

Permissions won't magically go bad. They won't break suddenly. They don't suffer from bit rot. In order for permissions to change something must change them. Even with Mac OS 9, as long as you didn't modify the files, their permissions wouldn't change. For this reason, repairing permissions as a maintenance task is just a complete waste of time. Granted, it won't harm anything*, but it won't help anything either. If you want to be paranoid about permissions then at least use some common sense when doing it and only run repair permissions after using an installer. Of course, I am not recommending repairing permissions after running an installer!

*Correction: Certain Mac OS X versions had a security bug that caused Repair Permissions to actually revert some security improvements made in Security Updates to there original, insecure state.  

I've only ever found two cases in which repairing permissions would actually help fix something. One is when backing up to a non-local (external) volume (like a FireWire HD, a USB 2.0 HD, or a disk from FireWire Target Disk Mode) via ditto (with the --rsrcForkoption passed) or Carbon Copy Cloner. You really shouldn't use drag and drop in the Finder or cp to backup OS X volumes as important metadata can get lost that way. There is one reason and one reason only I say that the use of ditto or CCC would require a repair permissions on the volume everything is being backup up to--when the volume has the "Ignore ownership on this volume" checkbox checked. Ignore Ownership on this volume should be unchecked. If it isn't, then none of the permissions will be copied over in the first place and then you have to run repair permissions to make the thing bootable. But I still don't recommend it as any files not installed by the installer won't have the permissions fixed and could be a huge security risk (especially if it is world writable file). So make sure that box is unchecked. It's checked by default on non-local volumes. Also note that if you don't uncheck that box, you will have to reinstall Application Enhancer as it is permission sensitive and if that box is checked while you're doing the copy/backup it won't have the proper permissions to run. This has come up a lot in support emails but by then it is too late to tell the user to uncheck the box. This item is preventable.

Update 2015-06-16: Sometime between when this article was originally written (when Mac OS X 10.4.6 was released) and when Mac OS X 10.9 "Mavericks" was released, the importance of correct permissions on files in /var/db/ increased significantly and Mac OS X will actually refuse to boot if any of the permissions in /var/db/ are incorrect. As /var/db/ is explicitly ignored by the Repair Permissions process, it has become absolutely impossible to resurrect an install of Mac OS X  that was damaged by forgetting to uncheck "Ignore ownership on this volume" in the above manner.

The other case when repair permissions are required is also in the user error category. But this is when installing system services using very outdated instructions that tell you to assign the wrong value to a now default OS X user. I can't blame them because I thought that I knew what I was doing.In this case repairing permissions was the fix (as long as I made the postfix uid 27). I could have easily fixed them manually, but that was a few more keystrokes and I was feeling keystroke lazy. This item is preventable but hasn't applied in a very long time. It has also become moot as Mac OS X started including postfix by default and can be controlled easily by installing the $20 Mac OS X Server application.

Anecdotes are not Proof of an Antidote

Now let's see what some very silly people are saying about repairing permissions and why it is flat out wrong.

7) Repair Permissions

8) Install Mac OS X 10.x.x update

9) Repair Permissions

Ugh. This is the one that annoys me the most and the reason why I wanted to write this rant. When 10.3.9 came out, I saw this all over on just about every single Mac related website. This really boils my blood. First we have the completely bizarre suggesting of repairing permissions before installing an update. Useless. When you install a system update (or pretty much any updater using Installer.app) you are asked for your password. This makes the installer process run as root. Wrong permissions, bad permissions, no permissions, it doesn't matter. root is god. It doesn't care about what some small little file has as its permissions. It will just ignore them completely. *chortles demonically* You can't stop root. Repairing permissions won't increase the chances of the install succeeding (nor will it decrease the changes). Point nine is equally as baffling. As the installer is installing/updating files it also reset the permissions to those that will be in the receipt that the repair permissions process reads from. You just installed these files, they are going to have the correct permissions.

I was having problems with permissions with files on the scratch disk so I repaired permissions on it and it fixed everything.

This is a paraphrase of something someone actually said to me. It is real. It is a lie. I can say that without even knowing how this person handles problems (even though I do know how he handles OS X problems). Why? He says "scratch disk". For those that have never worked with media or photoshop, scratch disks are dedicated volumes (usually separate physical disks, in this example it was another partition) that is used to store all the render files and temporary media files for a project. They are used often to increase render speed when applying compute heavy effects to files. They are separate disks so they can be read from/written to quickly without having to compete for I/O time with the boot disk. It's also good to keep them separate so if Project A needs scratch disk space, it can easily delete all the scratch space for Project B while making sure that Project B doesn't suffer by having to reimport media. Although if you do delete Project B's scratch files, you'll likely be stabbed anyways as it can take hours to rerender complicated projects (but on the upside, the people working on Project B don't really have to do any work to rerender).

Since, by definition, a scratch disk does not have an OS installed on it, there is no way he could have repaired permissions. No OS, no /Library/Receipts/ which means no repair permissions. The actual fix to his problem? Get info on the scratch disk and check"Ignore Ownership on this volume". The problem appears to be that multiple users are using that scratch disk with some of them working on the same projects. If a user named "rosyna" creates a file, its default permissions will be 0644 with owner rosyna and grouprosyna, This means if user "slava" comes on and tries to write to that file, he can't since he is neither me nor a member of me. Ignoring ownership would mean that everyone can read and write whatever file they want on that volume.

iTunes kept throwing up an error saying "Cannot save library. -54". So I opened a get info window on the iTunes application and gave everyone the ability to read and write to it along with all enclosing items and the error when away. But every time I repair permissions, it keeps stupidly resetting the permissions back and the error comes back.

Same person as above. A lot more paraphrasing was done. The error is the same (Library, can't save, and -54). His actions on the iTunes application bundle are the same, and he did indeed say it fixed the problem. Not only is it a huge security risk to give everyone write access to an application, it is a lie. Yes, a -54 error is a permissions error (on file open). But note that the error said "Can't save Library". Library refers to the iTunes library. This file is located at ~/Music/iTunes/iTunes 4 Music Library. Note that it is stored in the user's home folder (that's what the ~ means). This means two things. 1) Repairing permissions won't fix the problem. 2) His problem has nothing to do whatsoever with the iTunes application. Not that it matters since an application on OS X must not write to itself. This was a huge source of problems in Mac OS 9 so resource forks that are opened automatically on application launch are opened read only on Mac OS X.

This fix for this (as I know it, I've only had the problem once) is sadly rather invasive. Delete the iTunes 4 Music Library. Reopen iTunes, choose "Import" from the "File" menu and import ~/Music/iTunes/iTunes 4 Music Library.xml. It will take a while and you will lose some data. But it will import all the files that it can find on your computer. It sadly won't re-add files that it couldn't resolve at the time iTunes last made the XML copy of your library which should have been the last time you quit iTunes successfully, the last time a change was made to the iTunes library, or the last time someone connected to your iTunes share. Your playlists will be retained although you might get duplicate copies of your smart playlists (I did of a few) and the order of some of your larger playlists might be completely screwed up.

Normally I don't say something is a lie (or call someone a liar) flat out in public (or even privately) and this wasn't a lie done for malicious purposes. It was more of a lie in the sense that this person isn't quite sure what actually was happening and likes to argue endlessly and when presented with explanations for what actually happened tries to make it seem as if the people giving the explanation are wrong (inherently and always). He changes the past ever so slightly to suit him best hoping that no one will catch on. His memory isn't that good either.

I was having this huge problem with some huge third party application and repairing permissions fixed it.

Extremely doubtful. Most of the time huge third party application is something by Microsoft, Adobe, or Macromedia. None of which use Installer.app. Which means that repairing permissions couldn't have possibly fixed any issue that only one of these applications was experiencing. For example if the permissions for /Library/ were set to 000 then all hell breaks loose. And I'm not kidding about that. Disk Utility stopped working, ShapeShifter stopped theming, and Adobe Photoshop CS 2 (9.0) showed me this on launch, without holding down any keys as startup:

How to Solve Problems and Stop Repairing Permissions

It'd really help a lot if people would stop trying to assume repairing permissions does good and actually try to find out what's causing a problem they're having. For the people willing to try, I point to the the Console (/Applications/Utilities/Console). The console has the answers you seek, young one. For just about any problem with anything, the Console log will list it for the above Library example, Photshop actually told me that it couldn't read files from the Application Support folder (which is correct). But make sure that what you're looking at actually relates to your problem. If Virex keeps crashing and you see:

May 15 15:35:09 Stability-64 kernel[0]: Limiting icmp unreach response from 314 to 250 packets per second

In the console, do not assume it has anything to do with your problem. For my postfix example above, the console clearly has the following to say:

May 16 12:39:00 Stability-64 postfix/master[487]: warning: master_wakeup_timer_event: service public/qmgr: Permission denied

May 16 12:39:05 Stability-64 postfix/master[487]: warning: master_wakeup_timer_event: service public/pickup: Permission denied

Notice how it says "Permission denied"? That means it doesn't have the correct permissions and fixing permissions will definitely fix this problem. For what it's worth, I changed all the files in /var/spool/postfix to be owned by rosyna.

In summary, repairing permissions is useless 99.999% of the time. Permissions don't magically go bad. And you should really throw out "repair permissions" as a troubleshooting step until you've exhausted all other methods of troubleshooting. Reparing permissions can only fix problems with Mac OS X system files distributed by Apple installed by Apple's Installer.app and only as long as a valid receipt is available. Even then, it should never be suggested to others as it won't help and sadly will only make you look like you know less about Mac OS X than you really do. Repairing permissions is the strategy of last resort!

Update 2015-06-16: The useless of repairing permissions is now 99.999% and when Mac OS X 10.11 is released with rootless, the uselessness of repairing permissions will become 99.999999%. I hesitate to say 100% because I know at least one user will disable SIP/rootless in El Jefe and manually change the permissions of a file that's supposed to be protected despite all the protections added.

I guess what I'm trying to say with all this is that if you're going to heat up water in the microwave for use in tea, put the teabag (without the metal) in the cup of water before you heat it up and keep it in there (without metal) while heating it up.