A 20-year retrospective on reverse-engineering an Apple bug and a very desperate plea for help.

A little bit of History

I thought it’d be helpful to have a retrospective of a little reverse-engineering task I did about 20 years ago, when Mac OS X 10.4.x was at the dawn of the PowerPC to Intel transition. (Mac OS X 10.5 was the first merged release.) I gathered all this information through reverse engineering, including reading PowerPC disassembly and using a debugger. None of the information was acquired by actually examining source code, as I didn’t have access to said source code.

A brief background on the process/application with the bug

In Mac OS X 10.4, a process known as SystemUIServer was responsible for showing menu extras (but not status items at the time), taking screenshots, providing the Spotlight user interface, and handling some iPod/Digital Hub device events. It also held all the code for some of Apple’s own menu extras—the menu extra plugin bundles for those extras were just stubs to enable the menu.

A screenshot of SystemUIServer from Mac OS X 10.4.3 being decompiled in Hopper with the built-in menu extra code.

Most people’s interactions with SystemUIServer were solely with the true menu extras on the menu bar’s right-hand side.

Today’s SystemUIServer with menu extras, status items, Siri, and control center items

At the time, Apple only allowlisted specific menu extras by class name (checked in -[SUISStartupObject _canLoadClass:]). Any attempt to load a menu extra that advertised a different class name in its Info.plist’s NSPrincipalClass entry would fail. Menu extras were first-class citizens. You could hold the Command (⌘) key down to move or quit them. They loaded automatically and needed no backing application. This sent lots of developers looking for workarounds to Apple’s allowlist, as they wanted these features for their own menus.

Some developers would steal one of the allowlisted class names for their menu extra plugin. This was unwise. The Objective-C runtime only allowed one class instance with a specific name to be loaded at a time. When it found duplicates, the one it chose was an implementation detail that could cause unexpected crashes. Favorite choices of class names to hijack included AppleVerizonExtra or IrDAExtra, i.e., something a user isn’t likely to have enabled. In the rare case someone did enable these, or if more than one developer chose to steal the same class name, all hell could break loose.

This was the impetus for Menu Extra Enabler. It was an old-style InputManager¹ plugin that automatically loaded into SystemUIServer when installed and overrode the -[SUISStartupObject _canLoadClass:] instance method to return YES unregardless of what class name was used for the menu extra’s principal class.

¹InputManagers were a cheap and quick way to get into the address space of arbitrary Cocoa runtime applications at a determinate location.

In Mac OS X 10.3.x, this allowlist limitation bit Apple in the calyx. Apple added a Keychain menu extra in /Applications/Utilities/Keychain Access.app/Contents/Resources/Keychain.menu without updating the allowlist to include the new principal class, so Apple created a hacky Enable.menu to allow it. This Enable.menu enabled the Keychain menu extra, and just for a bonus, also disabled all third-party menu extras. I updated Menu Extra Enabler to prevent Enable.menu from loading—Menu Extra Enabler already allowed Keychain.menu to load, so the hack wasn’t necessary.

The Bug in Mac OS X 10.4

Mac OS X 10.4.x introduced a new problem. If the system delayed the launch of SystemUIServer for long enough, the Bluetooth menu extra would not load even if Bluetooth was enabled. In fact, SystemUIServer seemingly "erased" all menu extras to the right of the Bluetooth menu item! At Unsanity, we received many reports blaming missing Menu Extras on ShapeShifter, with a few people blaming it on Cee Pee You.

Neither caused the problem. In fact, the underlying "problem" could have occurred days before you applied a theme or started Cee Pee You, since it happened only when logging into Mac OS X and at no other time. It also occurred whether or not any third-party menu extras were in use (and even if Menu Extra Enabler wasn't installed). I remember browsing forums and seeing a screenshot of the late Dwyane McDuffie’s menu bar, and he had been bitten by the bug—only the Bluetooth menu extra was visible.

After five straight hours of reverse engineering and debugging, I found the sequence of events that triggers the bug:

1. The user enables Bluetooth and the Bluetooth menu extra.
2. At some later time, the user starts a new login session with a bunch of login items (iChat, Extensis Suitcase, TextEdit, Console, Word, Alfred, whatever)—just enough to delay the normal initialization of SystemUIServer.
3. When the system gets to it, it launches SystemUIServer. It reads its preferences from disk, including the list of menu extras installed last time it ran. It then registers for notifications, calling -[[SUISDocklingServer alloc] initWithController:docklingServer] in -[SUISStartupObject init] so it can respond and install future menu extras when they’re opened. Without this, you’d have to log out and back in to see any new menu extras. So far, so good.
4. Like other Cocoa runtime applications that implement NSApp delegate methods, SystemUIServer receives the -[SUISStartupObject applicationDidFinishLaunching:] call when everything’s ready to go. Loading menu extras here causes some kind of problem, so instead, SystemUIServer takes this opportunity to create a timer to call -[SUISStartupObject _loadMenuBarExtras] after a 20-second delay.
5. The Dockling Server decides it’s now time to load the Bluetooth menu extra, so it issues a notification (see step 3) that eventually calls -[SUISStartupObject addMenuExtra:position:reserved:data:] on the Bluetooth menu extra. The 20-second timer has not elapsed.
6. -[SUISStartupObject addMenuExtra:position:reserved:data:] takes the Bluetooth menu extra and passes it to -[SUISStartupObject createMenuExtra:atPosition:write:data:], which eventually calls -[NSMenuExtra initWithBundle:] and the Dockling server thinks it’s installed the menu extra.²
7. Now that the new menu extra is supposedly loaded, SystemUIServer saves its menuBarPlugins array, writing the preferences to disk.

• The 20-second timer has not elapsed, so the array contains one item: the Bluetooth menu extra. This list of installed menu extras is not reread from disk until SystemUIServer is relaunched, usually at the next login.

8. When the 20 seconds noted in step 4 have elapsed, the timer fires and the system finally calls -[SUISStartupObject _loadMenuBarExtras]. It’s time to load the menu extras! First, the method creates the NSMenuToolbar³ where all menu extras live. After that, it looks at the list of previously-installed menu extras (read from disk in step 3), loads each of them, and makes them into NSMenuToolbarItems added to the main NSMenuToolbar. Everything’s coming up Milhouse!

So what’s the problem? There’s no Bluetooth menu extra! Let’s go back to step 5 and see what really happened.

5. a. Since the NSMenuToolbar that hosts all menu extras doesn’t get created until step 8 (after the 20-second timer elapses), everything triggered by -[SUISStartupObject addMenuExtra:position:reserved:data:] fails.
6. a. SystemUIServer does not check for failure—if it can’t load a menu extra, no big whoop; it’ll just keep iterating through the ones it has.
7. a. But it does matter when it saves (to disk) the preferences containing only the Bluetooth menu extra, which isn’t even loaded. At this point, the list of all menu extras to load next time has been erased. The poor user probably doesn’t notice this—SystemUIServer has already loaded the list for this session (step 3) and doesn’t read back the mangled set of preferences until it launches again, usually at the next login.
8. a. If the list of menu extras from last session contains the Bluetooth menu extra, it still doesn’t load. All subsequent calls to -[SUISStartupObject createMenuExtra:atPosition:write:data:] with the Bluetooth menu extra return NULL (fail). That’s because _alreadyHasExtra: returns YES since it thinks it loaded the Bluetooth menu extra in step 5. So when the bug hits, the Bluetooth menu extra never gets loaded.

² This is foreshadowing!
³ Although NSMenuToolbar is named like an AppKit class, it only exists in SystemUIServer in this story.

I figured out all this after 5 hours of straight debugging/reverse engineering.

The Solution

Unsanity customers thought that Menu Extra Enabler was causing the problem since it optionally restarts SystemUIServer when applying a new theme, thereby reading the mangled preferences from disk, not trying to load anything but Bluetooth. Others blamed Cee Pee You since it restarts SystemUIServer during installation (it has to install Menu Extra Enabler, which is only loaded on process launch).

My first fix was to prevent SystemUIServer from saving menu extras if the new list contained only items to the left of the Bluetooth menu extra and the Bluetooth menu extra itself. This failed if the user manually removed menu extras to the right of the Bluetooth menu extra (by holding the command [⌘] key and dragging them off the menu bar in a “poof”), since the list of loaded menu extras was saved on a timer, and the user could remove multiple menu extras before the timer fired and the list was finally saved to disk.

It was also extremely hackish and required a lot of code (about 50 lines). I was not happy with this at all. Not only did it not fix the problem (just the outcome) since the Bluetooth menu extra still wouldn't load in these cases, but it could also break depending on the user's actions. It annoyed me so much that I woke from sleep after creating this horrible hack. So, when I woke up, I tried again, from scratch.

The new fix with this understanding is only 5 lines of code; it is future-proof and doesn't require the user to do anything.

Apple did fix this bug in Mac OS X 10.5, and Menu Extra Enabler already had code to disable my workaround of not allowing the Bluetooth menu extra even to attempt to load (step 5) if the NSToolbarMenu wasn’t set up yet (step 5a) if the user was running Mac OS X 10.5 or later.

When you do things right, people won't be sure you've done anything at all.

Using Xcode to debug plugins

Menu Extra Enabler was a plugin that had to be in a special location (~/Library/InputManagers/) to function, and loaded in an application to which I had no source code. I had to take advantage of a neat Xcode feature: you can set an arbitrary process/application to be a target’s executable, and it’ll automatically enable breakpoints in your plugin. It’s extremely useful for debugging plugins!

Xcode’s Scheme Editor, Run Section

The necessary Run Script build phase to copy the built plugin to the Proper plugins folder 

This was slightly difficult to do with SystemUIServer, as Mac OS X 10.4 automatically launched it if it wasn’t running. So, there was a lot of killall SystemUIServer in Terminal to keep it dead until Xcode could launch it.

For an application to be debugged in today’s macOS versions, it must have the com.apple.security.get-task-allow entitlement. This is the default for development apps, but those don’t usually run on other people’s computers—they won’t have the development certificate used to sign the app. To prevent developers from accidentally notarizing development versions of their apps that have the com.apple.security.get-task-allow entitlement by default, we at Apple made an explicit decision to allow this entitlement if and only if paired with the disable library validation entitlement (required to load third-party plugins if you notarize your macOS application).

This allows application developers (like Adobe) to ship notarized applications (like Photoshop) that run third-party plugins in-process, while still allowing third-party plugin developers to debug their plugins without having to disable any of the newer macOS security features, like System Integrity Protection or the Signed System Volume.

What’s changed in 20 years.

The Rules

Input Managers are now dead, the allowlist for class names for menu extras is no longer needed due to library validation, and you cannot attach to system processes such as SystemUIServer by default. To prevent (or at least discourage) people running Mac OS X on generic PCs, Apple “encrypted” SystemUIServer and a few other Mac OS X 10.5-and-later executables for use with Don’t Steal Mac OS X.kext.

To add to this, many elements that were previously part of SystemUIServer have been either completely removed (iPod support) or extracted from SystemUIServer into components like Control Center, Notification Center, the Screenshot process, and other plugins, app extensions, and other processes, as if there was a fire sale on the components and everyone wanted a souvenir.

Arbitrary in-process plugins, like menu extras, have always been inherent security and stability risks (and a way to do super neat, unexpected things, per my old job). Apple has bet heavily on out-of-process plugins to solve issues caused by the older ways, like XPC/ExtensionKit for Mail plugins, Xcode plugins, QuickTime plugins (yay!), and Kernel Extensions.

Furthermore, the main reason for using third-party menu extras instead of status items was finally addressed in macOS 10.12, when status items became first-class citizens in the menu bar. They still need a backing app, but they can be moved and removed by holding the command key down and dragon-dropping them around.

The Tools

Long before Gen Alpha was born, I would use otool (the variant known as otx) dumps and feed them into BBEdit to view the absolutely huge files it generated, as BBEdit was the only text editor that could/can handle that much text without falling over backwards onto a pike—as long as word-wrapping was disabled.

The Objective-C runtime requires special processing for connecting methods to call sites in reverse engineering tools. With Objective-C, method calls go to a runtime function (like objc_msgSend()), which then calls the correct method on the proper class after some magic. Due to this indirect calling and highly dynamic dynamism, reverse engineering tools can’t just do the usual “double-click to view function call” on Objective-C code; they need an extra step to connect the methods and calls through Objective-C runtime functions.

On the other hand, Swift adds a lot of boilerplate code to applications, even for basic operations on a class or struct. Great reverse engineering tools with decompilers should recognize the boilerplate code and remove it to show what the developer wrote, not just what the compiler wrote. It’s a minor pet peeve of mine when reverse engineering Swift code.

Hopper

Hopper is a first-class citizen on macOS. 

SystemUIServer in Hopper

For Hopper and Objective-C 4 2.0, Hopper kindly does the extra work of connecting runtime functions to the actual class and method being invoked, if those are available through static analysis. It has an SDK and some plugins. Hopper is a $99/year subscription.

National Security Agency’s Ghidra

When you hear that the US government’s spying agency decided to release a free reverse engineering tool, “NSA’s Ghidra," you may fear exactly what I did, getting shivers down your spine and making you wary: “It’s written in Java, isn’t it?”. Yes, it’s written in Java. It’s not a first-class citizen on macOS. It’s an ugly tool. Bitmap graphics are everywhere. It feels like it escaped from the late 1980s. The installation method is a mess.

SystemUIServer in the NSA’s Ghidra. Semi-supports decompiling 32-bit PowerPC apps!

But it’s completely free and has some really, really good plugins written for it to use with Apple platform (iOS, macOS, et fam) binaries, like LaurieWired’s absolutely fabulous Malimite. It also supports the Objective-C runtime well.

IDA Pro

IDA Pro is a very, very expensive reverse engineering tool. In 2022, private equity firms bought it, removed the perpetual licenses, and raised the prices by several thousand dollars. Prices for a secure version of IDA Pro range from $2,999 to $8,599. It’s not cheap.

SystemUIServer from Mac OS X 10.4 in IDA Pro. Look at the iPod SCSI notification! 

IDA Pro does have a vibrant plugin community. I’m using a very old version and I’m not sure if the built-in Objective-C support has been improved, but I’ve found it lacking.

A desperate plea for help.

One year ago, I was “let go” from Apple as I had used up all my disability leave. It was either “come back, or quit”. So I came back, even though my disability wasn’t treating me any better. I was shortly thereafter let go because my disability was incompatible with productivity, especially under a boss who couldn’t understand non-physical disabilities.

Over the past year, I’ve completely burned through all my savings. This month (May 2025) is the last month I’m able to pay rent. I’m completely broke, even in debt after paying May’s rent. I can’t really afford food. I’m living in Sunnyvale—I can’t afford to stay here and can’t afford to move, especially since I’d need a regular income to move to a new, cheaper place. But frankly, I don’t feel safe here. I don’t feel like this is a home.

I’m desperately looking for any assistance that can be provided. Specifically, I need some temporary help to afford the health insurance and rent. I’ve been seeing a therapist for many years, and it was determined that I definitely have ADHD (along with major depressive disorder). While I’m being treated for the major depressive disorder, I can’t afford the actual testing for ADHD, which also takes time I don’t have to get a proper medication.

My ultimate goal is to find some contracting work for macOS/iOS where I can use my reverse engineering and bug fixing/finding/working around skills. I miss figuring out how things work, something I could do in spades while working on macOS Notarization at Apple.

I also miss when people could launch a potato salad fundraiser on Kickstarter or another fundraising site. Since some people have asked, I do have the Apple Cash at my first name (see above) at the computer Apple makes dot com. And PayPal at the same address, but I think it’s misconfigured—it says you’re supposed to receive a product if anything is sent there, even though there are no products. Also, I have that Cash app thing, $<my first name>, I think? As for food, I have an Amazon Wishlist, but I don’t know how useful it is or what can be put on there.

I can help you solve problems with other people’s code. My résumé is available, if that’s worth something to you, please contact me.