Love Tropicana: The Fix for securityd Eating Gobs of RAM When Updating Keychain Entries
/A few nights ago I was updating some not-to-be-named software on my laptop. This piece of software had a few passwords stored in the Keychain. Since said application was recently updated and therefore the code was modified, the system asked me if I wanted to give access to the keychain to this updated application. The dialog that it shown to the user is shown below:
Bad things happened when I clicked the "Change All" button to once again allow this updated application to access all the passwords it was allowed to access. Specifically, the securityd process was using 1.3-1.7GBs of ram (the rprvt value is all that matters). This was really, really bad as it caused my machine to page-out and page-in like crazy. Due to the high memory usage, it also caused my boot volume to run out of space because of all of the swap files in /var/vm/. My point is that very, very, very bad things happened. After I cleared a lot of unused crap (Garage Band loops and old iDVD themes) off my boot volume, I rebooted. I then tried launching the updated application again. I got the same dialog and the same problem. However, since I now had enough hard drive space available, I just waited for about 10 minutes. The passwords were accessed successfully. I then relaunched the application and securityd crashed. Lovely. Rebooting just repeated the cycle. Also lovely.
The securityd daemon handles all authorization and marshals keychain access. If securityd is not running, you cannot authorize any application (although sudo still functions) and all calls from applications to the keychain either immediately return an error without asking for a password or permission or they just freeze indefinitely. This is also very, very bad. It amounts to a denial of service since you're unable to do much with your computer.
So I clearly had a problem. I remember seeing this problem before while browsing the internets. I specifically found it on 43 Folders. I started searching The Google for any solutions. The searches brought up some really amusing "solutions" such as a thread from Mac OS X Hints. People are talking about some crazy voodoo solutions there. Examples include updating the firmware or adding a "keychain" extension to the user's keychain. Some voodoo causes are also mentioned. One such example is that it is cause by syncing keychains between an ICBM and a PowerPC-based Mac.
The Troubleshooting
Ok, so obviously there were no known solutions to the problem that I could find. This kind of pissed me off. Especially since I had to wake up at 5am to make it to a Microsoft Vista launch event. It was 3am at the time. Obviously, there wasn't going to be any sleep for me.
I decided to do my own troubleshooting. The crash of securityd clearly pointed to a corrupted file of some sort. So I had to determine which file was hosed like a bear coming into contact with the strikingly handsome Stephen Colbert. First, I restored all my keychains from known-good backups that another Mac was actively using. I rebooted and tried to reproduce the problem. Bloody 'ell! The problem remained. So it obviously wasn't that the keychains were corrupted. I rebooted again, started up the terminal, ran sudo fs_usage -f filesys securityd to see what files securityd was accessing when it was trying to update the keychain reference for the application. /var/db/SystemKey repeatedly showed up. However, I was very, very tired and getting very, very angry so I decided to go with a machete approach to fix the problem. I ranstrings on securityd and it turned up four "interesting" files and folders:
/var/db/SystemKey
/var/db/TokenCache (this is a folder).
/var/db/SystemEntropyCache
/var/db/CodeEquivalenceDatabase
Since I was so angry, I just "moved aside" three of these. "Moved aside" meaning that I just renamed everyone of them to have a ".old" extension using the Terminal (via mv). The TokenCache folder did not exist on my ICBM so I couldn't move it aside. I rebooted.The problem was fixed!. I then went to the Microsoft event. I emailed the very sexy Merlin Mann with the possible shoot-everything fix. He reported back that it fixed the problem for him, at least. Hallelujah!
The Actual Fix
I was pissed off at the fact I wasn't able to narrow down the problem to one file. So after a very good 14 hour sleep (after being awake for over 30 hours), I set about to confirm the issue. I started by examining the files and comparing them to the files on my PowerPC Mac. The TokenCache folder didn't exist on the ICBM and the SystemKey didn't exist on the PowerPC Mac. So I could rule those two out. I examined theSystemEntrophyCache file and its contents were boring. So I was able to rule that out. I then opened the CodeEquivalenceDatabase in TextMate (an awesomely awesome overnight text editor) and the text strings matched the applications that currently have Keychain access. So I put the "moved aside" CodeEquivalenceDatabase file back to its original location and rebooted. I then opened Transmit (an awesomely awesome FTP application) which I had just updated and the problem reoccurred. YAY! We (the royal We) had just narrowed down the exact corrupted file that was creating the problem.
In order to fix this problem if you are having it, just open the Terminal (/Applications/Utilities/Terminal) and type:
sudo mv /var/db/CodeEquivalenceDatabase /var/db/CodeEquivalenceDatabase.old
or
open /var/db (and then manually move CodeEquivalenceDatabase to the trash, if you can).
Upon rebooting, God should be in His Heaven and all should be well with the keychain.
Note 1: If CodeEquivalenceDatabase is corrupted, then updating Mac OS X will also cause securityd to eat gobs of memory near the very end of the update cycle when the installer updates Apple applications. This may make it seem like the update stalled.
Note 2: The /var/db folder is interesting. It's not required to boot Mac OS X and all the files in there are created as needed by Mac OS X. It's also explicitly ignored by repairing permissions. However, it does hold important account information, so I would not delete the folder or the items inside unless you are trying to revert your Mac to "factory fresh" and do not wish to have any of the same accounts available without recreating them.