Love Tropicana: The Fix for securityd Eating Gobs of RAM When Updating Keychain Entries

A few nights ago I was updating some not-to-be-named software on my laptop. This piece of software had a few passwords stored in the Keychain. Since said application was recently updated and therefore the code was modified, the system asked me if I wanted to give access to the keychain to this updated application. The dialog that it shown to the user is shown below:

sexy_keychain.png


Bad things happened when I clicked the "Change All" button to once again allow this updated application to access all the passwords it was allowed to access. Specifically, the securityd process was using 1.3-1.7GBs of ram (the rprvt value is all that matters). This was really, really bad as it caused my machine to page-out and page-in like crazy. Due to the high memory usage, it also caused my boot volume to run out of space because of all of the swap files in /var/vm/. My point is that very, very, very bad things happened. After I cleared a lot of unused crap (Garage Band loops and old iDVD themes) off my boot volume, I rebooted. I then tried launching the updated application again. I got the same dialog and the same problem. However, since I now had enough hard drive space available, I just waited for about 10 minutes. The passwords were accessed successfully. I then relaunched the application and securityd crashed. Lovely. Rebooting just repeated the cycle. Also lovely.

The securityd daemon handles all authorization and marshals keychain access. If securityd  is not running, you cannot authorize any application (although sudo still functions) and all calls from applications to the keychain either immediately return an error without asking for a password or permission or they just freeze indefinitely. This is also very, very bad. It amounts to a denial of service since you're unable to do much with your computer.

So I clearly had a problem. I remember seeing this problem before while browsing the internets. I specifically found it on 43 Folders. I started searching The Google for any solutions. The searches brought up some really amusing "solutions" such as a thread from Mac OS X Hints. People are talking about some crazy voodoo solutions there. Examples include updating the firmware or adding a "keychain" extension to the user's keychain. Some voodoo causes are also mentioned. One such example is that it is cause by syncing keychains between an ICBM and a PowerPC-based Mac.

The Troubleshooting

Ok, so obviously there were no known solutions to the problem that I could find. This kind of pissed me off. Especially since I had to wake up at 5am to make it to a Microsoft Vista launch event. It was 3am at the time. Obviously, there wasn't going to be any sleep for me.

I decided to do my own troubleshooting. The crash of securityd clearly pointed to a corrupted file of some sort. So I had to determine which file was hosed like a bear coming into contact with the strikingly handsome Stephen Colbert. First, I restored all my keychains from known-good backups that another Mac was actively using. I rebooted and tried to reproduce the problem. Bloody 'ell! The problem remained. So it obviously wasn't that the keychains were corrupted. I rebooted again, started up the terminal, ran sudo fs_usage -f filesys securityd to see what files securityd was accessing when it was trying to update the keychain reference for the application. /var/db/SystemKey repeatedly showed up. However, I was very, very tired and getting very, very angry so I decided to go with a machete approach to fix the problem. I ranstrings on securityd and it turned up four "interesting" files and folders:

/var/db/SystemKey
/var/db/TokenCache 
(this is a folder).
/var/db/SystemEntropyCache
/var/db/CodeEquivalenceDatabase

Since I was so angry, I just "moved aside" three of these. "Moved aside" meaning that I just renamed everyone of them to have a ".old" extension using the Terminal (via mv). The TokenCache folder did not exist on my ICBM so I couldn't move it aside. I rebooted.The problem was fixed!. I then went to the Microsoft event. I emailed the very sexy Merlin Mann with the possible shoot-everything fix. He reported back that it fixed the problem for him, at least. Hallelujah!

The Actual Fix

I was pissed off at the fact I wasn't able to narrow down the problem to one file. So after a very good 14 hour sleep (after being awake for over 30 hours), I set about to confirm the issue. I started by examining the files and comparing them to the files on my PowerPC Mac. The TokenCache folder didn't exist on the ICBM and the SystemKey didn't exist on the PowerPC Mac. So I could rule those two out. I examined theSystemEntrophyCache file and its contents were boring. So I was able to rule that out. I then opened the CodeEquivalenceDatabase in TextMate (an awesomely awesome overnight text editor) and the text strings matched the applications that currently have Keychain access. So I put the "moved aside" CodeEquivalenceDatabase file back to its original location and rebooted. I then opened Transmit (an awesomely awesome FTP application) which I had just updated and the problem reoccurred. YAY! We (the royal We) had just narrowed down the exact corrupted file that was creating the problem.

In order to fix this problem if you are having it, just open the Terminal (/Applications/Utilities/Terminal) and type:

sudo mv /var/db/CodeEquivalenceDatabase /var/db/CodeEquivalenceDatabase.old

or

open /var/db (and then manually move CodeEquivalenceDatabase to the trash, if you can).

Upon rebooting, God should be in His Heaven and all should be well with the keychain.

Note 1: If CodeEquivalenceDatabase is corrupted, then updating Mac OS X will also cause securityd to eat gobs of memory near the very end of the update cycle when the installer updates Apple applications. This may make it seem like the update stalled.

Note 2: The /var/db folder is interesting. It's not required to boot Mac OS X and all the files in there are created as needed by Mac OS X. It's also explicitly ignored by repairing permissions. However, it does hold important account information, so I would not delete the folder or the items inside unless you are trying to revert your Mac to "factory fresh" and do not wish to have any of the same accounts available without recreating them.


Exercises in Futility: Repairing Permissions is Useless

 Update: I've reposted this blog entry here on Squarespace as a test. Some links do not work any more and some information is outdated, but the facts remain facts.

Update 2015-06-16: Common sense has won! Mac OS X 10.11 "El Jefe" removes the Repair Permissions functionality! And it now automatically checks for proper permissions during a Mac OS X Update or Security Update! Quoth the Raven: «System file permissions are automatically protected, and updated during Software Updates. The Repair Permissions function is no longer necessary.»

DEAD DEAD DEAD!

Update 2015-06-16: I've updated this article with some other relevant changes that have been made since it was originally written. Including information on Mac OS X 10.11's new System Integrity Protection (rootless) features. Search for "2015-06" to locate the changes quickly.

This is a rant I've wanted to write for an extremely long time. However, I prefer to let my anger/annoyance with some topics sit in the stew that is my soul and slowly boil until it is like you dropped a tea bag into a cup of very hot water housed in a smooth glass container that you just stuck in the microwave for 10 minutes. Yes, doing that will cause the water to spontaneously explode, leaving horrible burn marks all over your face (just be glad it wasn't maple syrup or something else that could stick to human flesh).

Update: Despite what Apple's Knowledge Base Article says, Repair Permissions does not repair permissions on any third party software (or any Apple software outside of the Base System). I just checked this by changing the permissions on FontAgent Pro and then repairing permissions. The permissions were not set to their correct values. This makes repairing permissions even more useless as it can't be used for any non-apple software. It also explains why installing CHUD or iPhoto from iLife '05 would cause the incorrect warnings to appear.

Now that that pleasantness is over with, the real issue I have is all these websites that suggest repairing permissions will actually fix/prevent problems. Even worse is when otherwise intelligent people are poised with a Mac OS X related troubleshooting problem and immediately suggest the user repair permissions. Repairing permissions won't fix your problem. As Jason Harris said (and I am paraphrasing), "Repairing permissions is zapping the PRAM for the twenty-first century". I couldn't agree more. Both are equally futile attempts to fix a completely unrelated problem. In other words, 99% of the time, neither will fix nor prevent any problem. Especially not the problems they are recommended for. Covering yourself in vaseline and rolling around naked in the dirt and repairing permissions are just as likely to fix your Mac OS X problem. People swear by repairing permissions as often as they save files and present as proof the fact they don't have any problems. That's rather specious reasoning. As with everything in life, The Simpsons has covered this topic well. Yes, I've shamelessly copied this text verbatim.

Homer: Not a bear in sight. The Bear Patrol must be working like a charm.

Lisa:That's specious reasoning, Dad.

Homer: Thank you, dear.

Lisa: By your logic I could claim that this rock keeps tigers away.

Homer: Oh, how does it work?

Lisa: It doesn't work.

Homer: Uh-huh.

Lisa: It's just a stupid rock.

Homer: Uh-huh.

Lisa: But I don't see any tigers around, do you?

[Homer thinks of this, then pulls out some money]

Homer: Lisa, I want to buy your rock.

[Lisa refuses at first, then takes the exchange]

Please ignore the fact that they're talking about tigers and Mac OS X version 10.4 just happens to be called "Tiger". The point stands. Just because something isn't around because you do something every day does not mean it would suddenly come around when you stop doing the something every day. Some people used to smear chicken blood all over their kin to keep the devils away too.

Call Me Lucy

Update 2015-06-16: The location of the Receipts folder moved in Mac OS X 10.5 to /Library/Receipts/db/. The Receipts folder moved again in Mac OS X 10.6 to /var/db/receipts/ and the BOM files are no longer stored inside packages (.pkg). The BOM files are now saved in a flat manner directly in the root of /var/db/receipts/. The information and commands listed below should be adjusted for these changes.

I guess I should explain what repairing permissions is and isn't. Repairing permissions goes through all the Package files (.pkg) in /Library/Receipts/. A receipt package is created when (and only when) you install something using Installer.app (Apple's installer). First it creates a temporary package based on all the files in the package. Then when you install, it creates the actual package that actually contains a listing of all the files you installed from that package. You can see these two steps if you open Installer.app's Log window and choose "Show everything". The package lists the paths for all the files along with the permissions Installer.app set for them when they were installed (those permissions are part of the actual package in which the files were installed from). The items in /Library/Receipts/ are basically just the shell package without any of the actual files inside. You can see the contents of these by using the lsbom utility. The usage is basically lsbom path/to/archive.bom. Like so:

lsbom /Library/Receipts/MacOSX10.4.pkg/Contents/Archive.bom

This will list all the files that were installed by the package (by absolute path, usually), their installed permissions, file size, and some other information. See the man page for lsbom for more information on the output.

Anywho, when repairing permissions, the disk utility goes through the permissions of all the files in the target volume's /Library/Receipts/ folder. Apple has a kbase article on this as well. In order for the "repair permissions" or "verify permissions" button to show up, the target volume must have a version of Mac OS X installed on it. Repairing permissions only works on volumes that have a /Library/Receipts/ folder. Which is only there if OS X is installed on that volume.

Based on this information (and the sheer stupidity of Installer.app) you can correctly assume that Repair Permissions won't touch any files in any of the user's home folders since Installer.app can't target user folders specifically, only any folder or a specific path, and there are no packages in ~/Library/Receipts/. The only way it'd ever touch any files in a user's folder is if you installed something that let you explicity select a folder to install in (there are very few of those, none are available from Apple publically) and you chose a folder inside your user's folder. The receipt would still be installed in /Library/Receipts/ and it would only affect the user that installed it. It also won't fix permissions for any files that were created during the normal (or abnormal) use of OS X. This means it won't touch any cache files, database files, swap files, or settings files not created by the installer. If a file isn't listed in a receipt, it doesn't exist to the repair permissions process. It's really as simple as that. And because it reads from /Library/Receipts/ on the target disk, you can boot from a Mac OS X 10.2 CD and still use it to (correctly) repair permissions on a volume with Mac OS X 10.4.6 installed on it and it will set the permissions to the ones that 10.4.6 requires. There is no need for you to boot off a volume in order to fix its permissions.

A Bit of History

Does anyone actually remember when Apple first offered the ability to repair permissions and why it was needed? I do. Apple introduced the Repair Privileges Utility as a download for machines running Mac OS X 10.1.5. This was back in the long ago time when Macs would still boot Mac OS 9 and the default environment for a lot of Mac users was still Mac OS 9. Mac OS 9 didn't care about permissions at all. Mac OS 9 was Mac OS X's worst enemy in this area. If you booted into Mac OS 9 and ran some common applications, compressed and decompressed files, moved or renamed files, or (worse) ran a disk utility like Norton, they could completely destroy the permissions for many files that OS X needed to boot or run correctly. Since this was a relatively common occurrence and ahuge support issue, Apple introduced the Repair Privileges Utility. When Mac OS X version 10.2 "Jaguar" came around, Apple rolled it into Disk Utility where it belonged. But by this time it wasn't needed nearly as much since many new Macs couldn't even boot Mac OS 9 thus rendering the fear of bad disk utilities that didn't pay attention to the rules set out by the HFS+ Technote ruining the ability to run OS X completely moot. Just because HFS+ didn't use the features when you wrote the disk utility doesn't mean it won't in the future (and the future is now, excluding named forks). None of this applies to using Classic as Classic lives in a mostly happy sandbox with just a trace amount of urine (which is more than I can say for my spacesuit). 

The other number one cause of permissions going wonky were 3rd party installers that asked for root on OS X and changed permissions on some folders that were in the path to the destination. I know that MindVision and Allume (Courtney Cox-Arquette) have long since updated their installers to prevent this kind of weirdness (these would be the same installers that told you to quit all applications when installing software on OS X). I know there is always a chance I could be wrong about things like this so I've been running Repair Permissions after every install using these two installers just to make sure, and I haven't seen it note anything at all. So these two companies' installers are good. So this cause is also deemed completely moot.

The final minor cause for incorrect permissions is, basically, the user. I've said it before, and I'll say it again, you can't account for stupidity. People that change permissions on something because they think it might fix a problem they're having because they know more than the system but it very obviously won't fix the problem. More on this reason later.

Update 2015-06-16: Mac OS X 10.11 "El Jefe" apparently can account for stupidity with a feature called System Integrity Protection (also known as "rootless"). With SIP enabled, even root (the superuser) cannot change the permissions or the content of required system files. This means that it is no longer possible to enter into a situation where the permissions on key system files have the wrong permissions. While SIP is enabled by default, an extremely determined idiot can boot into Recovery Mode to disable SIP. However, disabling SIP removes a lot of security protections in addition to permitting wonton permission changes when root. For example, it becomes possible for malware to hijack system processes with task_for_pid() (in layman's terms, it's a function used at the beginning of a list of steps needed to allow one running application to insert its code into another running application).

 

The Ugly

Permissions won't magically go bad. They won't break suddenly. They don't suffer from bit rot. In order for permissions to change something must change them. Even with Mac OS 9, as long as you didn't modify the files, their permissions wouldn't change. For this reason, repairing permissions as a maintenance task is just a complete waste of time. Granted, it won't harm anything*, but it won't help anything either. If you want to be paranoid about permissions then at least use some common sense when doing it and only run repair permissions after using an installer. Of course, I am not recommending repairing permissions after running an installer!

*Correction: Certain Mac OS X versions had a security bug that caused Repair Permissions to actually revert some security improvements made in Security Updates to there original, insecure state.  

I've only ever found two cases in which repairing permissions would actually help fix something. One is when backing up to a non-local (external) volume (like a FireWire HD, a USB 2.0 HD, or a disk from FireWire Target Disk Mode) via ditto (with the --rsrcForkoption passed) or Carbon Copy Cloner. You really shouldn't use drag and drop in the Finder or cp to backup OS X volumes as important metadata can get lost that way. There is one reason and one reason only I say that the use of ditto or CCC would require a repair permissions on the volume everything is being backup up to--when the volume has the "Ignore ownership on this volume" checkbox checked. Ignore Ownership on this volume should be unchecked. If it isn't, then none of the permissions will be copied over in the first place and then you have to run repair permissions to make the thing bootable. But I still don't recommend it as any files not installed by the installer won't have the permissions fixed and could be a huge security risk (especially if it is world writable file). So make sure that box is unchecked. It's checked by default on non-local volumes. Also note that if you don't uncheck that box, you will have to reinstall Application Enhancer as it is permission sensitive and if that box is checked while you're doing the copy/backup it won't have the proper permissions to run. This has come up a lot in support emails but by then it is too late to tell the user to uncheck the box. This item is preventable.

Update 2015-06-16: Sometime between when this article was originally written (when Mac OS X 10.4.6 was released) and when Mac OS X 10.9 "Mavericks" was released, the importance of correct permissions on files in /var/db/ increased significantly and Mac OS X will actually refuse to boot if any of the permissions in /var/db/ are incorrect. As /var/db/ is explicitly ignored by the Repair Permissions process, it has become absolutely impossible to resurrect an install of Mac OS X  that was damaged by forgetting to uncheck "Ignore ownership on this volume" in the above manner.

The other case when repair permissions are required is also in the user error category. But this is when installing system services using very outdated instructions that tell you to assign the wrong value to a now default OS X user. I can't blame them because I thought that I knew what I was doing.In this case repairing permissions was the fix (as long as I made the postfix uid 27). I could have easily fixed them manually, but that was a few more keystrokes and I was feeling keystroke lazy. This item is preventable but hasn't applied in a very long time. It has also become moot as Mac OS X started including postfix by default and can be controlled easily by installing the $20 Mac OS X Server application.

Anecdotes are not Proof of an Antidote

Now let's see what some very silly people are saying about repairing permissions and why it is flat out wrong.

7) Repair Permissions

8) Install Mac OS X 10.x.x update

9) Repair Permissions

Ugh. This is the one that annoys me the most and the reason why I wanted to write this rant. When 10.3.9 came out, I saw this all over on just about every single Mac related website. This really boils my blood. First we have the completely bizarre suggesting of repairing permissions before installing an update. Useless. When you install a system update (or pretty much any updater using Installer.app) you are asked for your password. This makes the installer process run as root. Wrong permissions, bad permissions, no permissions, it doesn't matter. root is god. It doesn't care about what some small little file has as its permissions. It will just ignore them completely. *chortles demonically* You can't stop root. Repairing permissions won't increase the chances of the install succeeding (nor will it decrease the changes). Point nine is equally as baffling. As the installer is installing/updating files it also reset the permissions to those that will be in the receipt that the repair permissions process reads from. You just installed these files, they are going to have the correct permissions.

I was having problems with permissions with files on the scratch disk so I repaired permissions on it and it fixed everything.

This is a paraphrase of something someone actually said to me. It is real. It is a lie. I can say that without even knowing how this person handles problems (even though I do know how he handles OS X problems). Why? He says "scratch disk". For those that have never worked with media or photoshop, scratch disks are dedicated volumes (usually separate physical disks, in this example it was another partition) that is used to store all the render files and temporary media files for a project. They are used often to increase render speed when applying compute heavy effects to files. They are separate disks so they can be read from/written to quickly without having to compete for I/O time with the boot disk. It's also good to keep them separate so if Project A needs scratch disk space, it can easily delete all the scratch space for Project B while making sure that Project B doesn't suffer by having to reimport media. Although if you do delete Project B's scratch files, you'll likely be stabbed anyways as it can take hours to rerender complicated projects (but on the upside, the people working on Project B don't really have to do any work to rerender).

Since, by definition, a scratch disk does not have an OS installed on it, there is no way he could have repaired permissions. No OS, no /Library/Receipts/ which means no repair permissions. The actual fix to his problem? Get info on the scratch disk and check"Ignore Ownership on this volume". The problem appears to be that multiple users are using that scratch disk with some of them working on the same projects. If a user named "rosyna" creates a file, its default permissions will be 0644 with owner rosyna and grouprosyna, This means if user "slava" comes on and tries to write to that file, he can't since he is neither me nor a member of me. Ignoring ownership would mean that everyone can read and write whatever file they want on that volume.

iTunes kept throwing up an error saying "Cannot save library. -54". So I opened a get info window on the iTunes application and gave everyone the ability to read and write to it along with all enclosing items and the error when away. But every time I repair permissions, it keeps stupidly resetting the permissions back and the error comes back.

Same person as above. A lot more paraphrasing was done. The error is the same (Library, can't save, and -54). His actions on the iTunes application bundle are the same, and he did indeed say it fixed the problem. Not only is it a huge security risk to give everyone write access to an application, it is a lie. Yes, a -54 error is a permissions error (on file open). But note that the error said "Can't save Library". Library refers to the iTunes library. This file is located at ~/Music/iTunes/iTunes 4 Music Library. Note that it is stored in the user's home folder (that's what the ~ means). This means two things. 1) Repairing permissions won't fix the problem. 2) His problem has nothing to do whatsoever with the iTunes application. Not that it matters since an application on OS X must not write to itself. This was a huge source of problems in Mac OS 9 so resource forks that are opened automatically on application launch are opened read only on Mac OS X.

This fix for this (as I know it, I've only had the problem once) is sadly rather invasive. Delete the iTunes 4 Music Library. Reopen iTunes, choose "Import" from the "File" menu and import ~/Music/iTunes/iTunes 4 Music Library.xml. It will take a while and you will lose some data. But it will import all the files that it can find on your computer. It sadly won't re-add files that it couldn't resolve at the time iTunes last made the XML copy of your library which should have been the last time you quit iTunes successfully, the last time a change was made to the iTunes library, or the last time someone connected to your iTunes share. Your playlists will be retained although you might get duplicate copies of your smart playlists (I did of a few) and the order of some of your larger playlists might be completely screwed up.

Normally I don't say something is a lie (or call someone a liar) flat out in public (or even privately) and this wasn't a lie done for malicious purposes. It was more of a lie in the sense that this person isn't quite sure what actually was happening and likes to argue endlessly and when presented with explanations for what actually happened tries to make it seem as if the people giving the explanation are wrong (inherently and always). He changes the past ever so slightly to suit him best hoping that no one will catch on. His memory isn't that good either.

I was having this huge problem with some huge third party application and repairing permissions fixed it.

Extremely doubtful. Most of the time huge third party application is something by Microsoft, Adobe, or Macromedia. None of which use Installer.app. Which means that repairing permissions couldn't have possibly fixed any issue that only one of these applications was experiencing. For example if the permissions for /Library/ were set to 000 then all hell breaks loose. And I'm not kidding about that. Disk Utility stopped working, ShapeShifter stopped theming, and Adobe Photoshop CS 2 (9.0) showed me this on launch, without holding down any keys as startup:

photoshop9.png

How to Solve Problems and Stop Repairing Permissions

It'd really help a lot if people would stop trying to assume repairing permissions does good and actually try to find out what's causing a problem they're having. For the people willing to try, I point to the the Console (/Applications/Utilities/Console). The console has the answers you seek, young one. For just about any problem with anything, the Console log will list it for the above Library example, Photshop actually told me that it couldn't read files from the Application Support folder (which is correct). But make sure that what you're looking at actually relates to your problem. If Virex keeps crashing and you see:

May 15 15:35:09 Stability-64 kernel[0]: Limiting icmp unreach response from 314 to 250 packets per second

In the console, do not assume it has anything to do with your problem. For my postfix example above, the console clearly has the following to say:

May 16 12:39:00 Stability-64 postfix/master[487]: warning: master_wakeup_timer_event: service public/qmgr: Permission denied

May 16 12:39:05 Stability-64 postfix/master[487]: warning: master_wakeup_timer_event: service public/pickup: Permission denied

Notice how it says "Permission denied"? That means it doesn't have the correct permissions and fixing permissions will definitely fix this problem. For what it's worth, I changed all the files in /var/spool/postfix to be owned by rosyna.

In summary, repairing permissions is useless 99.999% of the time. Permissions don't magically go bad. And you should really throw out "repair permissions" as a troubleshooting step until you've exhausted all other methods of troubleshooting. Reparing permissions can only fix problems with Mac OS X system files distributed by Apple installed by Apple's Installer.app and only as long as a valid receipt is available. Even then, it should never be suggested to others as it won't help and sadly will only make you look like you know less about Mac OS X than you really do. Repairing permissions is the strategy of last resort!

Update 2015-06-16: The useless of repairing permissions is now 99.999% and when Mac OS X 10.11 is released with rootless, the uselessness of repairing permissions will become 99.999999%. I hesitate to say 100% because I know at least one user will disable SIP/rootless in El Jefe and manually change the permissions of a file that's supposed to be protected despite all the protections added.

I guess what I'm trying to say with all this is that if you're going to heat up water in the microwave for use in tea, put the teabag (without the metal) in the cup of water before you heat it up and keep it in there (without metal) while heating it up.