For some, there should be new information about how launchd works in general, as I think for many admins its behavior is somewhat opaque. Along the way I also learned some new launchctl command options.

Introducing ‘Helper-Tool’

First, let’s paste the entire Helper-Tool script, and go through it. This is called by the com.oracle.java.Helper-Tool LaunchDaemon, and is triggered whenever the com.oracle.java.Java-Plugin LaunchAgent plist (which is actually a symlink to a plist in the plugin’s Contents/Resources directory) is modified.

According to the comments, as we suspected, the purpose of this is to “Make the appropriate Changes” to the LaunchAgent plist following an update.

Reading some preference values that were set by the installer postinstall script, suppressing error output in case the keys didn’t exist. Ok.

Two problems. One, ‘/usr/libexec/PListbuddy’ is a typo. It just goes unchecked because probably 99.9% of OS X systems are on a case-insensitive HFS+ filesystem, but OS X fully supports installation onto case-sensitive filesystems.

Two, defining commands’ absolute paths using `which command` is useless. The which command works by searching the PATH environment variable for the executable. If you depend on being able to locate an executable by `which` in your script, you can skip pretending you’re using absolute paths, because you’ve already assumed they’ll be located in your PATH. You can get the default PATH used by launchd with the command: launchctl getenv PATH.

If any of the HT variables are undefined (-z tests for a zero-length string), then store a random value in retValue, and use the sed command to perform an inline replace it in the plist.

One might ask, if this system is designed specifically to “reset” the LaunchAgent schedule after an update, why not simply put this logic into a postinstall script instead, and set the schedule to something like once per day?

Using sed to modify a plist is just silly. Plists are structured data, and there are tools, like PlistBuddy used in the following three lines, that were made for exactly this. The 00 minute and 09 hour values correspond to the values that were already in the LaunchAgent plist delivered by the installer payload. This sed command is also already performed by the installer script.

Despite all of these silly workarounds to update a schedule plist, there are also weak assumptions in the if statement:

• If any of the HT* variables stored in com.oracle.java.Helper-Tool by the installer don’t exist, then it must be able to find exactly these stock StartCalendarInterval times in the com.oracle.java.Java-Updater plist, and that these wouldn’t somehow conflict with numerical values elsewhere in the plist (we’ll see later that if one enables debugging for this LaunchAgent, it will).
• Two, PlistBuddy’s ‘Set’ command requires the key to already exist – it does not add a missing key as with the ‘defaults’ command. The logic used by this if statement is vague, because it’s assuming that if any one key in one domain com.oracle.java.Helper-Tool is missing, then we must be able to find all three of some other key in a different plist com.oracle.java.Java-Updater.

What’s worse, is that by littering these various preference domains with various keys that all seem to be related to a schedule but only coupled by flawed scripts, admins that may poke at these values attempting to shortcircuit its update check behaviour may just further confuse the roundabout logic used in these scripts. Of course, vendors don’t design their packages with poking in mind, but given that this concerns behaviour that most system administrators will immediately want to disable, it doesn’t help to make it as difficult as possible to disable something that’s usually trivial with other software, something even Adobe can document and support.

Continuing on…

Why would this be necessary? Permissions should be normally only handled by the installer payload, except in very particular circumstances that can usually be avoided. Maybe it’s here because the permissions are actually wrong (mode 664, when they should be mode 644) in the installer’s payloads for both launchd plists.

launchd and Session Types

This brings us to the topic of launchd and “Session Types”. LaunchDaemons and LaunchAgents can run in several different Session Types – for example, the ‘Aqua’ Session Type is run in the context of a user that is currently logged in at the GUI. You may have noticed before that when manually loading and unloading jobs, that you need to be root in order to manage jobs that are running at the system level, for example LaunchDaemons located in /Library/LaunchDaemons.

‘LoginWindow’ is another Session Type that can be specified if a job should be loaded only while the system is at the login window. The LimitLoadToSessionType key can be specified in a launchd plist to restrict in which Session Types it would normally be loaded, but using the ‘launchctl’ command permits the job to be loaded in other contexts. If LimitLoadToSessionType is omitted, then the default of Aqua is used. So in a normal scenario:

1. Machine boots up, and loads the loginwindow.
2. LaunchAgents that are able to run in the LoginWindow context are loaded.
3. User logs in.
4. Jobs running in the LoginWindow Session Type are unloaded, and the jobs available to run in the Aqua Session Type as the regular user are loaded.

There’s a lot more in-depth and lower-level detail available in Apple’s Daemons and Agents Tech Note. It’s over five years old, however.

In most if not all cases, if the machine was asleep when the StartCalendarInterval time arrived, the job will run immediately upon waking. Even if the LaunchAgent job was somehow being loaded at the loginwindow (remember, this is to launch the Java Updater app), it would simply die and complain that no connection to the window server was possible. Actually, for this package, it won’t, because the job’s StandardErrorPath is set to /dev/null. More on that later.

But since Helper-Tool is running as root, its invocation of launchctl will load the job as root, and now, guess what? We have two separate instances of the LaunchAgent running. What do you think happens when the StartCalendarInterval time arrives?

LSMultipleInstancesProhibited doesn’t prevent it the app from running as multiple users!

So now Java Updater is being run twice, once as you and once as root. The LSMultipleInstancesProhibited would prevent it from launching twice as a user (perhaps to prevent runs over weeks on an idle system from spawning the alert multiple times?), but it won’t help here, when the alert is being launched as different users. Moreover, depending on how long the machine has been running without a reboot, there may be some time during which the job is running with two different times set in StartCalendarInterval.

Because Java Updater uses Sparkle, selecting “Skip this version” will set the SUSkippedVersion key in the app domain being used, which in this case is com.oracle.java.JavaAppletPlugin. The version is as it is defined in the Sparkle XML feed (which may or may not be equal to a bundle version key). Because it’s running as two different users, these Sparkle-related preference keys are now defined in two different user homes. In other words, skipping a version as a normal user means that it will still run again as root, until it’s skipped when it launches as root. (For what it’s worth, these Sparkle preferences can be defined at the system level, but managing SUSkippedVersion keys for this application gets to be a very tedious game of catch-up, and is not at all how the key is intended to be used).

Just to be sure that this is really happening, here’s the output of execsnoop, a DTrace utility that logs new processes as they occur. Notice the first execution is UID 0 (root) with a PPID (parent process ID) of 1, and the second is 501 (me) and a PPID of 329. The PPIDs correspond to the launchd manager process for the System and my user’s bootstrap namespace. (You can check these yourself with the launchctl managerpid command.)

This relaunching-as-root issue is moot once the Mac reboots, of course, because then the LaunchAgent will load only in the user’s Aqua context as usual. But with laptops, it’s not uncommon to go for weeks without a reboot, which is about how frequently there have been recent security updates are being released.

No logging

I mentioned earlier that the LaunchAgent’s StandardOutPath and StandardErrorPath are both set to /dev/null. You’re free to run the Java Updater binary yourself to mimic what would happen at the time the LaunchAgent job would run, but there’s not much useful output. There’s also a debug flag you can set in your shell environment if you’d like to see a bit more: set the JPI_PLUGIN2_DEBUG flag to something (it can be anything, it just must be set): export JPI_PLUGIN2_DEBUG=1. You’ll then see some output like this if you run it manually:

So if we’d like to actually debug and log the behavior of the LaunchAgent itself, we can remove the StandardOutPath and StandardErrorPath keys (they default to the system log) and define our own environment variables in the job by setting the EnvironmentVariables key like so:

Of course, as soon as you modify this LaunchAgent to help you debug this, the Helper-Tool job helpfully runs and resets your modified StartCalendarInterval values and mangles your debug flag, because it just so happens to be looking for the string “2″ anywhere in the plist and sets it to a random day-of-week integer. When I was originally debugging this, I commented out enough of the Helper-Tool script to prevent it from resetting my changes to the plist. I’d then unload and load the LaunchDaemon.

Diagnostic self-obfuscation

While Oracle’s JRE package was clearly not meant to be consumed and scrutized in this manner by any user (or sane person), one has to seriously wonder why someone thought it helpful to go to such lengths to obfuscate the system’s own mechanisms, hiding all traces of useful logging and status info; compare to the verbose output of Google’s Keystone daemon during a background Chrome update. Setting aside the bizarre self-healing schedule – for something that probably should be nagging the user once a day to update, since Apple will block it the next anyway – it’s amazing how difficult the package even makes it to test and debug its behavior. It seems that the release engineer on this project was not interested in being able to debug and test this easily himself.

What’s useful about this file is that it describes in a human-readable way what Xcode uses to determine what component updates are available and what’s already installed. Up until yesterday, the CLI tools used only SHA-1 sums on specific binaries and libraries to determine whether the package was installed, which was somewhat frustrating to those of us deploying it, because it meant the actual package receipt version numbers were next to useless. Munki, for example, couldn’t use these to determine installed status, but one could at least use these to know what files to use to track the installation. Munki can use MD5 checksums to specify a file’s contents.

Here’s how they used to check their installed state:

And here’s the full entry for the latest version (4.5.9) of the CLI tools for Mountain Lion, released yesterday on March 14, 2013:

Notice we have a download path, it doesn’t require authentication to the Apple Developer Center (not long ago they did), and note in particular the InstalledIfAllReceiptsArePresentOrNewer key, with a dictionary of package receipts and versions. If I generate a new Munki pkginfo for this installer, my receipts array looks like this:

…which is much nicer than managing a bunch of file checksums. Hopefully they start using only receipts from now on.

And of course, the first thing administrators need to verify is that deployed software won’t periodically nag the user to install an update that they don’t have sufficient rights to install, or that they shouldn’t install for other reasons. I’ll cover a few ideas in this post specifically about the updater mechanisms and approaches to disabling it, and focus on other specific issues with this package in future posts.

deployment.properties a.k.a. com.oracle.javadeployment a.k.a. com.oracle.java.JavaAppletPlugin a.k.a. com.oracle.java.Java-Updater a.k.a. com.oracle.java.Helper-Tool

There is a place to disable update checks in the Java Control Panel:

Look! It unchecked the checkbox!

Java uses what’s known as a “Java properties” format to store preferences, that looks similar to a .ini file. Many user preferences seem to touch a file at /Library/Application Support/Oracle/Java/Deployment/deployment.properties. This file’s options are somewhat documented.

In more recent versions, some settings (perhaps those known to have OS X-specific implementations, like the boolean deployment.macosx.check.update) seem to be stored instead in the defaults domain com.oracle.javadeployment in a Java-style namespace:

<key>/com/oracle/javadeployment/</key>

<dict>

    <key>deployment.macosx.check.update</key>

    <string>false</string>

    <key>deployment.modified.timestamp</key>

    <string>1361734949180</string>

    <key>deployment.version</key>

    <string>7.0</string>

</dict>

It doesn’t actually matter for now where the deployment.macosx.check.update setting is actually stored, because it has no effect on whether or not Java 7 will check for updates.

Update 13/03/15: It turns out that this does have an effect, it controls the plugin’s own built-in update checker, that I managed to never see while scrutinizing the Sparkle-based updater. So while it suppresses the nag when the plugin is actually invoked, it doesn’t control the background update check mechanism that’s detailed below and in a related post.

What? Oh.

There’ve been a few discussion threads about configuring the update setting, but all these attempts seem to do is tell the Control Panel about the state of the checkbox, and not suppress the active checking (and prompting) for updates.

The Java-Updater/Helper-Tool Yin-Yang of Doom

Briefly, here are the basic mechanisms of the update-checking system as it is now (here’s the launchd manpage for reference):

• There is a LaunchDaemon (com.oracle.java.Helper-Tool.plist) and a LaunchAgent (com.oracle.java.Java-Updater.plist) installed in the usual place within /Library.
• They are actually symlinks to the web plugin’s installation area, in /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Resources.
• The LaunchDaemon is not run on a schedule, but rather triggered by changes to the LaunchAgent plist file using a WatchPaths entry.
• The LaunchDaemon runs Helper-Tool, a Bash script whose purpose is to enforce (based on default times set by an install script) or randomize the StartCalendarInterval key in the LaunchAgent plist, and reload the LaunchAgent. It modifies the file that triggers itself to run… to modify the file… that triggers it to run… you can probably see why this may not be a great idea.
• The LaunchAgent runs the Java Updater binary located inside a .app bundle in the aforementioned Resources directory.
• Java Updater.app is essentially a vessel for the Sparkle updater framework, which simply manually calls Sparkle’s “check in background” function. It is a pure Cocoa app. It does not seem to have any link to the Java plugin, settings configurable via the control panel, etc. It is not linked to any Java library.

There are a several major issues with this LaunchAgent/Daemon combo that I’ll cover in a future post.

For now, there are a couple possible workarounds I’d consider that are the least intrusive and most reliable way of suppressing the update check behavior. They each still have issues that are very important to understand if you want to deploy this and not dig yourself out of a hole later. I don’t consider either of them acceptable in the long term, and can’t recommend one over the other – it all depends on your environment.

Neuter Sparkle

One approach is to leverage the configurability of Sparkle. Because Java Updater is directly invoking Sparkle’s check method, overriding check behavior preference keys as outlined here won’t help, as far as I can tell. If you can at least read some Objective-C code you can look at Sparkle’s SUUpdater.m and see for yourself what it does, and make a pretty good guess at what methods Java Updater is calling by inspecting its binary strings (hint: the resetUpdateCycle and checkForUpdatesInBackground methods).

Sparkle supports overriding some preference keys that would typically be defined in a bundle’s Info.plist file, by setting them in the bundle’s user defaults domain instead (at either the user or system level). In this case, we’re looking at the com.oracle.java.JavaAppletPlugin domain, and the SUFeedURL string key. (It might be worth pointing out now that the Java 7 package uses about 4 different preference domains for the plugin and helpers, which doesn’t make figuring these out any easier).

Setting SUFeedURL to an invalid URL like “nil” (or one that doesn’t actually contain a Sparkle appcast feed) will cause Sparkle to fail silently. It’s worth noting, since this is being run as a LaunchAgent, that this doesn’t cause the actual Java Updater app to fail with a non-zero exit code. As far as Java Updater is concerned, it’s done its job and Sparkle just didn’t have anything for it to do.

There are alternate, more desperate Sparkle-related tweaks possible that will yield similar results to the above, which will be in a future post.

Advantages: This preference key can be managed like any other: a defaults command writing the value to /Library/Preferences/com.oracle.java.JavaAppletPlugin, MCX, or a Configuration Profile. The preference can later be removed/changed if desired, and it’s completely independent of the plugin installation.

Disadvantages: This overrides the update URL for anywhere the plugin may want to check for updates, meaning that if one checks the Control Panel manually for an update, it will not be able to check or verify whether it is up to date. It will state that it was last run whenever Java Updater was last run, but that it is “Unable to check for updates”, and to “Please check your internet connection and try again.” You’re essentially removing the ability for the plugin to update itself via its built-in mechanisms, even the user- or support-initiated ones. This puts it roughly on par with the state of deploying Adobe Flash.

Remove (but don’t disable) the LaunchAgent

A second approach is to prevent the LaunchAgent from ever running Java Updater in the first place.

After installing Java, unload the job temporarily, and remove the symlinks so that it’s not loaded again after a restart. You’d run something like this with elevated privileges:

You might think, like I did, that you can just launchctl unload -w the job to set it as permanently disabled. It turns out you absolutely should not do this, due to an oversight in the Java 7 installer’s postinstall script. Having these jobs disabled will cause the script, and thus the entire install, to fail.

Advantages: We don’t need to mess with any configuration of the updater mechanism itself.

Disadvantages: We’re still changing the “expected state” of the Java installation, and the fact that disabling the job permanently (ie. launchctl unload -w) causes future installations to fail doesn’t inspire confidence that any of Oracle’s future pre/postinstall scripts will be robust enough or perform even basic sanity checks. We also need to make sure that we unload the job and remove symlinks after every installation. Unloading is actually optional, since that the next update check will usually be a week from the time of installation, and a reboot will cause the launchd jobs to never be loaded again.

Current status

I don’t think either option is generally viable, especially if you’re in environment with a diverse set of client configurations (laptops, admin users, remote workers and sites, etc.) and need to make as few assumptions as possible about how client machines are used and maintained. If, on the other hand, you maintain control over at least software installations and updates, you might decide one of these two workarounds could work despite its limitations, at least until a better solution is discovered or implemented.

After I complained in ##osx-server on IRC, Michael Lynn found a channel through which to report bugs against the JRE, and Rich Trouton documented the steps and caveats in a blog post. If you support Java on your Macs and this is an issue for you, file a bug!

It’s clear that a management strategy could be very useful in environments where users aren’t admins on their computers and can’t install updates themselves. One such strategy is to simply disable the updater, but the definitions should still be pushed to clients as you roll out new plugin versions, to enforce minimum security requirements as well as be able to protect against known malware.

There was some talk on Twitter, IRC and multiple posts on Greg Neagle’s blog. I dug around the XprotectUpdater binary and posted some ideas on how one could monitor this feed for changes.

I later realized that there are also multiple definition files: one for each major version of OS X, starting with Snow Leopard, when XProtect was first implemented. This means there will soon be four separate defintions to keep track of (at least until Apple stops providing security updates for Snow Leopard).

I wrote a basic utility to automate packaging up these changes as the definition file (‘clientConfiguration.plist’) is updated from Apple. Because what XprotectUpdater does to synthesize the two Xprotect definition files on the client is very simple, this tool can do the same thing, but for all available client versions. Meaning, you can run the command on a single machine (and on a schedule, if you wish) and automatically build new packages for all OS X client versions you support.

It’s called XProtect Packager, available here:

https://github.com/timsutton/XProtectPackager

It’s also able to automatically push these updated packages to a Munki repository.

On the subject of Munki, there are a few different “auto-packager” tools that have been made available by different people. Because I wanted this tool to be self-contained and finished as quickly as possible, the mechanisms for building the package and importing into Munki are quite basic, and the tool’s functionality would probably be better served by integrating into a tool like Per Olofsson’s AutoPkg or another recipe-based checking/building/importing tool for Munki integration. If one tool starts to see some more widespread adoption and community contribution, it would be great to integrate XProtect Packager’s functionality into it.

Maybe you’d like to at the very least know when this has been updated, and what are the nature of the changes. Here’s another example where the strings command proves useful, and it’s quickly obvious what’s going on.

If we look in /System/Library/LaunchDaemons for something related to XProtect, we find com.apple.xprotectupdater.plist. Opening it, we see it simply runs the executable at /usr/libexec/XProtectUpdater every 86400 seconds (or 24 hours).

Now, run the strings command on this binary, and see a few telltale methods and values (this is taken from somewhere in the middle):

Taking note of the URL at the top, we can simply curl this URL to see that it’s nothing more than a plaintext XML plist prepended with a security signature (the same signature that’s getting written to XProtect.meta.plist. Looking at the other strings, it’s clear that it’s using the Last-Modified HTTP header to compare this date with the current system date (“Last-modified date is later in time than current date”). It’s also using the Version key in the plist to determine whether the plist available from Apple is more recent than the one already installed.

Note, the “2″ in the URL ..xprotect/2/clientConfiguration.plist is what was returned on my Lion machine. Snow Leopard clients look for “1″, Mountain Lion clients look for “3″, and so on.

Now we can throw this plist URL into a site like ChangeDetection.com, and ask it to check this URL once per day and send us an e-mail if it’s changed. We can see the details on the change, and when it was last modified. Update: Given that Apple’s updated this list within hours of new Flash/Java releases and will probably continue to do so, checking daily is probably not frequent enough.

We could also use an application like Jenkins, that already handles polling, jobs, and notifications. The URLTrigger works well for this, and we can simply ask it to track the last modified date of the URL however frequently we’d like, up to the minute if we so wish. From this point on we could write the XProtect.meta.plist file ourselves, package it and automatically push it to test clients, or anything we could dream up.

If we want to extract more information, we can also write a very simple script to do so, put it into a versioning system along with the PluginBlacklist information, etc. This example just prints out the version of the meta plist:

Lots of ways to solve a trivial problem! I’m not advocating whether or not to take measures to disable the XProtect mechanism on all your managed clients, but this might at least give some ideas on how you can at least be kept up to date with Apple’s new minimum security requirements for plugins, known malware, and anything else Apple will add to XProtect in the future.

It’s written in Python, but if you want to run it on Windows, there’s a single-file executable available as well, so that Python isn’t required to be installed.

This might be interesting to you if you are doing an automated install of the drivers as a post-imaging task for deploying Windows images, because it handles downloading the correct package for the hardware running it. It’s also a convenient way to get the right install package for a particular model (if, for example, you support other technicians setting up Boot Camp, but for whatever reason they’re having problems getting the drivers.)

It can be downloaded from its GitHub repo here.

fpm -t osxpkg

Jordan Sissel’s FPM packaging tool was designed for systems administrators to very easily roll packages for RedHat, Debian and other platforms and abstract away the obscure details of the package formats themselves.

FPM is written in Ruby, and I decided (somehow) it could be a fun learning exercise to look at implementing support in it for building OS X packages. FPM version 0.4.27 was released a few days ago, available as a gem install, and now supports OS X package input and output, at least on OS X platforms with pkgbuild installed (built-in on OS X 10.8 and 10.7, for 10.6 requires an installation of Xcode 3.2.6 or later).

OS X packages are almost always used in a very different context than rpms and debs (and you are hopefully never going to need to convert an rpm to an OS X package), but building them with FPM still allows one to benefit from some cool features FPM provides all with a single command invocation:

• automatically build packages from RubyGems, Python packages and others. fpm -s python -t osxpkg psutil builds a pkg of the latest psutil Python module that will install to the build system’s default site-packages folder. You can specify an alternate version to fetch with --version.
• pre/postinstall scripts can be ERB-templated with any properties of the package, ie. the version.
• a package’s source can be a simple tarball, and you define the installation prefix.

In addition to what we get with FPM for free, the osxpkg package type in FPM also supports a few extras, the more noteworthy ones being: custom restart-action and dont-obsolete values as simple command options rather than requiring manual PackageInfo templates. More options could easily be added as FPM already has built-in methods for templating package metadata files.

brew pkg

I was recently learning how to update a Homebrew package (ie. ‘formula’) so that I could compile an up-to-date version of dfu-programmer on my Mac. I was reminded of several times at work when recent versions of packages I wanted to install were available in Homebrew, but didn’t want to just install Homebrew on a server, as it is targeted more towards developer systems and assumes ownership of /usr/local. I did like the idea that specific ‘formulae’ are maintained voluntarily by people who probably know more than I about configuring/running a particular software on OS X, and that these are shareable and forkable if I wanted to improve one myself.

Homebrew provides a nice way to provide added functionality via custom scripts, and so was born brew-pkg, a command to build an OS X installer package from an installed formula, with a couple bonuses:

• put service-related launchd plists in /Library/LaunchDaemons rather than a user folder
• optionally include a package’s dependencies in the built package

Packaging is somewhat of a black art on OS X. The Flat Package format has been in existence since 10.5, but only recently are more 3rd-party packaging tools like JAMF Composer starting to move to this format by default. PackageMaker’s days as a hidden download in the Apple Developer Center Auxiliary Downloads package are numbered. In this post I’ll look at one aspect of the package system that’s perhaps less widely known, the “ownership” of a file to a package, and how this affects behaviour that can be tweaked when building flat packages, using pkgbuild as the reference package-building tool.

By default, when an OS X package is installed, if it is an upgrade of a previous version (in other words, there is a package with the same package identifier of a lower version number already installed), any files in its payload that were present in the previous version and not in the new version will be removed from the filesystem. This is because these files are associated with that version of a package in the package (ie. “receipts”) database. When a new version of the package is installed, the Installer framework helpfully removes these files that are no longer part of the package’s payload. In other words, if you install file /usr/local/bin/my_script in version 1, and instead only /usr/local/bin/my_new_script in version 2, my_script will be deleted if it was present when version 2 was installed, assuming that this package identifier is still known to the package database. (It wouldn’t be if, for example, pkgutil --forget my.package.identifier was ever run manually or by an installer script).

We can use the built-in pkgutil tool to query the package database for metadata about packages (BOMs, identifiers, versions, etc.). To examine what package installed pkgbuild, for example, on this OS X 10.8.2 system:

com.apple.pkg.BSD is part of a standard OS X install.

One curious PackageInfo element documented by Stéphane Sudre is the dont-obsolete element. As its name might suggest, “obsoleting” seems to be responsible for removing these files that are no longer present in upgraded package versions.

A real-world example: We recently had an internal support package included in our thin deployment image, one of whose payload items contained a template configuration file that was modified later as part of our DeployStudio workflow. I updated this support package with newer versions of unrelated components to be imported into Munki to push out to clients automatically, wanted to not overwrite this file when Munki would automatically install the package, and at the same time not remove the old one. The dont-obsolete element turns out to be one solution to solve this problem within the logic of the package payload itself, without requiring workarounds in postinstall scripts that could potentially cause issues and greater complexity in the long term.

We can use the undocumented (but documented here by Greg Neagle) --info option for pkgbuild to supply the dont-obsolete element to be included in the final built PackageInfo file, using a template PackageInfo that looks like this:

And to build the package:

Assuming the package we’re upgrading uses the same identifier and is a lower version number, this should do what we want, which is to persist the file at /usr/local/bin/my_script after the installation, even if it’s not included in the 2.0 version of our package.

Instead of using the --info option, we could also build the package without it, then later use pkgutil with the --expand option, edit the PackageInfo file manually and finally --flatten it as I documented in this earlier post on iOS Simulators. This is of course much more work, but is easily automated.

Two asides:

pkgbuild also includes a useful --prior [pkg-path] option, which will automatically derive the identifier, version and install-location parameters from a package located at pkg-path. According to the pkgbuild manpage, however, this will convert the version number to an integer and increment it, which may not be what you want.

Also, there are some exceptions to when files are actually “obsoleted”. For example, removing old files will not necessarily occur if the PackageInfo file has bundle-version elements defined, which would happen if one is packaging, for example, an application bundle. Using pkgbuild with the --root option will perform some analysis on the contents to extract bundle versions and place these in the PackageInfo file. Look into the --analyze and --component-plist plist options in the pkgbuild manpage to customize these further. Managing bundle metadata with the pkgbuild and productbuild tools is quite a bit more advanced, so the example in this post is mainly useful for application/site-specific data in simple files.

Then Lion was released, and supported nearly every Intel Mac, leaving only the first generation models with 32-bit Core processors behind. Then Mountain Lion was released, and the compatibility matrix became more complex. With Apple announcing at the same time that they would be releasing a major new version of OS X every year, we can expect this trend to continue, and that we’ll need to know exactly which models can boot which versions of OS X. We’ll take a look at how we can offload this decision to the NetBoot server to make the process as simple as possible on the client end.

Enter model filtering

NetBoot has had support for model filtering for some time, which allows you to filter what models of Mac will be allowed to boot a given image via the NetBoot service. The goal is that you can maintain multiple OS versions of an installation or utility NetBoot image (like DeployStudio’s Runtime image) and let the server handle offering the version that a client can actually boot. NetBoot allows you to specify a ‘default’ image, which can be useful for older Intel Macs whose EFI firmware doesn’t support selecting from multiple NetBoot images, but the end goal is that someone can hold the ‘N’ key to NetBoot a machine and not need to know what OS they need to boot what hardware.

DeployStudio recently added a new feature in its Assistant application (which effectively drives the sys_builder.sh script contained within its application bundle) for configuring NetBoot Runtime images to support hardware model filtering. Specifically, they needed to add the DisabledSystemIdentifiers and EnabledSystemIdentifiers arrays to the NBImageInfo.plist which resides at the root of the resulting .nbi folder. Even if these are empty arrays, the Server GUI will enable the editing of model filtering once these keys are present.

One downside to Apple’s provided GUI for the NetBoot service is that this database of Mac models is only kept up to date for the lifespan of that version of Server. For example, if you are still hosting NetBoot on Snow Leopard Server, you can’t instruct the service via the GUI to whitelist a 10.8 image to a 2012 MacBook Pro, because this model metadata was never backported to Server Admin for 10.6. In fact, it’s the last month of 2012 and I can’t find metadata pertaining to any
2012 Mac in the most recent versions of OS X Server for both Lion and Mountain Lion (and as of Mountain Lion Server, NetBoot is now NetInstall).

As of December 2012, no hardware more recent than “Early 2011″

Configuration

The model filtering selections are stored using hardware model identifiers (in the familiar style ‘iMac10,1′, ‘MacBookAir4,1′) in the NBImageInfo.plist file. You can edit this file directly as it’s in XML format, though you should probably work off a copy for safety. You may also need to restart the NetBoot service in order for it to reload the changes.

For my environment, I decided to offer two NetBoot Runtime images: one for 10.6 to cover all older hardware, and a 10.8 image for all other Macs. Here’s my current NBImageInfo.plist for the 10.8 image:

My goal was to offer only one image for any given model I want to support, so here I’ve put each model in either one of DisabledSystemIdentifiers or EnabledSystemIdentifiers arrays (I’ve skipped Xserves, but I don’t think I’ve missed any others). The contents of these arrays are reversed in the 10.6 image, so that every model identifier is only enabled for one image. In the near future when we have 10.9-based images, it’s likely that we’ll still only require two generations of NetBoot images, but using this method we could have more if we desired.

Because this plist is per-image and it’s labour-intensive to build the lists of system identifiers, it would be a good idea to have backups of these lists to use as new versions of NetBoot sets are built for new versions of OS X and DeployStudio. Or, script adding them automatically after calling sys_builder.sh with pre-defined arguments.

Sources of truth

Apple makes use of the ‘board-id’ to determine whether a version of OS X 10.7 or 10.8 can be installed on a given machine. This can be retrieved via the ioreg command (ie. ioreg -l | grep board-id) and looks something like Mac-F4208CA9. Lists of these board-ids are in the Distribution files for the OS installers, and are also in a plist at /System/Library/CoreServices/PlatformSupport.plist within an InstallESD.dmg. What’s also interesting is that as of OS X 10.7.5 and 10.8.1, this plist contains a second array called SupportedModelProperties, which is a list of model identifiers. It would seem at first that perhaps this list could be directly mapped to the arrays we’re managing in the NBImageInfo.plist. However, there are several hardware models missing from SupportedModelProperties that I know are supported versions of the OS on those models, so unfortunately it looks like this isn’t a definitive list of models that’s used by installer, or at least not for determining eligibility for installation. A Mac will send its board-id and serial number to Apple via HTTP during Internet Recovery, but I’m not aware of a model-to-board-id mapping source anywhere. It’s possible that an organization with enough Macs could compile one itself.

For the many models that can boot both 10.6 and 10.8, there’s no systematic reason I had for choosing one OS version over another for a given model. You might have your own reasons – newer OS versions of Runtime NBIs have sometimes taken longer to support all workflow features, for example live package installs were not working for some time on Lion-based Runtime images.

Testing with VMs

Lastly, if you do implement this method and use VMware Fusion to test your NetBoot environment, your OS X VMs will not boot without a minor change. In Fusion 4.1, VMware added a new configuration option for VMs: hw.model.reflectHost. This option can be set in the VM’s .vmx file located at the root of its .vmwarevm folder, and passes the host’s hardware model string through to the VM. Pepijn Bruienne also pointed out on Twitter the hw.model configuration option, which is even better: it allows arbitrary model strings, which allows you to test that a specific model will boot the image you intend it to. You could also use this to test any installer or script logic that depends on hardware model identifiers (for example, hardware-specific Apple Software Updates).

It’s well-known that all this database information is stored in plain XML plist files in the DeployStudio repository, one per computer, named after the value of the computer’s primary key (serial number or MAC address). Sometimes people have wanted to manage this data from an external source like a web form or script that can be used by technicians deploying new hardware, but run up against the fact that changes to these files can only be loaded by restarting the DeployStudioServer service. That’s by design. These plists are DeployStudio’s database, and we don’t directly interact with an applications’s database if we can ever help it; that’s what APIs are for, and DeployStudio has a basic REST-style API which it uses to perform all its communications between the server, admin client and runtime instances. This post will show some basic examples of how simple it is to interact with DeployStudio via command-line tools, and a Python example for setting arbitrary properties in the computer database.

Navigate to http[s]://your.deploystudio.server:port in your browser and fill in your credentials for a user with rights to DeployStudio Admin. You’ll see a list of methods, the left-hand column using GET for retrieving values, the right-hand column using POST for setting values.

If you use a web debug tool like Charles to inspect your traffic going out to your DeployStudio server, you can pretty quickly get a sense of what’s going on. For example, when DeployStudio Admin connects to your repo, one of its requests is to /computers/get/all, which will retrieve all computer database information that the Admin app will use to populate its various text fields and table view. This information comes in the same form it’s stored, as XML plists, and your Admin client is sending changes back out in that format as well. Only difference is that instead of editing plist files on disk, DeployStudio Admin is making an HTTP request and sending the XML plist data using the POST method. There are some subtleties to account for in the contents of these plists, but it’s fairly straightforward.

We can use curl to take a look at the structure of what’s received. DeployStudio uses HTTP Basic Auth everywhere, so you can have curl encode the appropriate header with the -u username:password option, or leave out the :password component and it will prompt you. We’re also using the -k option because in these examples we’re connecting using SSL and want to bypass the warning that DeployStudio is using a self-signed certificate. We can fetch all available computer info using the /computers/get/all endpoint and save it to a file at out.plist:

Opening it up, this is how it looks:

In the XML returned, the computers dictionary contains dictionaries each named by the computer ID, either a serial number or a MAC address. Here we have DOUGLASFIRS and HOWSANNIE. In this example we’ll look mainly at the serial number, computer name and hostname (dstudio-host-serial-number, cn and dstudio-hostnamekeys, respectively), but there’s lots of other empty fields, particularly all the networking-related information.

Say we want to just rename the DOUGLASFIRS computer, by changing its computer name and local host name to ‘diane’. (While I don’t typically use hostnames for anything, the hostname has the privilege of being one of the few columns available in DeployStudio Admin for identifying a computer in the list.) We’ll use PlistBuddy to change the cn and dstudio-hostname keys in the plist we just output:

There’s one more detail to take care of. We’re going to send this updated computer record plist to the server using POST to the /computers/set/entry endpoint, but this endpoint expects to be passed only the contents of a single computer dict, using an id parameter to identify the record we’re modifying. So in order to simplify our XML slightly and only send the contents of the DOUGLASFIRS dict, we’ll again use PlistBuddy – this time using the -x parameter to output in plist format – to retrieve just the nested DOUGLASFIRS dict, and write this out to a new file at douglasfirs-renamed.plist:

We could just have easily done all this with a text editor, but I’m using PlistBuddy because it’s one way these kinds of changes could be made automatically via a script rather than manually. Now our updated computer info in douglasfirs-renamed looks like this:

We’re ready to commit these changes back to DeployStudio, so we’ll again use curl to POST the data using the --data option and prepend the file with an @ symbol to specify we want to feed it the contents of the file:

The same endpoint is used if we want to create a new record – we would just format a new computer plist containing the fields we care about and submit it using the appropriate ID. A new record need not contain all these blank placeholder keys; a new computer record created in DeployStudio Admin will only contain dstudio-host-primary-key and one of dstudio-host-serial-number or dstudio-mac-addr. It’s important to know that these other keys can exist (and definitely will if the record has ever been modified in Admin), because a plist sent to this endpoint will completely overwrite the existing plist. Therefore, if one is modifying an existing record and wants to preserve any other values that may be contained in the record, one should always fetch the existing record and make changes. This is why we used PlistBuddy above to copy the individual computer dict from the existing data out to a new plist.

One other thing to note if you’re testing this and not seeing the changes in DeployStudio Admin: in order to get the changes to the computer database, DeployStudio Admin needs to do another GET to /computers/get/all, which can be triggered by switching to a different section in the left-hand sidebar. DeployStudio Admin is not refreshing all the computer information periodically, likely because it expects to be the only application editing it and it’s already keeping track internally of local edits.

I used curl and PlistBuddy because they’re capable tools that any seasoned Mac admin is familiar with. There’s certainly nothing wrong with using them in this context, although a higher-level scripting language like Python is particularly well-suited to this task for a couple reasons. It has a built-in module to handle XML plist data as native Python dictionaries, meaning no dealing with PlistBuddy or defaults and their esoteric syntaxes is required. It ships with a great standard library of modules that can perform all this functionality without requiring a shell or external (platform-specific) binaries. This also means any code written in Python using these built-in modules can be ported to a web application running on Linux or Windows with few or no changes.

Here’s a short Python example of a couple simple operations using the same endpoints we used above, as well as manipulating a custom property representing an organization’s asset tag. The updateHostProperties() function allows us to specify a computer ID and a dictionary of properties we’d like to change. It transparently handles creating a new record if one doesn’t already exist. There’s a lot of room to improve its robustness as it performs no real error handling, but this is omitted to keep the example shorter. The two example operations it performs can be found commented in the main() function at the end, and with each invocation it will simply generate a new computer/hostname of the form ‘random-id-n’.

Some downloads, like the Command-Line Tools and earlier iOS Simulator/SDK versions, show up in a new “Components” download area located in Xcode’s Preferences. We’ll look at where this index comes from, how we can inspect it to get the iOS simulator .dmg download URLs, and one method of modifying the simulator installer packages so that they install to the correct location via any standard package distribution method like Munki or Casper. If you manage installing the Command-Line Tools as well, you’ll find metadata here that will help with tracking version installs (anyone who’s tried to manage deploying/updating them knows they don’t use Apple package versioning).

Xcode Preferences: Downloads

We’ll also quickly review the other steps typically required to “finalize” the Xcode installation for most deployment scenarios. Big thanks to Nate Walck for testing what I’d originally documented for the iOS Simulator installation and determining that I’d skipped an important step!

Deploying Xcode via a software management mechanism such as Munki, Casper or ARD usually requires a few steps for a fully-functional install. These usually include:

• copy the Xcode.app bundle from the .dmg to /Applications
• copy out the MobileDevice.pkg (and MobileDeviceDevelopment.pkg as of Xcode 4.5) from Xcode.app/Contents/Resources/Packages and install them separately
• assuming your users are not admins, adding an appropriate user group to the _developer group, and running /usr/sbin/DevToolsSecurity -enable, which handles modifying the security policies in /etc/authorization
• optionally install the Command Line Tools for the appropriate OS version
• optionally accept EULAs and configure the options for downloading extra components and documentation, via the defaults domain at com.apple.dt.Xcode

Installing these additional components has been already documented in several places, but I hadn’t yet seen anyone describe how they deployed versions of the iOS Simulator/SDK, which normally would be an optional component download.

Xcode seems to automatically include the most current version of the SDK in its .app bundle, so this is primarily useful if you want to be able to deploy older versions as well. But if you manage Xcode it’s a good idea to become familiar with this index file anyway.

I recently discovered via the Charles web proxy tool what Xcode actually downloads from Apple to populate its list of additional components for download, which is also used to determine how they’re installed. It turns out these indexes are also cached locally on the client, so no web traffic sniffing is even needed.

Once you’ve launched Xcode at least once as a user, head over to that user’s Xcode cached downloads folder at ~/Library/Caches/com.apple.dt.Xcode/Downloads.

In mine, I have some cached downloads (which Xcode helpfully renamed to saner filenames than it stores on its downloads site), and a few different index files. Each one is from a different version of Xcode. On this machine I’ve gone from version 4.3 to 4.4. to 4.5, and I believe these index filenames remain the same for patch versions. The only one getting updated now is the last one dated Nov 13, f9556a…dvtdownloadableindex.

These cached versions are binary plists, so either first convert them to the XML plist format or open them an editor like TextWrangler or TextMate, which support transparent conversion of binary plists. If we open this up, we can see it serves a purpose similar to Apple’s Software Update .sucatalog files, except all its logic and metadata is inline. One item to take note of is the source key at the bottom:

The GUID in this URL seems to be unique to the Xcode major version, as is the SHA-1 that makes up the filename stored in the local cache. A couple useful header values from the web server for this URL:

The ETag in this case is an md5 of the file and the time it was modified in the Unix time format. Given this, you could monitor changes to either of these headers to know when there may be new changes in the index. (Of course, this index URL would likely change when there’s a new major release of Xcode. At the time of writing the most recent version is 4.5.2, but inside the index there is at least one update restricted to only the preview release of Xcode 4.6).

The meat is in the downloadables array, so let’s look at one dict entry from it:

Lots of useful, readable information here. We see the ‘displayed name’ Name key, the identifier and version, the .dmg download at source, and some logic for how Xcode determines it’s installed via InstalledIfAllPathsArePresent.

Notice the RequiresADCAuthentication key, which is fairly new. Previously these legacy Simulator downloads were hosted at http://adcdownload.apple.com, and would prompt you for an Apple ADC login. Currently there aren’t any downloads that require an ADC login, so Apple’s likely just reserving this behaviour for future use. It means it’s easier for us to actually grab the .dmg without needing an auth cookie set by our browser, which used to require a manual download of Xcode in order to have it set.

The other very important piece here is the InstallPrefix key, set to $(DEVELOPER). We’re going to need this to actually install the .dmg correctly.

Once we’ve downloaded and mounted the installer .dmg, we can see it’s just a .pkg installer. The installer is a flat package, and we can use the pkgutil to expand it to a working directory on our desktop and take a look:

We’ve got the most basic flat package structure possible: a Bom, PackageInfo and Payload file. We can use the lsbom command to get a list of all the payload items. If we take a look at the first few items, we see the top few items in a folder hierarchy:

Those ‘./’s at the beginning look like relative paths, don’t they? They are. If we’d install this package now using the installer command and specify a root OS volume like /, we’d wind up with this Platforms folder at ther root of our drive, which is not what we want. If we dig around inside the Xcode app bundle, we can find where the more current iOS Simulator is stored, at:

This correlates with the InstallPrefix key we saw earlier, telling us that we should effectively prefix this package’s target location with folder with /Applications/Xcode.app/Contents/Developer/. To do this, we’ll edit the PackageInfo file that contains metadata about the package. Specifically, we want to set the install-location attribute in the pkg-info element. See Stéphane Sudre’s excellent Flat Package Format page for more info on flat packages. With our small modification, the PackageInfo should now look something like this:

Note the less-than-useful version number in the pkg itself. Now we can compile this back to a new flat package using the --flatten option for pkgutil:

The resulting file, iPhoneSimulatorSDK-5.0.0.1.pkg, should now be installable as an addition to your Xcode installation.

Unfortunately, we’re still not done. If we skim through the index for this SDK’s identifier, we come across another entry, that contains a PatchFor key:

Again, the logic is clear. It’s a patch for version 5.0.0.1 of Xcode.SDK.iPhoneSimulator.5.0, and Xcode will know it’s installed according to the contents of the InstalledIfAllSHA1SumsMatch key. In this case, it’s the SHA-1 hashes of a bunch of library files. If you inspect the BOM of this patch installer, you’ll see it’s only installing the same patched libraries given in this InstalledIfAllSHA1SumsMatch dict. So, to install this you’ll need to perform the same modification to the PackageInfo file to set the install-location attribute.

It’s important to apply this patch, because if it is not applied, Xcode may detect that an installed component is not fully up to date, and prompt a dialog at launch, similar to when the MobileDevice support package is not installed or the correct version:

A user can skip through this by failing the authorization prompt, but there’s otherwise no option to simply bypass the update.

With that, we should have everything we need to deploy a specific legacy iOS Simulator/SDK version. We know where to get the installers, what version they are, how Xcode knows whether they’re installed, and information for dependencies and patches that may see new uses in future versions. Your software distribution mechanism should be able to use all of this metadata to manage installations and updates for Xcode. If it doesn’t, you may need to supplement it with some of your own checking mechanisms.

This post’s specific use case of iOS Simulators may seem esoteric, but it’s a good example of the index metadata Xcode uses for its supplemental downloads, and a practical use of the pkgutil command-line tool. With more and more packages being distributed in the flat package format, being familiar with using it to audit and modify packages is essential for administering OS X.

Mountain Lion introduced a new iOS-like feature to allow users to be notified when an application requests access to that user’s contacts:

…and to be able to modify this access later:

Why does Final Cut Pro 7 want to access contacts? Final Cut Pro 7 introduced a feature that uses iChat (which doesn’t even really exist in Mountain Lion), therefore when a user first launches FCP, OS X will ask permission to allow FCP to access that user’s contacts.

It might be nice to be able to pre-allow or -disallow access for applications without user intervention, especially in scenarios where user Library data isn’t persistent across logins in a multi-user environment, where users would otherwise be nagged frequently to access what’s likely an empty Contacts database.

These contacts turn out to be stored in an SQLite3 database, located in the user’s library folder:

~/Library/Application Support/com.apple.TCC/TCC.db

This database is managed by the tccd daemon located at:

/System/Library/PrivateFrameworks/TCC.framework/Resources/tccd

We can query it with the built-in sqlite3 command-line tool:

There’s some useful information in the access table:

Here we’ve got FCP and Google Chrome (the latter triggered because we visited GMail). The last three integer columns store the values for client_type, allowed, and prompt_count.

Given this very simple database schema, it’s pretty trivial to update this database ourselves directly. In my experience, when the database gets updated, the Security & Privacy preferences pane responds to the change as soon as its window again receives focus.

I wrote a Python script that allows for ad-hoc changes to the allowed column, given an app bundle id. It should also be usable in scenarios where a user may be brand new and not yet had the TCC database created, so it will handle the initial schema creation of the database if it doesn’t already exist.

This is available here. It currently assumes you’ll be running the script as the user whose database will be modified.

On OS X, Acrobat Pro X and Reader 10 became distributable in the standard Apple pkg format, and this was generally a huge improvement for the deployment and update process. Acrobat Pro 9 currently requires twenty sequential patches required to bring Acrobat Pro 9 to an up-to-date version.

Too many updates.

Things are much better now, but configuring a common setting such as disabling update checks for all users has remained unnecessarily complicated, for despite Adobe using a property list to store these parameters, they were per-user only, requiring these to be managed either using MCX/Profiles or a manual script to apply the appropriate preference in every user’s Library folder (ie. at login time with a LaunchAgent).Adobe has alluded to built-in functionality for disabling updates in Pro X in the past, so we’ll summarize what’s been presented to date:

Adobe Provisioning Tool

The Adobe Provisioning Tool would have you believe, going by its usage statement, that there is a -M option to suppress updates. There is no such functionality anywhere in the adobe_prtk binary on OS X.

FeatureLockDown file

The AAMEE Technical Note on “Installing and Configuring Exception Payloads” suggests you can configure the FeatureLockdown system. It says to just modify a file inside the Acrobat Pro X .app bundle at Contents/MacOS/Preferences/FeatureLockDown and change this 822-character line:

to this 843-character line:

I did not find this method to work reliably.

You might also come across the topic of updates in the Enterprise Administration Guide for Acrobat, and see a few options there (Section 15.6, page 139). We’ll examine these in reverse order, because up until recently this would have been my order of preference (given that the first option was broken until very recently.)

Option 15.6.3: Remove the Updater plugin itself

This option is looking pretty good at this point, which is to remove the Updater plugin entirely to prevent it from checking for updates. However, we would need to make a point of doing this every time we run an updater pkg for Acrobat.

It’s clear that we’ve reached a less read-over section of the guide, as there are ambiguous terms and incorrect filenames/paths in nearly every written line of this section. Because Adobe so rarely follows system conventions in these implementations, I didn’t feel like could easily gloss over an error and assume the writer meant to write something else.

Option 15.6.2: Set the update mode to manual

“The update mode is set on a per user basis as follows: (…)”

We can skip this – this is what we’re already having to do, set per-user preferences with MCX, a Profile or script. In this case, we can assume the writer meant to use the correct path to the plist file, and not one that doesn’t normally exist on an OS X system.

Option 15.6.1: Disabling and locking the Updater

This sounds familiar to that FeatureLockdown no-line-break mess we’re still trying to forget we were ever desperate enough to try in the first place. And in fact, it does seem to be using the same FeatureLockDown subsystem to disable the plugin, except it’s writing the bUpdater boolean value in a plist. The documentation doesn’t say at all where this plist is, but we can take a good guess.

The obvious one to try is the system default preferences folder at /Library/Preferences. We can use these plist contents for both Acrobat Pro X and Reader, using the plist names com.adobe.Acrobat.Pro.plist and com.adobe.Reader.plist, respectively:

It’s documented to work as of 10.1.1, but it only works as of version 10.1.4 of Acrobat and Reader. Adobe was pretty responsive in fixing the broken functionality once it was reported.

So now, the answer is pretty simple. Push a simple-enough plist to a system location or manage the preference as you would most others that work with the defaults system. I went through the previous options just as a demonstration of how Adobe manages to, in duplicating their own efforts in software implementations, effectively duplicate efforts by many orders of magnitude across IT organizations worldwide, who simply need to deploy a common application without resorting to scripted hacks and workarounds to manage very trivial behaviour.

It’s great that Acrobat Pro X and Reader are getting easier to deploy and configure on OS X, but it’s been a long road getting there.