A Tour of Charles, Your HTTP(S) Swiss Army Knife

April 22, 2014

There are times when it’s helpful to be able to know exactly what HTTP traffic is being sent or received on your Macs. Perhaps you’re auditing a 3rd-party application to see what connections it makes to outside servers, or maybe you’re interacting with – or writing – a REST API. Perhaps you just want to see every transaction between you and Apple’s servers when you use the Mac App Store to download apps, or use Internet Recovery.

Anyone doing systems administration long enough will have eventually used the packet capture library in some form, usually in the form of tcpdump and/or the Wireshark application, a powerful set of tools for analyzing all types of network traffic. This is very useful if you’re writing a NetBoot server replacement and need to inspect at the packet level, but if we’re only interested in HTTP(S) traffic, there are better, more specialized tools available. In this post I’ll introduce Charles, a web proxy and GUI tool for inspecting and diagnosing HTTP traffic. Since I’m often interested in knowing how software performs update checks, I’ll use this as an example.

For a simple example for getting familiar with Charles, let’s say we want to know how an application checks and notifies the user about new updates. This would be useful to know if we’d like to write an AutoPkg recipe that’s able to automatically download the latest version of an application. We’ll look at the Adium instant messaging client. Note that these URLs will eventually be out of date and may not work to try on your own, but hopefully the output here will illustrate our work well enough.

Like many other applications on the Mac, Adium uses the Sparkle framework to handle the application’s built-in updater mechanism, which retrieves update information from an RSS feed URL. AutoPkg, conveniently enough, has a processor that’s able to take a Sparkle RSS URL as input to its downloader processor, so we don’t need to write our own parsing mechanisms. This RSS URL is usually given in the application bundle’s Info.plist file in the SUFeedURL key. In this case, it’s http://www.adium.im/sparkle/update.php. So, what do we need Charles for? Can’t we just feed this URL to AutoPkg? Let’s test it with cURL:

curl -L http://www.adium.im/sparkle/update.php

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle">
<channel>
<title>Adium Updates</title>
<language>en</language>
<item>
<title>Adium 1.1.4</title>
<pubDate>Monday, November 5th, 2007 20:00:00 GMT-5</pubDate>
<sparkle:releaseNotesLink>https://www.adium.im/changelogs/1.1.4.html</sparkle:releaseNotesLink>
<enclosure sparkle:md5Sum="a3013db6c56a8bd249191ced6d749b30" sparkle:version="1.1.4" url="http://adiumx.cachefly.net/Adium_1.1.4.dmg" length="16463533" type="application/octet-stream"/>
</item>
</channel>
</rss>

What’s going on? Version 1.1.4 is old. The most recent version of Adium at this time of writing is 1.5.9, and if we’d go and actually check for updates in Adium, it would report that it’s at the latest version. We need to see exactly what’s going on, and this is where we can use Charles.

Charles is a paid software, but it also functions as a time-limited trial, which is more than enough to get our feet wet. The first time you launch Charles, you’ll get a couple prompts. One will be an an admin user prompt so that Charles can register the rights to manage your system’s network proxy settings. This allows Charles to automatically configure the local system’s HTTP and HTTPS proxy to point to itself (by default using port 8888). This feature is extremely convenient. Launch Charles and immediately see traffic flowing through it. Close Charles, and the system is automatically reconfigured as it was before.

Another dialog you’ll see on first launch is the offer to help configure Firefox’s proxy settings, which are independent of the system’s. Some other applications will use their own proxy settings rather than those managed in OS X’s Network settings - command-line tools like cURL require setting the http_proxy and https_proxy environment variables, for example.

With no other configuration, you should start seeing some traffic showing up in Charles on the left-hand sidebar, as Charles by default begins recording HTTP traffic in a new session automatically when it is launched. The traffic you see comes from whatever applications may be running on your computer, website sessions, etc. Now we can open up Adium, choose “Check for Updates” in the Adium menu, and look at what Charles has saved for us.

Adium checking for updates. Note the HTTPS redirect, but no recorded data in the HTTPS URL.

Sure enough, there’s the request to http://www.adium.im, and we can open the disclosure triangles that open with each path component in the URL. We can see the final part of the path, and also click on it to copy the full URL: http://www.adium.im/sparkle/update.php?generation=2&type=release. This is what the client actually requested. Try checking the “Update to beta versions when available” and “Include anonymous system profile” preferences in Adium and then see how the request’s query string changes. Explore the different tabs on the right hand view, particularly the different views for the request and response; here we can see each query string broken down. Ever wonder what those “include system profile” settings in applications are actually sending? This is how you can find out!

In fact, we can even edit the request details right here and re-execute them, which can be especially useful if you’re experimenting with a REST API:

Editing request data right within the recording

During this time, if your window is getting cluttered with other requests, you can clear them from the recording with the Trash can icon, selectively delete them, or open the Proxy Menu -> Recording Settings and add something like *.adium.im into the Include Locations, which will filter out all other traffic. Don’t forget to clear it later if you do.

Going back to our initial results, the response was actually a 301, a permanent redirect to an https:// URL. Charles doesn’t proxy SSL by default, so if you look at the https://www.adium.im:443 URL that appears immediately after this request, in the Overview tab, you will see “SSL Proxying disabled in Proxy Settings.” You can still look at the response, but it’s just the encrypted HTTPS traffic, not much use to us here. We’ll come back to configuring SSL in Charles, but let’s quickly use cURL to check the redirected response. You won’t see this traffic in Charles unless you set http_proxy and https_proxy environment variables to localhost:8888 or 127.0.0.1:8888, but we don’t need to bother with that here.

The curl command, again following redirects and quoting the URL to prevent the shell from interpreting characters like ?:

curl -L "http://www.adium.im/sparkle/update.php?generation=2&type=release"

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle">
<channel>
<title>Adium Updates</title>
<language>en</language>
<item>
<title>Adium 1.5.9</title>
<pubDate>Sunday, December 3rd, 2013 18:15:00 GMT+2</pubDate>
<sparkle:releaseNotesLink>https://www.adium.im/changelogs/1.5.9.html</sparkle:releaseNotesLink>
<sparkle:minimumSystemVersion>10.6.8</sparkle:minimumSystemVersion>
<enclosure sparkle:dsaSignature="MC0CFBVnsDG2zHdP3Veq0LckjqrPbI/NAhUAlavqFjFioklXR1L9CdARdle+p2Q=" sparkle:version="1.5.9" url="https://adiumx.cachefly.net/Adium_1.5.9.dmg" length="24516000" type="application/octet-stream"/>
</item>
<item>
<title>Adium 1.4.5</title>
<pubDate>Tuesday, March 20th, 2012 20:30:00 GMT-5</pubDate>
<sparkle:releaseNotesLink>https://www.adium.im/changelogs/1.4.5.html</sparkle:releaseNotesLink>
<sparkle:minimumSystemVersion>10.5.8</sparkle:minimumSystemVersion>
<enclosure sparkle:dsaSignature="MC0CFQDGpxksd++JLPa1+2AVZw/ruHsQSAIUB5REX5PJxM3bYtAKfwvnaR1pfKo=" sparkle:version="1.4.5" url="https://adiumx.cachefly.net/Adium_1.4.5.dmg" length="23065688" type="application/octet-stream"/>
</item>
<item>
<title>Adium 1.3.10</title>
<pubDate>Monday, January 12th, 2010 23:30:00 GMT-5</pubDate>
<sparkle:releaseNotesLink>https://www.adium.im/changelogs/1.3.10.html</sparkle:releaseNotesLink>
<sparkle:minimumSystemVersion>10.4.0</sparkle:minimumSystemVersion>
<enclosure sparkle:md5Sum="16309a78add9dc7695ccc14079baae10" sparkle:version="1.3.10" url="https://adiumx.cachefly.net/Adium_1.3.10.dmg" length="22369877" type="application/octet-stream"/>
</item>
<item>
<title>Adium 1.0.6</title>
<pubDate>Monday, August 13th, 2007 22:12:45 GMT-7</pubDate>
<sparkle:releaseNotesLink>https://www.adium.im/changelogs/1.0.6.html</sparkle:releaseNotesLink>
<sparkle:minimumSystemVersion>10.3.9</sparkle:minimumSystemVersion>
<enclosure sparkle:md5Sum="9e19c217f945b7fd82e46d0fa25a5a9b" sparkle:version="1.0.6" url="https://adiumx.cachefly.net/Adium_1.0.6.dmg" length="13795246" type="application/octet-stream"/>
</item>
</channel>
</rss>

This looks better. We can see that there are both new and older versions, targeted towards clients running older versions of OS X.

We can stop here, because we now know exactly what Adium sends to its update server in order to check for new versions, and we can write an AutoPkg recipe to do the same thing. Of course, one already exists, and you can see how the recipe integrates these additional query strings in the region highlighted in that link.

But we’ve gotten this far; how would we have inspected the encrypted data in Charles if we’d needed to? First we must configure a root certificate authority to be trusted by our system, so that the certificate used by Charles’s SSL proxy can be verified in the handshake process. Charles also makes this easy: From the Help menu, choose “Install Charles CA SSL Certificate…”. This will open the Keychain Access application and prompt you to trust this in the system.

Charles's SSL Proxying cert after it's been added to Keychain

You can also obtain this same certificate from the Charles web site at https://www.charlesproxy.com/ssl.zip, if you are ever setting up another client to proxy to Charles, and install it with the security command, which for this will require root privileges:

sudo /usr/bin/security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain charles-proxy-ssl-proxying-certificate.crt

In addition to this, we need to tell Charles to actually proxy SSL requests, which it does not do by default. We can set this in the Proxy Settings menu. This setting also requires us to specify a host filter, which can be a wildcard *, but could also be limited to the domain we’re interested in. You may find that with it enabled and set to all URL domains, that some services cease functioning due to the use of an alternate SSL CA.

We conclude this tour of Charles - if you’ve read this far, it should be obvious that this tool is well worth its $50 price tag and that it’s is useful for both simple uses (as in this example) and the many advanced features that we haven’t explored in this post.