evanseries.org

I tweeted a few days ago about how long it took Ruby 1.8.6, Ruby 1.9.1, and MacRuby 0.3 to sample 10,000 words from a probability distribution. Then, yesterday, Charles Nutter (one of the lead developers on JRuby) replied to my tweet and asked if I could send the script along so that he could try it in JRuby.

Well, after a day or so of talking back-and-forth with him about it, I’m happy to present the results:

Experiment Details

These tests were run on my 15″ MacBook Pro (MacBookPro3,1) with a 2.4 GHz Intel Core 2 Duo processor and 4 GB of RAM, running Mac OS X 10.5.6. During script execution, Terminal was the only application running (as indicated by the Dock and menu bar).

For each implementation, the script was asked to sample 10,000 words from a list of words and their associated probabilities (which was contained in a file). This task was performed 100 times, and the time it took to execute the task was measured by Ruby’s benchmark library.

The sampling script and the word probability distribution file are available as a gist. (If you distribute the word probability distribution file further, the professor from whom I received it would appreciate it if you’d mention that it came from Carnegie Mellon University’s Language and Statistics class.)

A text file containing the timings for each implementation is also available.

The script was evaluated with JRuby 1.1.6 using Apple’s Java 1.6.0_07, with the 64-bit server VM (build 1.6.0_07-b06-57, mixed mode). The version string for this configuration is jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [x86_64-java].

I built the version of Ruby 1.9.1 used in the experiment on my machine, using the build tools that come with the iPhone SDK for iPhone OS 2.2.1 (19M2621A). (Ruby was compiled for Mac OS X Intel; the iPhone SDK for iPhone OS 2.2.1 is just the version of Apple’s developer tools that I had installed when I built it.) The version string for this configuration is ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-darwin9.6.0].

The version of Ruby 1.8.6 used in the experiment is the version included with Mac OS X 10.5.6 with all available updates installed. Its version string is ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0].

The MacRuby implementation used in this experiment was compiled from the testing branch of MacRuby on January 31, 2009 at 2:44 PM EST, using the same development environment as was used to build Ruby 1.9.1. The version string for the MacRuby configuration is MacRuby version 0.4 (ruby 1.9.0 2008-06-03) [universal-darwin9.0, x86_64].

Discussion

It’s clear that JRuby 1.1.6 is the fastest, taking a median of 1.9805 seconds. Ruby 1.9.1 clocks in more slowly, with a median of 3.29 seconds. Ruby 1.8.6 is slower still, requiring a median of 5.465 seconds to complete. MacRuby’s testing branch is considerably slower than the other three, completing the task in a median of 20.455 seconds.

If we take MacRuby out of the equation, we get an even better picture of the difference between the three fast implementations:

In this view, an interesting pattern emerges. Ruby 1.9.1, built on the YARV/KRI bytecode implementation, is indeed faster than Ruby 1.8.6, as one would expect. In fact, it’s substantially faster: the median time for Ruby 1.9.1 is only 60.2% of the median time for Ruby 1.8.6. And the slowest non-outlier Ruby 1.9.1 time (4.310 s) is still faster than the fastest Ruby 1.8.6 time (4.56 s).

But what I found surprising is that JRuby 1.1.6 is faster than Ruby 1.9.1 in the same way that Ruby 1.9.1 is faster than Ruby 1.8.6. JRuby’s median time is also 60.2% of Ruby 1.9.1’s median time, just as Ruby 1.9.1’s median time was as compared to Ruby 1.8.6, and JRuby’s slowest non-outlier time (2.1590 s) is also still faster than Ruby 1.9.1’s fastest time (2.21 s).

I’m hesitant to extrapolate these results much beyond the sample domain and the present time. It’s obvious that MacRuby has a lot of work to do if it’s to become as fast as the other three at these kinds of tasks, but it’s still in the early stages of development, so a lot of work is to be expected. In the same vein, Ruby 1.9.1 is a milestone on the way to Ruby 2.0, but it’s not there yet, so perhaps some speedups are on the way.

But if you’re looking for a fast Ruby implementation, at least as far as random word sampling is concerned, my data and I can recommend JRuby to you as the speediest solution of the four.

Update (February 5^th, 2009)

I just finished running the word sampling script in the most recent nightly build of JRuby available, and it’s even faster than JRuby 1.1.6:

The median time for the nightly build, 1.724 seconds, is only slightly slower than 1.7200 seconds, the fastest time for the stable build. In addition, the nightly build’s median time is 87% of the stable build’s median time.

From these results, it’s fair to say that the JRuby team certainly isn’t standing still when it comes to performance, either.

According to the Apple Support document for the OS X 10.5.6 update, “Contacts, calendars, and bookmarks on a Mac automatically sync within a minute of the change being made on the computer, another device, or the web at me.com.”

Excellent. I’m glad to see that the MobileMe team was able to achieve their goal of making syncing “near-instant”.

Seen at the UPMC Shadsyide hospital:

Brent Simmons just put a post up on his design philosophy for category methods, in which he presents the following rule:

If something is used just once, don’t make a category for it, just make it a static C function. Unless it’s big.

His reasoning is as follows:

1. Less code to maintain. (And, in this case, two fewer files: one deleted .h and one deleted .m.)
2. The code appears where it’s used: no need to jump to it if I need to edit it.

I’ve had this rule for a while, as well, and I think it’s a good one. But I have a third reason for employing it: if you’re only using a method in once place, you have no idea how it’s going to generalize. Maybe you can reuse it, as-is, in many other places. Perhaps it will need to be decomposed.

But trying to figure this stuff out the first time you need the functionality is a waste of time, because you’re likely to guess wrong. It’s better to save your energy for when you don’t have to guess at all: the second time (or the third time, or the fourth time) that the problem shows up, when you can refactor your way to a truly general solution.

I got bored the other day and ran the SunSpider JavaScript benchmark in Safari on my iPhone 3G running the 2.0.1 OS release, with no other tabs open. Here are the results.

But the punch line is that Safari 3.1 running on my MacBook Pro is 45 times faster at JavaScript than Safari 3.1.1 running on the iPhone.

(Safari on the iPhone is still really impressive, of course.)

I was replying to an email that Brian had sent me about the pros and cons of MobileMe, and was about to write about how the iPhone will always send email with the From address set to that of the “default” account.

But, for some reason, I wondered if that was still true. So I added my CMU account to the iPhone (in addition to my MobileMe account), started to compose a new email, and saw, much to my delight, that it is now possible to choose which email address to send from! This functionality works in new messages, replies, and forwards.

Perhaps the only downside is that Mail on the iPhone doesn’t automatically choose the From address to which the email was sent, like Mail on Mac OS X does, but I wouldn’t be surprised to see that feature added in a future update.

All-in-all, though, this is a strict improvement, and I’m glad to see it (and be able to use it)!

I just received this email from Apple:

We have recently completed the transition from .Mac to MobileMe. Unfortunately, it was a lot rockier than we had hoped.

Although core services such as Mail, iDisk, Sync, Back to My Mac, and Gallery went relatively smoothly, the new MobileMe web applications had lots of problems initially. Fortunately we have worked through those problems and the web apps are now up and running.

Another snag we have run into is our use of the word “push” in describing everything under the MobileMe umbrella. While all email, contact or calendar changes on the iPhone and the web apps are immediately synced to and from the MobileMe “cloud,” changes made on a PC or Mac take up to 15 minutes to sync with the cloud and your other devices. So even though things are indeed instantly pushed to and from your iPhone and the web apps today, we are going to stop using the word “push” until it is near-instant on PCs and Macs, too.

We want to apologize to our loyal customers and express our appreciation for their patience by giving all current subscribers an automatic 30-day extension to their MobileMe subscription free of charge. Your extension will be reflected in your account settings within the next few weeks.

We hope you enjoy your new suite of web applications at me.com, in addition to keeping your iPhone and iPod touch wirelessly in sync with these new web applications and your Mac or PC.

Thank you,

The MobileMe Team

Foremost, I appreciate the fact that they took the time to write an apology, specifically naming MobileMe’s downtime and their use of “push” to describe Mac to MobileMe syncing. And the 30-day extension is a nice touch, as it makes concrete their stated desire for people to experience and enjoy the service.

I’m also encouraged by their assertion that they’re “going to stop using the word ‘push’ until it is near-instant on PCs and Macs, too” (emphasis mine). While this may mean that they never use the word again, I hope (and think it’s more likely) that we’ll see it return with the arrival of Snow Leopard, whose Exchange support will presumably require a framework for quickly pushing updates.

So, while it’s a bummer that they, indeed, had a rocky start, I feel like things have been set right by this email.

Thanks, MobileMe Team.

Since I just struggled with this for a couple of hours, I figured I’d post it here for posterity.

If you want to generate the string representation of some XML in JavaScript, here’s how to do it (using the Prototype 1.6 syntax for element creation), courtesy of Captain’s Universe:

  var contactsElement = new Element("contacts");
  var evanElement = new Element("contact");
  Element.insert(evanElement, new Element("name").update("Evan Series"));
  Element.insert(evanElement, new Element("blog", {'type': 'personal'}).update("http://www.evanseries.org"));
  Element.insert(evanElement, new Element("email", {'type': 'school'}).update('[email protected]'))
  Element.insert(contactsElement, evanElement);
  var zachElement = new Element("contact");
  Element.insert(zachElement, new Element("name").update("Zach Paine"));
  Element.insert(zachElement, new Element("blog", {'type': 'personal'}).update("http://zachpaine.info"));
  Element.insert(contactsElement, zachElement);
  alert(new XMLSerializer().serializeToString(contactsElement));
  

I don’t know if this works with IE yet; I’ll update this post when I find out.

I was really excited about MobileMe, because, in its promotional material, Apple claimed that a change made on your Mac, iPhone, or PC would be pushed “within a matter of seconds” to any other computers hooked up to the MobileMe cloud (see about 40 seconds into the MobileMe guided tour).

For another source, here’s an excerpt from an Apple’s MobileMe features page:

MobileMe stores all your email, contacts, and calendars in the cloud and pushes them down to your iPhone, iPod touch, Mac, and PC. When you make a change on one device, the cloud updates the others. Push happens automatically, instantly, and continuously. You don’t have to wait for it or remember to do anything — such as docking your iPhone and syncing manually — to stay up to date.

The reason that I was so excited about this is because there’s a lot of safety in having your changes quickly pushed to your other devices. For example, if I find out that one of my friends has changed their phone number right before I leave work, it would be great to just update their entry in Address Book, close my MacBook Pro, and feel secure in the knowledge that the change has been pushed to my iPhone, in case I need to call them on the way home.

In this situation, the best part is that I didn’t have to wait some amount of time for OS X to sync with MobileMe, or force a sync by myself. The update is pushed seamlessly, and I don’t even have to think about it. That is one of the main pillars on which Apple promotes MobileMe.

Once it went live, however, I (and many others) found that changes made on our computers were not being pushed “immediately” to our other devices; it was taking several minutes for the updates to show up. But because the MobileMe launch was less than smooth, I chalked up the delay to temporary deployment problems.

Unfortunately, it’s now clear that Apple doesn’t intend for MobileMe to work the way they promise. MobileMe Push: About automatic syncing, a document on Apple’s support site, “resolves” the problem in which “changes made on your computer may not immediately sync to MobileMe” as follows:

Changes made on your computer will be synced to the MobileMe “cloud” once every 15 minutes (or every hour in Mac OS X 10.4.11).

Is it just me, or is there a cognitive disconnect here? Apple sells MobileMe as a solution in which “push happens automatically, instantly, and continuously,” but, of those three things, apparently “instantly” and “continuously” really mean “once every 15 minutes.”

I’ve purchased MobileMe for myself and for others based on Apple’s claims, only to find out that those assertions are blatantly false. That doesn’t just disappoint me; it makes me angry. Apple’s not just spinning the truth.

They’re lying about it.

Update (2008-07-14): The MobileMe Features page’s description of push now says,

MobileMe stores all your email, contacts, and calendars in the cloud and keeps them in sync across your iPhone, iPod touch, Mac, and PC. When you make a change in one place, MobileMe pushes the new information up to the cloud, then pushes the change down to your other devices. Choose a sync interval for your Mac or PC. On your iPhone and the web, sync happens continuously.

I’m glad that they changed the text to be accurate. I still wish that they had implemented MobileMe push as they originally described it, however, because that would have been very cool.

Update (2008-07-16): Apple has apologized and extended MobileMe subscriptions by 30 days.