evanseries.org

I tweeted a few days ago about how long it took Ruby 1.8.6, Ruby 1.9.1, and MacRuby 0.3 to sample 10,000 words from a probability distribution. Then, yesterday, Charles Nutter (one of the lead developers on JRuby) replied to my tweet and asked if I could send the script along so that he could try it in JRuby.

Well, after a day or so of talking back-and-forth with him about it, I’m happy to present the results:

Experiment Details

These tests were run on my 15″ MacBook Pro (MacBookPro3,1) with a 2.4 GHz Intel Core 2 Duo processor and 4 GB of RAM, running Mac OS X 10.5.6. During script execution, Terminal was the only application running (as indicated by the Dock and menu bar).

For each implementation, the script was asked to sample 10,000 words from a list of words and their associated probabilities (which was contained in a file). This task was performed 100 times, and the time it took to execute the task was measured by Ruby’s benchmark library.

The sampling script and the word probability distribution file are available as a gist. (If you distribute the word probability distribution file further, the professor from whom I received it would appreciate it if you’d mention that it came from Carnegie Mellon University’s Language and Statistics class.)

A text file containing the timings for each implementation is also available.

The script was evaluated with JRuby 1.1.6 using Apple’s Java 1.6.0_07, with the 64-bit server VM (build 1.6.0_07-b06-57, mixed mode). The version string for this configuration is jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [x86_64-java].

I built the version of Ruby 1.9.1 used in the experiment on my machine, using the build tools that come with the iPhone SDK for iPhone OS 2.2.1 (19M2621A). (Ruby was compiled for Mac OS X Intel; the iPhone SDK for iPhone OS 2.2.1 is just the version of Apple’s developer tools that I had installed when I built it.) The version string for this configuration is ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-darwin9.6.0].

The version of Ruby 1.8.6 used in the experiment is the version included with Mac OS X 10.5.6 with all available updates installed. Its version string is ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0].

The MacRuby implementation used in this experiment was compiled from the testing branch of MacRuby on January 31, 2009 at 2:44 PM EST, using the same development environment as was used to build Ruby 1.9.1. The version string for the MacRuby configuration is MacRuby version 0.4 (ruby 1.9.0 2008-06-03) [universal-darwin9.0, x86_64].

Discussion

It’s clear that JRuby 1.1.6 is the fastest, taking a median of 1.9805 seconds. Ruby 1.9.1 clocks in more slowly, with a median of 3.29 seconds. Ruby 1.8.6 is slower still, requiring a median of 5.465 seconds to complete. MacRuby’s testing branch is considerably slower than the other three, completing the task in a median of 20.455 seconds.

If we take MacRuby out of the equation, we get an even better picture of the difference between the three fast implementations:

In this view, an interesting pattern emerges. Ruby 1.9.1, built on the YARV/KRI bytecode implementation, is indeed faster than Ruby 1.8.6, as one would expect. In fact, it’s substantially faster: the median time for Ruby 1.9.1 is only 60.2% of the median time for Ruby 1.8.6. And the slowest non-outlier Ruby 1.9.1 time (4.310 s) is still faster than the fastest Ruby 1.8.6 time (4.56 s).

But what I found surprising is that JRuby 1.1.6 is faster than Ruby 1.9.1 in the same way that Ruby 1.9.1 is faster than Ruby 1.8.6. JRuby’s median time is also 60.2% of Ruby 1.9.1’s median time, just as Ruby 1.9.1’s median time was as compared to Ruby 1.8.6, and JRuby’s slowest non-outlier time (2.1590 s) is also still faster than Ruby 1.9.1’s fastest time (2.21 s).

I’m hesitant to extrapolate these results much beyond the sample domain and the present time. It’s obvious that MacRuby has a lot of work to do if it’s to become as fast as the other three at these kinds of tasks, but it’s still in the early stages of development, so a lot of work is to be expected. In the same vein, Ruby 1.9.1 is a milestone on the way to Ruby 2.0, but it’s not there yet, so perhaps some speedups are on the way.

But if you’re looking for a fast Ruby implementation, at least as far as random word sampling is concerned, my data and I can recommend JRuby to you as the speediest solution of the four.

Update (February 5^th, 2009)

I just finished running the word sampling script in the most recent nightly build of JRuby available, and it’s even faster than JRuby 1.1.6:

The median time for the nightly build, 1.724 seconds, is only slightly slower than 1.7200 seconds, the fastest time for the stable build. In addition, the nightly build’s median time is 87% of the stable build’s median time.

From these results, it’s fair to say that the JRuby team certainly isn’t standing still when it comes to performance, either.