1. Feb 8th, 2007

    Timing is everything

    Time waits for no one

    Justin Gehtland over at Relevance has a post about large number performance, Ruby vs Java. How quickly can you calculate 2^2^16 in either one. So I went there, added my 2 cents, and followed the comments stream.

    Shortly after, two readers chimed in, measuring performance using time. As in:

    time ruby test.rb
time java Test

    A part of me was screaming inside: “You’re not measuring the calculation! You’re measuring the cost of loading the runtime engine and library! You’re looking at the wrong thing!”

    Understand this. You can’t just measure Java with time. You need to wait for the JVM to fire up and load the classpath, then you run a few iterations, depending on whether you’re running -client or -server, to let the JIT warm up, and then, and only then, you hit start on the timer.

    That’s how you measure things.

    But I got distracted by so many other things, I forgot to tell them that. And good thing I didn’t. Because, between then and now a few things happened that made me change my mind.

    What’s important is not how long it takes the JIT to find its bearings, what’s important is how long it takes to complete the job. Period.

    Every time I run Ant, I have to wait for it to complete. It doesn’t matter that Ant gets incrementally faster in each step, I guess if I wait long enough it will start doing quantum computing. I’m still going to wait five minutes for the build to finish. Five minutes of sitting there and doing nothing.

    It doesn’t matter that in production the server will get the code to run 5x faster, I’m still going to wait a few minutes between checking a change and seeing the code crash, so I can try to narrow down on the bug, repeated several times over the next hour.

    And it doesn’t matter that Java can calculate that big number 10x faster than Ruby if it’s going to take me five minutes to write that program in Java, where all I need to do is:

    ruby -e 'puts 2**2**2**2**2'

    There’s a reason why Java became such a hard language to work with, why the development cycle is so painfully slow, and why it’s all Big Design Up Front. It lives and dies by the JIT. Everything is measured and optimized for the JIT. Not the user, not the developer.

    So from now on, I’m going back to my trusty time. No more warm ups, no more waiting. I’m measuring from the second I start waiting until it’s done.

    PS: I’m not complaining about the benefit of a JIT that can speed up critical section code: that’s one of the better features of Java, and hopefully will translate well to JRuby. I’m criticizing the methodology of discounting real wait time to get more impressive performance numbers, and any design mistakes the follow from measuring the wrong thing.

    Photo by lukasd2009

    Posted by Assaf Filed in general

    1. Feb 8th, 2007

      Michael Schuerig

      Assaf, thanks for not screaming. I’m the first of those people who wrote a comment using time to compare the execution times of Ruby vs. Java. My only point was to show how very pointless this comparison is. Ah, well, there was another point. If you can get past the comment formatting, you’ll notice that the Ruby version of the code is plainly understandable. In contrast, the Java version is three times as long, almost inscrutable, and requires explicit use of BigInteger, something that Ruby does automatically.

    2. Feb 8th, 2007

      cyu

      I agree with criticizing design mistakes from measuring the wrong thing, but wouldn’t you be making the same mistake if you’re blindly including the startup times in your measurements if startup only amounts to a small portion of your application’s lifecycle?

      Including startup time when measuring development performance is one thing, but I don’t know about criticizing the discounting of it from a *design* perspective.

    3. Feb 8th, 2007

      Charles Oliver Nutter

      However, there’s one other aspect you’re ignoring: a large part of the world’s computing is not done in response to a single command, it’s in a long-running process where startup time pales compared to the runtime of the application as a whole. Yes, it’s probably quicker to use something like Ruby to run a single command at a command line, but it’s certainly faster to run a long-running process in Java. It’s all a matter of tradeoffs; to say that either is always the correct answer is foolhardy.

      And also…if your Java-based command runs in five seconds plus two seconds to warm up while your Ruby-based command runs in ten, you do get more done with Java in less time. And the vast majority of computing problems will dwarf startup time with their longer runs.

    4. Feb 8th, 2007

      Assaf

      This is not a competition to decide which language is “faster”. I’m not trying to distort time/space to prove Ruby beats Java.

      I’m trying to write better software by changing what I measure. By looking at how to waste less of the user’s time, using whatever tool works best for each specific problem.

      There’s a benchmarking doctrine, very strong in the Java world, of only measuring code in a way that would prove Java is as fast as C++. Or was it C? I forget. That’s misguided. The design should focus on one thing: saving people’s time and energy.

      Charles, of the two commands, the one that takes seven seconds is faster than the one that takes ten. It don’t care that it’s in Java, and I don’t care how that time is split internally between warm up and everything else, it’s still faster. And that’s exactly what time will tell me.

    5. Feb 9th, 2007

      Chipping the web - Rosa Parks Highway — Chip’s Quips

      [...] goes for timing strategies, [...]

    6. Feb 10th, 2007

      SOB: Scion Of Backronymics » Java benchmarks, enterprisey apps, and the future of Ruby (is Smalltalk)

      [...] over at Labnotes recently wrote that timing is everything. I read that a couple days ago when it was first posted, but the discussion in comments has [...]

    7. Feb 10th, 2007

      dnl2ba

      Users of my company’s web service don’t notice the JVM startup time. It started long before they visited. They notice how long it takes to serve pages, and they’ll notice if the server got pwned because it couldn’t scale serve to millions of users.

      Would you argue that Java and Ruby are for the tasks because they’re both programming languages?

    8. Feb 10th, 2007

      dnl2ba

      Drat. “Are for the *same* tasks” is obviously what I meant.

    9. Feb 10th, 2007

      Labnotes » How well do you know Java?

      [...] recent post, Timing is everything, started a mini-controversy. One that can teach you a lot about where Java is coming from, and [...]

    10. Feb 10th, 2007

      Assaf

      dnl2ba,

      How slow exactly is the Java startup time?

      Measure it. You’re in for a surprise.

      My point was that, in the Java world, we’re trying to look at the language through a special lens intent to make it competitive with C/C++. We do that at the expense of looking at real world requirements, we hold on to believes instead of fact checking.

      And to all who commented, why not run a little benchmark and note the numbers. You’re trying to justify Java, where in fact there’s nothing to apologize for. It’s plenty fast.

      How well do you know Java?

    11. Feb 10th, 2007

      dnl2ba

      My point was that it doesn’t matter if it takes little or lots of time to start if the start time isn’t infringing on the user interaction.

      It’s different when the program runs on the user’s machine. Firefox lurches to a stop for 4-8 seconds while a Java applet fires up, and I find myself avoiding sites that have them.

    12. Feb 10th, 2007

      dnl2ba

      I should add, after reading your most recent post, that I (as a user) don’t have to distinguish between a Java plugin and the JVM itself. I see “Java” on the screen while my browser is frozen for up to 10 seconds. The experience sucks.

    13. Feb 10th, 2007

      Assaf

      dnl2ba,

      Applets are all kinds of wrong. But nobody, well, nobody I know of nor any site I use, uses them, so I’m going to preted they don’t exist. For the record, I’m talking about Java 1.5 or 1.6 running outside the browser. I develop with either one.

      As for startup time …

      If the only thing you measure is the individual HTTP request/response handling, you’re going to optimize for that alone.

      If what you measure is the overall picture, which includes development, testing, maintenance, iterating with users to add/improve features (aka Agile), you’re going to optimize for that.

      You wonder why so many people are moving from Java to Ruby? One of the reasons is that the Java world is obsessed with optimizing for the individual HTTP request/response cycle, instead of looking at the big picture.

      Read my post again. Notice how all the use cases I put there deal with writing — not running — code.

    14. Feb 11th, 2007

      dnl2ba

      Right, and on that, I’m with you 100%.

      Time = bugs = time. If you have to write more code to get something done, you have that many more opportunities to botch something. Then you’ve got a bug, and then someone has to notice it, and then you have to find it, and then you have to fix it.

      Typically, code spends much more time running than being written, so that’s another reason for leaning toward optimization. But then you start adding in maintenance and feature addition, and the total opportunities for making more bugs proliferate. There’s that quote about premature optimization…

    Your comment, here ⇓

    Or using OpenID