1. Feb 10th, 2007

    How well do you know Java?

    My recent post, Timing is everything, started a mini-controversy. One that can teach you a lot about where Java is coming from, and where it’s heading.

    Smart readers pointed out that Java’s raw performance will more than make up for any cost of startup time. Others pointed that so many applications are started once and left running for years. Those are all good and true.

    So let me ask you this question: how well do you know Java?

    I didn’t write the post to claim Java has a slow startup time, I wrote it to propose a different way of measuring things that will include the startup time. And in doing so I hit a few nerves.

    So let’s take a break and look at the numbers for a second.

    I’m measuring this on a 2GHz Duo Core 2, running Linux. The startup time for the JVM (Java 1.5) is 30x slower than the startup time for the Ruby interpreter (version 1.8.5). 30x is a freaking large number. But remember from my post that I wasn’t looking for relative numbers, but how long I, the user, am going to wait. How long?

    About 0.1 seconds.

    As fast as a blink.

    This is 2006. In ’96 the startup time for the JVM was a serious issue you had to work around. Those days are long gone.

    There’s no need to propose theories that explain and rationalize the incredibly sluggish startup time that never was. Java does not need to be excused.

    Proving this is just a matter of firing up the command line and running:

    time java Test
time ruby test.rb

    Of course, it uses time. And people who are caught in the slow startup myth, have a subconscious gag reflex when it comes to time. It’s amazing how myths can play with your head.

    I ran this little test at home, so I knew Java didn’t have to worry about its startup speed. It’s more than adequate.

    If your initial thoughts were to rationalize the supposedly poor startup time of Java, my apology for not laying out all the facts. But consider this. You may have internalized limitations of the language that no longer exist. Your knowledge of Java might be burdened with believes and myths, instead of facts.

    In which case that post was written for you. Software adapts, hardware improves, requirements change. And every once in a while, we need to step back, forget all that we believe is true, and look at the situation with a fresh eye and an open mind. To judge that which is true now.

    The point of my post was not that Java sucks and Ruby rocks. The point of my post is that, in the Java world, we still build software like it’s 1996. It’s time we let go of old believes and move on.

    Printable PDF | Posted by Assaf Filed in general 6 Comments »

    1. Feb 10th, 2007

      http://openid.claimid.com/cyu

      Here’s the problem with doing the measurement this way. Say you benchmark some calculation in Java that took 0.1 seconds, bringing the total real time to 0.2 seconds (startup time + calculation time). Not so bad, but what happens when you bring that calculation in your server in loop 1000 times, or is requested 1000 times simultaneously? Using the real wait time measurement as your basis, you would estimate that the calculation executed in 200 seconds, when in reality it is probably about 100 seconds. Then the measurement becomes significant. So then you could ask, ‘why not just include the looping into benchmark?’ – but that’s not what you should do when benchmarking, you should only test the code you are concerned about, and in this case it is the calculation.

      I don’t see this as an issue of Ruby v. Java, as I can make the same statement above about Ruby and it would still apply (the number just wouldn’t be as significant). The issue is your criticism of a very valid benchmarking technique, and your statement that using it would lead to poor design decisions.

    2. Feb 11th, 2007

      Assaf

      cyu,

      Ruby is not immune from measuring the wrong thing, and I think there’s enough material for future posts on that.

      But Java has a particular obsession, a doctrine for only measuring that which was JITted, probably dating back to when Java was fighting the performance war against C++. Right now, for most applications, it’s fast enough that we can start worrying about other things.

      Some people don’t think the “waiting for the JIT” doctrine is a problem. But hopefully enough people will recognize it and do something to fix it. Before Java ends up executing as fast as C++, and being as painful to work with as C++.

    3. Feb 11th, 2007

      apotheon

      The actual VM startup time isn’t the only thing at issue here, though. Java suffers (or benefits, if you’re a Java-head) from a number of phenomena in the way it’s implemented that significantly affect performance over the lifetime of a process. The startup time of the VM is one factor: the optimizing capabilities of the VM are another. Finally, there’s the persistence of the VM environment.

      This adds up to a big (in terms of performance benchmarks) difference between short-run application performance and long-run application performance for Java. If you think MS Office is slow when you just want to quickly edit a document and shut down again, you should try writing it in Java and try that again. Perl actually outperforms Java in a significant number of cases for short-run benchmarks — Perl, which actually does its parse-tree compilation (which is a more-than-single-pass compiler, by the way) at runtime! Count the startup time of both languages in a several second computation benchmark, and Perl will probably win. On the other hand, if you’re running a server process that churns along for months at a time, and measure its performance on day three, it will rival roughly identical code in C++ for speed (though anyone claiming it matches C doesn’t understand the differences between C and C++, or writes incredibly bad spaghetti C).

      Not only does measuring at the beginning capture VM startup time — it also measures unoptimized code. Java is a bit like vehicle with a big engine and a huge flywheel: it doesn’t accelerate worth a damn, but it sure does burn up road once it gets up to speed.

      In general, I agree that the actual startup time of the JVM is trivial, and mostly ignorable — and thus, might as well be measured. Sure, it can prove significant if you’re measuring something so brief that a 0.1 second startup is problematic — or more, on an older machine (which most of us are using, believe it or not, as I know from personal experience since a JVM would start up somewhat slower on this 1600MHz AMD than on your Core Duo 2). That’s of little import when measuring actual human wait time, though, as you point out. The real problem is the code optimization, which doesn’t happen instantly. Let it run through its paces a few dozen times, and it’s much snappier.

      Basically, I guess what I’m saying is that it’s not really that simple. You just have to measure all the parts that are relevant to your purpose, adjusted every time your purpose changes. Saying Ruby does some calculation faster (or slower) than Java requires context to be worth crap, simply because of the strange (as compared with other languages) way that Java works.

    4. Feb 11th, 2007

      Assaf

      Apotheon,

      The narrow focus on only measuring performance in the app server is the problem. Let’s drop desktop apps and Applets, because Java is not even a contender there.

      Java’s worst enemy is not Ruby or Perl or C#; Java’s worst enemy is Java.

      If I can write a simple task in Ruby in five minutes, that will take 10 seconds to execute. And I can write the same task in Java in two hours, that will take 5 seconds to execute. Which one is faster?

      There’s the holisitic view to measuring performance. And then there’s the beauty contest view to measuring performance, where you only show the flattering profile.

      Now, someone may say apples vs oranges. But that’s real life. Software is about automating things, at least the kind of software I write about, and what you gain from automation comes after the cost it takes to create it.

      If some tool allows me to automate more things, cheaper and better, I’d use it.

      The beauty contest approach to measuring performance made Java incredibly fast in a very narrow context, but incredibly slow in so many other tasks a language should perform.

      I don’t think people would even bother to respond to my post if Java wasn’t, right now, on the defensive from other languages that are eating its lunch, percisely because it optimized the wrong thing.

    5. Feb 12th, 2007

      http://dvae.livejournal.com/

      If you use a more recent version of java, then it’s start up is faster than ruby.

      At least that is what Charles Nutter found with these numbers:

      $ time java -cp lib/jruby.jar:lib/asm-2.2.2.jar:. fib_recursive
      5702887
      real 0m8.126s
      user 0m7.632s
      sys 0m0.208s

      $ time ruby fib_recursive.rb
      5702887
      real 0m14.649s
      user 0m12.945s
      sys 0m1.480s

      Full (long) blog post on jruby JIT: http://headius.blogspot.com/2007/01/jruby-compiler-in-trunk-and-ready-to.html

    6. Feb 12th, 2007

      Refresh Your Mind – Hendy Irawan

      [...] from Assaf: … every once in a while, we need to step back, forget all that we believe is true, and look [...]

    Your comment, here ⇓