1. Introducing Buildr, or how we cured our Maven blues

    April 18th, 2007

    It’s no secret I have a love/hate relationship with Maven 2.0. I hate it. And I love using it as an example of how not to build software. Maven 2.0 is proof that good intentions, hard work and an open source community can still lead to all kinds of wrong.

    And apparently I’m not the only one who feels that way.

    It has come to my attention that you are responsible for the loss of many, many hours of my life, hours that could have been spent on more worthwhile endeavours but that now I will never get back. In addition, the resultant fluctuations in my blood pressure have likely reduced my life-expectancy by a corresponding amount.

    – The Fishbowl: An Open Letter to Maven 2

    The road to Maven started full of hope and promise, quickly adding one module after the other. But in less than a month the build escalated into a maintenance nightmare. I seriously could not believe spending a whole day on the trivial task of copying one file from here to there. Can you imagine it takes 34 lines of XML code to concatenate two SQL scripts into one? And that’s before we start using them to populate the test database.

    So everyone around me is wondering “what’s taking you so long?”, and I’m asking myself the same question. Did I finally meet my foe? The one technology that’s groundbreaking, forward thinking, and just beyond my grasp? Is it time to admit defeat and move on to something else. Maybe landscaping?

    But I stayed the course, got something half working, enough to get the other developers to start using the new build. I’m glad I did. Now that everyone else was in the same sinking boat, I no longer had to explain the unexplainable. I finally felt vindicated. It wasn’t me. I wasn’t a retard, just unfortunate enough to use the build system that couldn’t.

    We were already committed to make a release, and so the project went live, Maven and all. But we knew what will come next. The continuous build server reported random bugs, sometimes it would build, sometimes it won’t. Every few days, a developer would e-mail to let us know the build system doesn’t build. When it just did, a minute ago.

    We quickly dubbed it the Maven Uncertainty Principle, though in fairness we knew there was little uncertainty or doubt whether the build will work: it won’t.

    Almost everything I’ve ever downloaded that used maven for its build process, didn’t build.

    – Tomcat Scales to 16,000 Concurrent …

    Finally, one day we just had enough. The test cases stopped working. The surefire plugin surely did not fire. Even worse, a few days before I had to go and disable one of our critical test suites. Maven couldn’t deal with the fact that we excluded an old Log4J brought in by one of the dependencies, so we can then include the Log4J version we certify against. Perhaps Maven was telling us not to use test cases and we were too thick to listen.

    Either way, we had to choose. Agile or Maven. One of them will have to go.

    So we started imagining the happy life we could all live with Maven out of the way. Maybe we’ll switch back to Ant? For all its shortcomings, at least Ant builds work. Repeatedly. With certainty. You tell Ant what to do, and it does exactly that. No more guessing, hurting knuckles from crossed fingers.

    Sure, Ant is not “declarative”, the holy grail of software developers everywhere. But declarative done badly is worse than any imperative spaghetti code. Between the phases, goals, profiles, modules, transitive dependencies, configurations, other people’s POMs, plugins, build.xmls all over the place, and the funky reactor, how do you know what your build will end up doing? It requires running a lot of imperative code. In our head, and by trail and error builds for hours on end.

    With Ant, we just had to imagine and make it happen.

    Or maybe we should try something else? How about Rake, the Ruby equivalent of Make? It lets you declare tasks and handles the dependencies between them. It slices through file dependencies like a hot knife on butter. It even has a Make-like rules system. It uses Ruby which, we’re told, is more capable than the Ant mock-language. It has … get ready for this … mutable variables, functions and classes! We can use these to keep our code DRY and neat, write and reuse code. Heck, we could even write “plugins” with just a few lines of code

    So we did it. We started with Rake. Let me spoil the surprise for you. It turned out to be an excellent choice.

    Rake was a good start, but not what we’d hope for. The typical Java application we work on consists of several modules, all of which have the same common lifecycle tasks: compile, test, package, deploy. Writing those over and over for each and every module would make it not much better than Ant. There’s got to a a better way.

    They don’t just all have the same lifecycle tasks, they also have the same directory structure. So we used that to build a little framework that, by just pointing at a project could infer all the common tasks based on the files it finds. And for everything that deviates from conventions, we could just script with a few lines. There’s no escaping all these edge cases and special considerations, but with Ruby we can make them easy.

    Want to see some examples?

    Build me a JAR:

    define "utils" do
  compile.with COMMONS.logging, COMMONS.pool, LOG4J, XERCES, JAVAX.stream
  package :jar
end

    With XML beans generated from the schema, if you don’t mind:

    define "bpel-schemas" do
  compile_xml_beans "src/main/xsd", "src/main/xsd/*.xsdconfig"
  package :jar
end

    No, wait, I want to merge two SQL files I just generated, and build a database from that:

    derby_sql = concat("derby.sql"=>[engine_sql, quartz_sql])
derby_db = Derby.create(”derby/jpadb”=>derby_sql)

    The nice thing about the little example, is that it will build the database once, but regenerate it every time you chance one of the source files. There’s a free t-shirt for you if you can show me how to implement this in two lines of Maven or Ant code.

    Now, let’s compile, package and install all of that in the local repository, so we can use these modules in other projects:

    rake install

    That, in a nutshell, is how we went from the deeps of frustration to a build system that … hold on a second … is actually fun to use. And dead easy to customize. And works repeatedly.

    Say goodbye to frustrations and the Maven Uncertainty Principle. Stop arguing over declarative vs imperative, just have both.

    Go ahead, give it a try.

    Now I hate using the Beta tag, but let’s be clear, we’re a few short weeks from making a final release. We’re using it in several projects already, enjoying every minute of it. We’re happy, the continuous integration server is happy, life is good. But it’s still rough around the edges, and some bits will change before we hit 1.0. And the documentation is terse.

    So if you want to get started today, I recommend picking up an existing Rakefile (and check out some of the custom tasks). It’s a complex project, but it will show you the extent of what Buildr can do.

    Posted by Assaf Filed in ruby

    1. Lovable Lyle » Blog Archive » Buildr as a replacement for Maven

      April 19th, 2007 at 11:40 am

      [...] Arkin blogs about how his team finally gave up on Maven and replaced it with Buildr, a build tool written in [...]

    2. bokmann

      April 19th, 2007 at 2:27 pm

      Wow… This sounds beautiful. I loved Maven as of version 1, but when I started looking at Maven2 I just couldn’t believe how much they had changed everything. As if Maven1 wasn’t controversial enough - why fragment your community? Some really cool stuff is dirt simple (build reports, metrics, etc), but some common stuff is just… extreme.

      I have been doing a lot with capistrano, and am anxious to bring that knowledge to the Java community when I get a chance. I’m not working on any projects that would benefit from it at the moment; if you want to incorporate some other maven-like ideas (such as publishing to a repository, etc). Consider capistrano, and shoot me an email - I’d love to help.

    3. Tim O’Brien

      April 22nd, 2007 at 7:41 pm

      Interesting, well, rake certainly does have advantages. I use Maven, hell, I helped write the easy part of the first Maven book, but I, like you, tend to have an intense dislike for what it sometimes requires of me. For example, right now, I’m using the cobertura plugin to generate unit test coverage reports, but…..err…..guess what….The only way to use the cobertura plugin at the moment is to look at the source code for the Maven 2 plugin because the documentation is wrong. So, in other words, the only way you can really use maven is if you are comfortable reading the source code. Ugh.

      For the database build - I’ll give you a better idea. Just use Rails migrations. I’m doing this know for a J2EE application I’m working on. The DB is built with rails migrations - we never looked back, it works perfectly. Everything else builds with maven, with the exception of some customer Ant script for a very complex packaging task and (i think) we’re using the native build.

    4. Labnotes » Buildr, or when Ruby is faster than Java

      May 3rd, 2007 at 10:22 pm

      [...] started when Maven hit the fan, and we had enough of the clunky XML pseudo-code, unreliable builds and maintenance straight out of [...]

    5. ejboy

      May 5th, 2007 at 12:45 am

      We use Ant/Scriptella stack for projects builds and migrations. I’ve used maven several times, but it’s just a waste of time …

      Scriptella differs from Rails Migrations in its declarative, SQL-centric approach, at the same time you have all the benefits of an ETL (Extract-Tranform-Load) tool: importing data from CSV, XML, working with several data sources (RDBMS, LDAP, JNDI) and many more.

    6. TooBad4You

      May 7th, 2007 at 10:00 am

      Looks like someone doesn’t know how to generalize a build and use best practices…

    7. Adam Fisk

      May 7th, 2007 at 12:28 pm

      I have to agree with the last comment. If your tests are failing with the kind of frequency you’re talking about, I’d place a bet on your tests being the culprit. Maven and ant aren’t so different under the hood for the simple stuff like that — it’s all just javac at the bottom folks.

      I’m also wary of the answer being to move on to the next build system with a lot of buzz behind it. Now there’s a great way to waste a lot of time on a project — continually moving between whatever build system people are talking about on the blogs these days. Guaranteed each one has its own issues. If your tests fail, it doesn’t matter what build system you use.

      I personally like maven quite a bit. It’s frustrating at times for sure, but writing plugins and configuring it is a breeze once you get the hang of it. Sure, the documentation sucks, but that’s generally the case in many projects. “Agile or Maven” is just a ridiculous question.

    8. Assaf

      May 7th, 2007 at 12:53 pm

      When you can’t build test cases, you obviously can’t run them and that’s a problem. Switching to Buildr made all these problems go away.

      We’ve been using Maven 2.0 for over a year now, perhaps that’s not enough time to take the beast. I accept that. But a build system that requires so much energy is just not worth using.

    9. monkinetic | Blog Archive » I hate RDoc

      May 7th, 2007 at 1:28 pm

      [...] a Googlesearch finds the introductory blog post, which Tim did point to and which is slightly useful. At least the authors point to some rake files [...]

    10. Greg Wilson

      May 8th, 2007 at 4:55 am

      I’ve been arguing for years that reliance on “declarative” configuration files is holding a lot of tools back, because there is effectively no way to debug them when they go wrong — there’s no real equivalent of setting a breakpoint or single-stepping. I think this is why tools like CONS (the original, in Perl), SCons (Python), and Rake (Ruby) are going to win.

      Now, if only we could foster a similar revolution against httpd.conf, config.xml, and everything else my web stack relies on… :-(

    11. Agylen » Introducing Buildr

      May 8th, 2007 at 5:53 am

      [...] 8 May 2007 Introducing Buildr Posted by ugo under devel , java(bookmark this)   Assaf Arkin: “That, in a nutshell, is how we went from the deeps of frustration to a build system that [...]

    12. monkinetic | Blog Archive » Structured Docs v. The Blank White Sheet

      May 8th, 2007 at 8:58 am

      [...] hoping I can wrap up this little rant over Buildr’s RDocs. I just left a comment over on Assaf’s post, Patience, Buildr docs coming up, [...]

    13. David Bolton Strikes Again » Blog Archive » O’Reilly Radar, Buildr, Mingle & “The Corporate World”

      May 9th, 2007 at 11:18 pm

      [...] such as Buildr and Mingle are the first steps towards corporate acceptance.  Ruby will first start at the [...]

    14. Carlos Sanchez

      May 20th, 2007 at 10:02 pm

      Maven couldn’t deal with the fact that we excluded an old Log4J brought in by one of the dependencies, so we can then include the Log4J version we certify against. Perhaps Maven was telling us not to use test cases and we were too thick to listen.

      sounds like you used exclusions to force dependency versions, which is not the right way to do it. If you explicitly depend on another version maven will resolve to that one.

    15. Assaf

      May 21st, 2007 at 10:01 am

      Carlos, true, the easy case of forcing a new version is easy. Except we needed to exclude it in one context, so the dependency doesn’t propagate, and include it back from another context (strictly for testing). The not so easy case was impossible.

      And I’m sure everyone realizes we didn’t dump Maven on account of a couple of issues we had that could be resolved by a second reading of the docs. But I’m writing a blog post, keeping to a few short examples; if I wanted to publish a “Maven Beware!” book, I would have enough material for that with (non)-working code samples for you to try out.

    16. Jean-Baptiste Quenot

      June 4th, 2007 at 6:31 am

      Interesting, but just curious, how do you manage the dependency resolution?

    17. Kris Bravo

      June 12th, 2007 at 11:15 am

      Given that v. 2.0 of Maven was a total rewrite, and I don’t see your name in the codebase anywhere, isn’t it a little crazy to be hopping around build processes like a frog on pads? The codehaus project will accept just about any plugin you write, including, in the case of your sql script woes, a file concatenator. But you didn’t write it - in java, the codebase you are developing in - instead you switched build processes?

      I personally got dragged into Maven 1.x land. It’s horrible - I’d never allow anyone to stay on it and recently moved a large application for a client from it TO Maven 2. There are issues - for example, version 2.0.6 fails to add ejb-client jar’s to the classpath properly in multiproject builds. And scripting is pushed aside (thank you!) to assert the notion of standardization over customization.

      You balk at the values of Maven, but I’d like to hear your opinion on the maintainability of your Rake build once applied to a large project & without the artifacts of your lessons from Maven. I believe you may have quietly adopted some principals from maven you will take with you, even while you are berating that projects efforts.

    18. Assaf

      June 12th, 2007 at 11:28 am

      Kris,

      I am not a Maven 2.0 contributor. I do have a way to distribute plugins, just not the time to bother with a complex development cycle for something that can be done in one, maybe two lines of Ruby code.

      This post was written after we converted about a dozen projects over to Buildr, some of which qualify as large and complicated. I believe anything that requires 5,400 lines of Maven/Ant code to build (check out Ode), qualifies.

      Our experience so far? It rocks. We spend the last amount of time maintaining it, no more broken builds, clunky XML or insurmountable bugs.

      We adopted some principles from Maven, some principles from Apache, some principles from Ant, some principles from Rake. We just did it in a way that works and takes the pain out of builds.

    19. drehorgelmann

      June 27th, 2007 at 9:42 am

      I’m sorry, but your “trust me it doesn’t work” attitude is not really convincing. How about actually showing a few examples of non-working Maven pom.xml-s?

    20. Georges

      June 27th, 2007 at 2:04 pm

      Well, I have moved Ant / make scripts into Maven 1 a couple of years back. It was hard but did work really well. The project had 40 modules - both Java, C++.
      Last year I helped a big telecom company move a 60 module Ant build to Maven2. Again - it was hard but well worth the effort. We reduced developer setup time from 2 days to 2 hours. It contained all sorts of different packagings - jars, wars etc. Generated stub/skeletons for Corba and web services. Combined with CruiseControl is served as a backbone for developers in 4 different countries and really helped us achieve our agile goals.
      The last 6 months I have used Maven again - this time together with Continuum. Works like a charm. The nice thing was that when joining the project a knew my way around the modules and source trees.
      Great work from the Maven guys!!!

    21. prozz’s blog » Blog Archive » Ruby researches

      July 5th, 2007 at 4:04 am

      [...] plus Ruby expressiveness. I think it’s not just yet another build tool Check for evangelism here and [...]

    22. JavaSPEKTRUM Blogosphäre » Blog Archiv » Blogosphäre (aus JavaSPEKTRUM 04/07)

      July 9th, 2007 at 1:35 pm

      [...] Alternative gesucht – und diese auf Basis von Rake, dem Buildwerkzeug für Ruby, unter dem Namen Buildr realisiert. Rake ist eine in Ruby implementierte “Internal DSL”, also eine [...]

    23. IT-eye » Maven, Ant, Buildr, or what?

      August 18th, 2007 at 7:35 am

      [...] But recently I’ve noticed projects moving away from Maven. Some people complain that it’s too hard to do non standard stuff. It supposedly it’s also pretty hard to debug when your build isn’t working: Introducing buildr or how we cured our maven blues. [...]

    24. Xanthros

      January 18th, 2008 at 3:55 pm

      This is the best maven user guide I’ve seen.

      http://www.sonatype.com/book/maven-user-guide.pdf

      @Tim O’Brien

      “The only way to use the cobertura plugin at the moment is to look at the source code for the Maven 2 plugin because the documentation is wrong. So, in other words, the only way you can really use maven is if you are comfortable reading the source code. Ugh.”

      This is a completely idiotic statement. Maven is based upon plugin architecture. Maven *ISN’T* to blame for a bad plugin or that plugin’s documentation. So get your story straight. The documentation for Cobertura is terrible not Maven. Not to mention that the Cobertura plugin is *open source*. If you found discrepancies in the docs then submit those and get them fixed! But I guess it is easier for some of us to just complain!

    25. Tomohawk

      February 14th, 2008 at 6:29 pm

      Completely agree with unreliability/complexity of Maven2. Tried it. Went back to ant, with maven antlib. Allows us to interact with artifact repositories very easily from ant. Much more reliable than straight maven for exactly the same tasks.

      Another key problem with maven is dual use of pom file as both an artifact descriptor and a build script. I guess they never heard of separation of concerns? We now generate poms with ant, and poms are pure descriptors only.

      Definately agree that xml is a lousy scripting language. Looking forward to giving buildr a try.

    Leave a Reply | Trackback | Track with co.mments

    Where's my comment? I get too much comment spam, so I have to moderate comments. Damn those spammers. If you don't see your comment immediately, be patient. I'll approve it the minute I see it. Want to know when your comment shows up, or check if anyone responded? Track it.

    Or using OpenID