Labnotes » Build Testing For The Rest Of Us

Nick Sieger, on testing your Rakefiles:

Perhaps someone out there will run with this idea and take up the challenge and write a Rakefile completely in a test-driven or behaviour-driven style. Itâ€™s always been a sore point for me with Make, Ant, Maven, and virtually every other build tool in existence that you have no other way of automatically verifying your build script is doing what you intended without manually running it and inspecting its output â€“ it just feels so dirty!

I don’t know of many people who actually test their builds and automated tasks. But I do know someone who manages to break the builds every so often. Like that time I experimented and accidentally erases the LICENSE file, and then made a release with an empty license. Oops. Or that time I moved stuff around and ended releasing a WAR that passed the integration tests, but still missed some critical files. Or that time the other day … well, you get the point.

So I started thinking, what would it look like if I tested the build file. And I started with the simplest thing that could possibly work:

check do
  Zip::ZipFile.open(package(:jar)) do |jar|
    fail "No MANIFEST.MF" unless jar.entries.include?("META-INF/MANIFEST.MF")
    license = jar.read("META-INF/LICENSE")
    fail "Empty license" unless license =~ /Apache License/
    classes = jar.entries.select { |entry| entry.to_s =~ /org/apache/ode/utils/.*class/ }
    fail "No classes" if classes.empty?
  end
end

That one is more defensive than foresight, it checks for problems I ran into self inflicted before, to make sure they won’t happen again. But it’s a good start.

If you used Ruby for any length of time you’ll immediately recognize two key characteristics of this code. It tests stuff. And it’s crap. A month from now I’ll want to add something else and look at the code and wonder what the hell it does. It’s write only. And that’s not good enough for Ruby.

I’m a big fan of RSpec, so I decided to write the same thing using RSpec and see what it would look like:

check do
   describe package(:jar) do
    it "should contain MANIFEST.MF" do
      package(:jar).should contain("META-INF/MANIFEST.MF")
    end
    if "should contain an Apache license file" do
      package(:jar).file("META-INF/LICENSE").should contain(/Apache License/)
    end
    it "should contain classes" do
      package(:jar).should contain("org/apache/ode/utils/*.class")
    end
  end
end

Much. Better. Now at least I know what I’m testing. Because, hate it as we may, tests must be maintained.

But it’s longer, and a few tests like this will quickly turn any build file into a haystack. There’s some redundancy. When you work with RSpec you’re writing test cases outside the code, so you need to organize them into logical units: contexts. We don’t need that here. We already have a context that happens when we build something. We don’t need to isolate it, set it up, or tear it down. So let’s get rid of unnecessary describe:

check do
  it "should contain MANIFEST.MF" do
    package(:jar).should contain("META-INF/MANIFEST.MF")
  end
  if "should contain an Apache license file" do
    package(:jar).file("META-INF/LICENSE").should contain(/Apache License/)
  end
  it "should contain classes" do
    package(:jar).should contain("org/apache/ode/utils/*.class")
  end
end

So we got rid of the contexts but not entirely, we still have something like that, only it happens to be the object we’re testing. I call them subjects. And we write expectations against the subject. So let’s separate the descriptive part, where we decide on the subject and say what it should, and the code that complains if it doesn’t:

check package(:jar), "should contain MANIFEST.MF" do
  it.should contain("META-INF/MANIFEST.MF")
end
check package(:jar).file("META-INF/LICENSE"), "should contain an Apache license file" do
  it.should contain(/Apache License/)
end
check package(:jar), "should contain classes" do
  it.should contain("org/apache/ode/utils/*.class")
end

You might recognize that we moved away from RSpec, which is perfectly fine, we’re testing the build not running unit tests in isolation from the code. But we are using the ever so sleek should and should_not, and the niceness of expectations and custom matchers.

What about turning the entire build file upside down and make it behavior-driven? I think that would work. But the key to testing is saying the same thing twice, once to make it happen and once to prove that it works. Flipping it around would change the syntax but prove nothing. So we do want the duplicity of code that builds and expectations that match.

If you’re using Buildr 1.2, you just got this cool feature (documented here). If you’re using Rake, I can’t imagine it would be too hard to rip the code and use it elsewhere. And if anyone is interested in getting this into the next Rake release, please do!