Twitter GitHub Facebook Instagram dirv.me

Daniel Irvine on building software

From C# to Ruby: Test assertions in loops

22 May 2014

A unit test pattern I’ve used in the past with C# and NUnit has proved to be problematic with Ruby and RSpec. Imagine you have a method which takes an integer and returns a boolean:

bool FuncUnderTest(int)

If we have a set of values which cause this function to return true and a set of values which cause this function to return false, then we can write a couple of tests like this:

[Test]
public void returns_true()
{
  int[] truths = // ... some values which will return true 
                 //     when passed to FuncUnderTest

  foreach( var truth in truths )
    Assert.IsTrue( 
      FuncUnderTest(truth), 
      string.Format(FuncUnderTest failed for {0}, truth) );
}

[Test]
public void returns_false()
{
  int[] falsehoods = // ... some values which will return false
                     //     when passed to FuncUnderTest

  foreach( var falsehood in falsehoods )
    Assert.IsFalse( 
      FuncUnderTest(falsehood), 
      string.Format(FuncUnderTest failed for {0}, falsehood) );
}

The key point here is that we are asserting within a loop, so that although only one Assert statement appears in code, actually a multitude of assertions will be checked.

Crucially, both Asserts are called with additional message text explaining the assertion failure. Without this, the assertion does not carry enough information for a tester to pinpoint the exact error in the event of test failure. If only one input value fails, the explanatory text appears in the test output to notify the tester which value caused failure. So it’s critically important to use that text in your tests.

Unfortunately this antipattern becomes more apparent when used with a specification-style test framework like RSpec. RSpec’s expectations generally don’t make use of message text (although possible, it’s very rarely used in practice).

# warning: antipattern, don't use
it returns true for all truths do
  TRUTHS.each |t| do
    func_under_test?(t).should eq true
  end
end

So why is this an anti-pattern?

  • It’s non-obvious, requiring the tester to know what to look for in the test output. In other words, it’s never going to produce a standard test failure and that will introduce confusion at some future time. If you’re new to the codebase, you won’t know to look for this. Put another way: You made me think. Don’t make me think.
  • It’s all too easy to forget the explanatory message, and your test framework won’t know to warn you should you forget to write it out.
  • Testing a large set of values is probably not even necessary. This is a design issue, not a nitty-gritty technical detail. You only need as many values as are required for you to fully write the feature under test, with complete coverage. So in the example above, a few key test values may have sufficed, each with its own assert statement.
  • Assert.IsTrue( FuncUnderTest(keyValue1) );
    Assert.IsTrue( FuncUnderTest(keyValue2) );
    Assert.IsTrue( FuncUnderTest(keyValue3) );
    

    In a traditional unit testing framework like NUnit, writing explanatory message text is normal and so this antipattern is generally acceptable. It’s only through using RSpec that I’ve realised how bad this is. Even though NUnit does a good job of masking the issue, it’s still an antipattern--regardless of framework or language--and I’ll be avoiding it in future.

    About the author

    Daniel Irvine is a software craftsman at 8th Light, based in London. These days he prefers to code in Clojure and Ruby, despite having been a C++ and C# developer for the majority of his career.

    For a longer bio please see danielirvine.com. To contact Daniel, send a tweet to @d_ir or use the comments section below.

    Twitter GitHub Facebook Instagram dirv.me