Chapter 19. EUnit and Test-Driven Development

As you are writing a program, how do you understand how the program will behave? You might have a model in your mind of what the program will do, but you can be sure of it only when you exercise or interact with your program in some way. Chapter 18 showed you how you can use –spec to express what you think the input and output types of a function should be; TypEr can check whether this is consistent with the code itself.

Types don’t tell you how a program behaves, however, and testing is one of the best ways to understand how your code will function. We have been doing this informally throughout the book; each time we have given some definitions, we have immediately gone to the Erlang shell and tried them out in practice. When you’re developing in Erlang your coding and test cycles tend to be small. You write a few functions, and you test them. You add a few more, and you test them again. Repeating all of the tests in the shell every time becomes both time-consuming and error-prone.

In this chapter, we’ll introduce the EUnit tool, which gives you a framework for unit testing in Erlang. We’ll show how it is used in practice, as well as discuss how it supports what is known in software engineering circles as test-driven development.

Test-Driven Development

The waterfall model of software development saw a software system being developed in a series of steps: first, the requirements of the system would be elucidated; on the basis of this, the system would be designed, and only once this was complete would the implementation begin. No wonder so many software projects failed to deliver what the customer wanted! Needless to say, the waterfall model is as disastrous in Erlang as it is in any other programming language.

A test-driven approach turns this on its head. Implementation begins on day one, but with a focus: each feature that is to be implemented is first characterized by a set of tests. This increment is added to the code and is accepted only if the tests pass. At the same time, this should not invalidate any earlier test; regression tests must also pass.

The implementer benefits from such an approach because she has a clear target at each stage: pass the tests. The customer is also in the happy position of being able to interact with or exercise the system continually: each time the evolving system builds, he can see whether it behaves as it ought to.

Test-driven development (TDD) has been associated with agile programming, but it would be wrong to identify the two. An informal test-driven approach has characterized functional programming since the early LISP systems: most functional programming systems have at their top level a read-evaluate-print loop, which encourages a test- or example-based approach. Erlang comes out of that tradition, with its roots in functional programming and Prolog, and the examples of interactions in the Erlang shell show this informal test-driven approach in practice.

Test-driven development is not confined to the single-person project. There is anecdotal evidence from industry that TDD can be effective in larger software projects, too, principally because it means the development team engages with potential problems earlier, and in the process develops more comprehensive test suites than it would with a more traditional approach. In turn, this leads to better software being delivered. It is needless to say that Erlang and TDD go hand in hand. In the remainder of this chapter, we will cover the EUnit system, which provides support for a more formal test-driven approach, and which is included in the standard Erlang distribution.

EUnit

EUnit provides a framework for defining and executing unit tests, which test that a particular program unit—in Erlang, a function or collection of functions—behaves as expected. The framework gives a representation of tests of a variety of different kinds, and a set of macros which simplify the way EUnit tests can be written.

For a function without side effects, a test will typically look at whether the input/output behavior is as expected. Additionally, it may test whether the function raises an exception (only) when required.

Functions that have side effects require a more complex infrastructure. Side effects include operations that might affect persistent data structures such as ETS or Dets tables, or indeed operating system structures such as files and I/O, as well as operations that contain message passing among concurrent processes. The infrastructure needed to test these programs includes the following:

  • Testing side-effecting programs typically requires some setup and initial modification of the data before checking the behavior of a particular operation. This needs to be followed by a cleanup of the program state.

  • Testing units within a concurrent program typically requires a test rig in which some mock objects or stubs are written to stand in the place of other components expected to interact with the unit under test.

EUnit supports testing side-effecting and concurrent programs. Let’s start with the simplest case of functional testing.

How to Use EUnit

EUnit provides a detailed representation of tests, as well as a layer of macros making tests easier to write. It also provides a mechanism for gathering all the tests in a module, executing them all, and providing a report on the results. For a module to use EUnit it needs to include the eunit library:

-include_lib("eunit/include/eunit.hrl").

In EUnit, a single test is identified by a function named name_test; a test-generating function will be named name_test_ (i.e., with an _ at the end). We define test-generating functions in the section The EUnit Infrastructure.

With the EUnit header file included, compilation of the module (mod, say) will result in a function test/0 being exported, and all the tests within the module are run by calling mod:test().

It is possible to separate the test functions from the module (mod, say) into the module mod_tests.erl, which should also include the eunit.hrl header file. The tests are invoked in exactly the same way, using mod:test().

It may be that you want to use EUnit on your code—for example, to use the ?assert macro—but you don’t want to generate tests. In this case, you need to insert the following code before the line that includes the eunit.hrl library:

-define(NOTEST, true)

If this macro appears in a header file included by all modules in an application, it gives a single point controlling whether testing is enabled. Now we will illustrate how tests are written, using an example you’ve seen before.

Functional Testing, an Example: Tree Serialization

In the section Serialization in Chapter 9, we looked at how a binary tree could be serialized as a list, and then reconstructed from the list. We presented an optimized version of this, but left the original version as an exercise. In this unoptimized version, the first element of the representation gives its size recursively through the tree?

treeToList({leaf,N}) ->
  [2,N];

treeToList({node,T1,T2}) ->
  TTL1 = treeToList(T1),
  [Size1|_] = TTL1,
  TTL2 = treeToList(T2),
  [Size2|_] = TTL2,
  [Size1+Size2+1|TTL1++TTL2].

listToTree([2,N]) ->
  {leaf,N};

listToTree([_|Code]) ->
  case Code of
    [M|_] ->
      {Code1,Code2} = lists:split(M,Code),
      {node,
        listToTree(Code1),
        listToTree(Code2)
       }
  end.

The function treeToList/1 converts the tree to a list, and listToTree/1 should convert that representation back to the original tree. We can now test the functions with some example trees:

tree0() ->
  {leaf, ant}.

tree1() ->
  {node,
    {node,
     {leaf,cat},
     {node,
      {leaf,dog},
      {leaf,emu}
     }
    },
    {leaf,fish}
   }.

We require that the two functions applied one after the other return the original value:

leaf_test() ->
  ?assertEqual(tree0() , listToTree(treeToList(tree0()))).
node_test() ->
  ?assertEqual(tree1() , listToTree(treeToList(tree1()))).

where we use the eunit macro assertEqual to test for equality between the value of two Erlang terms.

We can also test for particular values of the treeToList function:

leaf_value_test() ->
  ?assertEqual([2,ant] , treeToList(tree0())).
node_value_test() ->
  ?assertEqual([11,8,2,cat,5,2,dog,2,emu,2,fish] , treeToList(tree1())).

In testing listToTree, we do something different. This function is partial, and we can check that it is undefined outside the range of treeToList:

leaf_negative_test() ->
  ?assertError(badarg, listToTree([1,ant])).
node_negative_test() ->
  ?assertError(badarg, listToTree([8,6,2,cat,2,dog,emu,fish])).

These tests use the assertError macro, which captures a raised exception and checks that it is of the form specified by the first argument (here, a badarg).

With these tests included in the file, running the test/0 function in the shell gives the brief report that all tests are successful:

2> serial:test().
  All 6 tests successful.
ok

There is nothing more to say when tests are successful. Alas, not all tests are successful. What does EUnit report when tests fail? In true test-driven development style, as an illustration, we can regression-test the optimized version of serialization with the same test set and see what happens:

11> serial2:test().
serial2:leaf_negative_test...*failed*
::error:{assertException_failed,[{module,serial2},
                               {line,66},
                               {expression,"listToTree ( [ 1 , ant ] )"},
                               {expected,"{ error , badarg , [...] }"},
                               {unexpected_success,{leaf,ant}}]}
  in function serial2:'-leaf_negative_test/0-fun-0-'/0

serial2:node_value_test...*failed*
::error:{assertEqual_failed,[{module,serial2},
                           {line,72},
                           {expression,"treeToList ( tree1 ( ) )"},
                           {expected,[11,8,2,cat,5,2|...]},
                           {value,[8,6,2,cat,2|...]}]}
  in function serial2:'-node_value_test/0-fun-0-'/1

serial2:node_negative_test...*failed*
 ... details similar ...

=======================================================
  Failed: 3.  Aborted: 0.  Skipped: 0.  Succeeded: 3.
error

This report shows that three of the six tests have failed, and gives detailed feedback on the cause of failure. In the case of leaf_negative_test, it is that the particular function application succeeds unexpectedly, instead of raising a badarg exception. In the second case, the actual result was different from the actual value, both of which are printed in the report.

The failed tests either cover values for which the original functions failed, or are sensitive to changes in the new implementation, where the details of the list representation have changed. The first two tests that check that applying the functions one after the other returns the original argument, however, confirm that the crucial property still holds.

Note

If you want to put the test functions into a separate serial_tests module, you can use an import directive to include the tests without making any changes from the serial module:

-module(serial_tests).
-include_lib("eunit/include/eunit.hrl").
-import(serial,
       [treeToList/1, listToTree/1
        tree0/0, tree1/0,]).

leaf_test() ->
  ?assertEqual(tree0() ,
               listToTree(treeToList(tree0()))).
... etc ...

The EUnit Infrastructure

In this section, you will learn about the foundations of the EUnit system, with which you can build tests and test sets.

Assert Macros

The basic building block of EUnit is a single test, given by a ..._test() function. On the right side of the earlier code examples, we used assertEqual and assertError to check values and exceptions. Other assert macros include the following:

assert(BoolExpr)

Can be used not only in tests, but anywhere in a program to check the value of a Boolean expression at that point

assertNot(BoolExpr)

Is equivalent to assert(not(BoolExpr))

assertMatch(GuardedPattern, Expr)

Will evaluate the Expr, and if it fails to match the guarded pattern, an exception is reported on test()

assertExit(TermPattern, Expr) and assertThrow(TermPattern, Expr)

Will test for a program exit or a throw of an exception, similar to assertError

In the example, we used assertEqual(E, F) instead of assert(E =:= F) because assertEqual generates more informative messages when the test fails.

Test-Generating Functions

Beyond a single test, you can define test-generating functions that combine a number of tests into a single function. A test generator returns a representation of a set of tests to be executed by EUnit.

The simplest way to represent a test is as a fun expression that takes no arguments:

leaf_value_test_() ->
     fun () -> ?assertEqual([2,ant] , treeToList(tree0())) end.

In the preceding code, the differences from the definition of leaf_value_test are highlighted. The macro library allows you to write this more succinctly:

leaf_value_test_() ->
     ?_assertEqual([2,ant] , treeToList(tree0())).

In this code, the _assertEqual macro plays the same role as assertEqual, but for test representations rather than tests.

A test-generating function will, in general, return a set of tests. For instance, the following code encapsulates two tests into a single function:

tree_test_() ->
  [?_assertEqual(tree0() , listToTree(treeToList(tree0()))),
   ?_assertEqual(tree1() , listToTree(treeToList(tree1())))].

When EUnit runs this test, all the tests in the list are performed.

EUnit Test Representation

EUnit represents tests and test sets in a variety of different ways. Here is a list of the most useful; a full description is in the EUnit documentation. A test representation TestRep is run by calling eunit:test(TestRep):

Simple test objects

The simplest test object is a nullary fun, that is, a fun that takes no arguments. A simple test object is also given by a pair of atoms, {Module, Function}, referring to a function within a module.

Test sets

Test sets are given by lists and deep lists. A module name (atom) is also used to represent the tests within the module.

Primitives

The primitives do not contain embedded test sets as arguments, but instead are descriptions of tests that lie within a module, as in {module, Module}; within a directory, as in {dir, Path::string()}; and within an application, a file, and so forth. Generators are also embedded as {generator, GenFun::(() -> Tests)}.

Control

It is possible to control how and where tests are to be executed:

{spawn, Tests}

Will run the tests in a separate subprocess, with the test process waiting until the tests finish

{timeout, Time::number(), Tests}

Will run the tests for Time seconds, terminating any unfinished tests at that time

{inorder, Tests}

Will run the tests in strict order

{inparallel, Tests}

Will run the tests in parallel where possible

Fixtures

Fixtures support the setup and cleanup for a particular set of tests to run; we discuss them in more detail in the next section.

Testing State-Based Systems

To explain this topic, we will go back to the example of the mobile user database. When we introduced the example in Chapter 10, we tested it from the Erlang shell (see the section A Mobile Subscriber Database Example). This section shows how you can incorporate this style of testing into EUnit. We will also look in more detail at the way in which tests are represented.

Fixtures: Setup and Cleanup

It is characteristic of state-based systems that you can test particular properties only after you have set up the right configuration; once you’ve done this, the test can take place, but after that, you need to clean up the system to prepare for any further tests.

The first test we’ll write will test a lookup on an empty database after the tables are created with create_tables("UsrTabFile"):

?_assertMatch({error,instance}, lookup_id(1))

After the test is run, we clean up by removing the file UsrTabFile. This is implemented by writing a fixture: a test description that allows setup and cleanup. The simplest fixtures have the following form:

{setup, Setup, Tests}
{setup, Setup, Cleanup, Tests}

where, for some type T:

Setup   :: (() -> T)
Cleanup :: ((T) -> any())

the Setup function is executed before the Tests, returning a value X of type T. After the tests, Cleanup(X) is performed; this allows information from the setup—for example, about pids or tables—to be communicated to the cleanup phase.

For our example, we can write the following:

setup1_test_() ->
   {spawn,
    {setup,
     fun ()  -> create_tables("UsrTabFile") end,           % setup
     fun (_) -> ?cmd("rm UsrTabFile") end,                 % cleanup
     ?_assertMatch({error,instance}, lookup_id(1))
    }
   }.

Note in the cleanup that we can call an external Unix command to remove the file using the ?cmd macro, and that the test is executed by a call to eunit:test/1. To test that the database is functioning correctly, we need a rather more elaborate setup; on termination of the spawned process, the ETS tables constructed by the program will be destroyed:

setup2_test_() ->
  {spawn,
    {setup,
      fun ()  ->
        create_tables("UsrTabFile"),
        Seq = lists:seq(1,100000),
        Add = fun(Id) -> add_usr(#usr{msisdn = 700000000 + Id,
                                      id = Id,
                                      plan = prepay,
                                      services = [data, sms, lbs]})
              end,
        lists:foreach(Add, Seq)
      end,
      fun (_) -> ?cmd("rm UsrTabFile") end,
      ?_assertMatch({ok, #usr{status = enabled}} , lookup_msisdn(700000001) )
    }
  }.

We can run these two tests by calling Mod:test() or eunit:test(Mod), where Mod is the name of the module containing the tests.

Testing Concurrent Programs in Erlang

When a program consists of a number of objects evolving concurrently, it is harder to see how a unit-testing framework such as EUnit can directly help. Although EUnit provides the scaffolding for an expression to be applied at a particular point during system evolution, it is harder to monitor the evolution of the system itself. The biggest challenges in testing concurrent systems are race conditions. You run your test, and you can easily reproduce your error. You turn on the trace on these processes, and all of a sudden everything works as expected, as the extra I/O causes your processes to execute in a different order.

Some EUnit facilities can be useful; it is possible to ?assert a property at any point in the program, and so to monitor when a pre- or post-condition or system invariant is broken. EUnit also provides support for debugging, with messages reported to the Erlang console rather than to standard output. These macros include the following:

debugVal(Expr)

Will print the source code and current value of Expr. The result is always the value of Expr, and so the macro can be written around any expression in the program without affecting its functionality.

debugTime(Text, Expr)

Will print the Text followed by the (elapsed) execution time of the Expr.

If the NODEBUG macro is set (to true) before the eunit header file is included in the module, the macros have no effect in that module.

Other systems that support the testing of concurrent programs include Quviq QuickCheck,[46] which implements property-based random testing for Erlang; McErlang,[47] a model checker for Erlang written in Erlang; and Common Test, a systems-testing framework based on the OTP Test Server application, and part of the standard Erlang distribution.

Exercises

Exercise 19-1: Testing Sequential Functions

Revisit the exercises in Chapter 3, and devise EUnit tests for your solutions: do your solutions pass all the tests? Is that because of faults in the solutions or in the way you have defined the tests?

Exercise 19-2: Testing Concurrent Systems

Devise tests within EUnit for the echo process introduced in Chapter 4. How do you have to change the tests when the implementation is modified to register the process?

Exercise 19-3: Software Upgrade

How would you define EUnit tests for a software upgrade in the db_server example given in Chapter 8?

Exercise 19-4: Testing OTP Behaviors

Incorporate the test examples given in Chapter 12 into the EUnit framework.

Exercise 19-5: Devising Tests for OTP Behaviors

Devise EUnit tests for the solutions to the exercises in Chapter 12.



[46] http://www.quviq.com/; an earlier version of the tool, and its application in testing telecom software, is discussed at http://doi.acm.org/10.1145/1159789.1159792.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset