As you are writing a program, how do you understand how the program will behave?
You might have a model in your mind of what the program will do, but you can
be sure of it only when you exercise or
interact with your program in some way. Chapter 18 showed you how you can use –spec
to express what you think the input and
output types of a function should be; TypEr can check whether this is
consistent with the code itself.
Types don’t tell you how a program behaves, however, and testing is one of the best ways to understand how your code will function. We have been doing this informally throughout the book; each time we have given some definitions, we have immediately gone to the Erlang shell and tried them out in practice. When you’re developing in Erlang your coding and test cycles tend to be small. You write a few functions, and you test them. You add a few more, and you test them again. Repeating all of the tests in the shell every time becomes both time-consuming and error-prone.
In this chapter, we’ll introduce the EUnit tool, which gives you a framework for unit testing in Erlang. We’ll show how it is used in practice, as well as discuss how it supports what is known in software engineering circles as test-driven development.
The waterfall model of software development saw a software system being developed in a series of steps: first, the requirements of the system would be elucidated; on the basis of this, the system would be designed, and only once this was complete would the implementation begin. No wonder so many software projects failed to deliver what the customer wanted! Needless to say, the waterfall model is as disastrous in Erlang as it is in any other programming language.
A test-driven approach turns this on its head. Implementation begins on day one, but with a focus: each feature that is to be implemented is first characterized by a set of tests. This increment is added to the code and is accepted only if the tests pass. At the same time, this should not invalidate any earlier test; regression tests must also pass.
The implementer benefits from such an approach because she has a clear target at each stage: pass the tests. The customer is also in the happy position of being able to interact with or exercise the system continually: each time the evolving system builds, he can see whether it behaves as it ought to.
Test-driven development (TDD) has been associated with agile programming, but it would be wrong to identify the two. An informal test-driven approach has characterized functional programming since the early LISP systems: most functional programming systems have at their top level a read-evaluate-print loop, which encourages a test- or example-based approach. Erlang comes out of that tradition, with its roots in functional programming and Prolog, and the examples of interactions in the Erlang shell show this informal test-driven approach in practice.
Test-driven development is not confined to the single-person project. There is anecdotal evidence from industry that TDD can be effective in larger software projects, too, principally because it means the development team engages with potential problems earlier, and in the process develops more comprehensive test suites than it would with a more traditional approach. In turn, this leads to better software being delivered. It is needless to say that Erlang and TDD go hand in hand. In the remainder of this chapter, we will cover the EUnit system, which provides support for a more formal test-driven approach, and which is included in the standard Erlang distribution.
EUnit provides a framework for defining and executing unit tests, which test that a particular program unit—in Erlang, a function or collection of functions—behaves as expected. The framework gives a representation of tests of a variety of different kinds, and a set of macros which simplify the way EUnit tests can be written.
For a function without side effects, a test will typically look at whether the input/output behavior is as expected. Additionally, it may test whether the function raises an exception (only) when required.
Functions that have side effects require a more complex infrastructure. Side effects include operations that might affect persistent data structures such as ETS or Dets tables, or indeed operating system structures such as files and I/O, as well as operations that contain message passing among concurrent processes. The infrastructure needed to test these programs includes the following:
Testing side-effecting programs typically requires some setup and initial modification of the data before checking the behavior of a particular operation. This needs to be followed by a cleanup of the program state.
Testing units within a concurrent program typically requires a test rig in which some mock objects or stubs are written to stand in the place of other components expected to interact with the unit under test.
EUnit supports testing side-effecting and concurrent programs. Let’s start with the simplest case of functional testing.
EUnit provides a detailed representation of tests, as well as a layer of
macros making tests easier to write. It also provides a
mechanism for gathering all the tests in a module, executing them all,
and providing a report on the results. For a module to use EUnit it
needs to include the eunit
library:
-include_lib("eunit/include/eunit.hrl").
In EUnit, a single test is identified by a function named
name
_test
;
a test-generating function will be named
name
_test_
(i.e., with an _
at the end). We
define test-generating functions in the section The EUnit Infrastructure.
With the EUnit header file included, compilation of the module
(mod
, say) will result in a function test/0
being exported, and all the tests
within the module are run by calling
mod
:test()
.
It is possible to separate the test functions from the module
(mod
, say) into the module
mod
_tests.erl
, which should also include the
eunit.hrl header file. The tests
are invoked in exactly the same way, using
mod
:test()
.
It may be that you want to use EUnit on your code—for example, to
use the ?assert
macro—but you don’t
want to generate tests. In this case, you need to insert the following
code before the line that includes the eunit.hrl
library:
-define(NOTEST, true)
If this macro appears in a header file included by all modules in an application, it gives a single point controlling whether testing is enabled. Now we will illustrate how tests are written, using an example you’ve seen before.
In the section Serialization in Chapter 9, we looked at how a binary tree could be serialized as a list, and then reconstructed from the list. We presented an optimized version of this, but left the original version as an exercise. In this unoptimized version, the first element of the representation gives its size recursively through the tree?
treeToList({leaf,N}) -> [2,N]; treeToList({node,T1,T2}) -> TTL1 = treeToList(T1), [Size1|_] = TTL1, TTL2 = treeToList(T2), [Size2|_] = TTL2, [Size1+Size2+1|TTL1++TTL2]. listToTree([2,N]) -> {leaf,N}; listToTree([_|Code]) -> case Code of [M|_] -> {Code1,Code2} = lists:split(M,Code), {node, listToTree(Code1), listToTree(Code2) } end.
The function treeToList/1
converts
the tree to a list, and listToTree/1
should
convert that representation back to the original tree. We can now test
the functions with some example trees:
tree0() -> {leaf, ant}. tree1() -> {node, {node, {leaf,cat}, {node, {leaf,dog}, {leaf,emu} } }, {leaf,fish} }.
We require that the two functions applied one after the other return the original value:
leaf_test() -> ?assertEqual(tree0() , listToTree(treeToList(tree0()))). node_test() -> ?assertEqual(tree1() , listToTree(treeToList(tree1()))).
where we use the eunit
macro assertEqual
to test for equality between the
value of two Erlang terms.
We can also test for particular values of the treeToList
function:
leaf_value_test() -> ?assertEqual([2,ant] , treeToList(tree0())). node_value_test() -> ?assertEqual([11,8,2,cat,5,2,dog,2,emu,2,fish] , treeToList(tree1())).
In testing listToTree
, we do
something different. This function is partial, and we can check that it
is undefined outside the range of treeToList
:
leaf_negative_test() -> ?assertError(badarg, listToTree([1,ant])). node_negative_test() -> ?assertError(badarg, listToTree([8,6,2,cat,2,dog,emu,fish])).
These tests use the assertError
macro,
which captures a raised exception and checks that it is of the form
specified by the first argument (here, a badarg
).
With these tests included in the file, running the test/0
function in the shell gives the brief
report that all tests are successful:
2> serial:test().
All 6 tests successful.
ok
There is nothing more to say when tests are successful. Alas, not all tests are successful. What does EUnit report when tests fail? In true test-driven development style, as an illustration, we can regression-test the optimized version of serialization with the same test set and see what happens:
11> serial2:test().
serial2:leaf_negative_test...*failed*
::error:{assertException_failed,[{module,serial2},
{line,66},
{expression,"listToTree ( [ 1 , ant ] )"},
{expected,"{ error , badarg , [...] }"},
{unexpected_success,{leaf,ant}}]}
in function serial2:'-leaf_negative_test/0-fun-0-'/0
serial2:node_value_test...*failed*
::error:{assertEqual_failed,[{module,serial2},
{line,72},
{expression,"treeToList ( tree1 ( ) )"},
{expected,[11,8,2,cat,5,2|...]},
{value,[8,6,2,cat,2|...]}]}
in function serial2:'-node_value_test/0-fun-0-'/1
serial2:node_negative_test...*failed*
... details similar ...
=======================================================
Failed: 3. Aborted: 0. Skipped: 0. Succeeded: 3.
error
This report shows that three of the six tests have failed, and
gives detailed feedback on the cause of failure. In the case of leaf_negative_test
, it is that the particular
function application succeeds unexpectedly, instead of raising a
badarg
exception. In the second case,
the actual result was different from the actual value, both of which are
printed in the report.
The failed tests either cover values for which the original functions failed, or are sensitive to changes in the new implementation, where the details of the list representation have changed. The first two tests that check that applying the functions one after the other returns the original argument, however, confirm that the crucial property still holds.
If you want to put the test functions into a separate serial_tests
module, you can use an import
directive to include the tests
without making any changes from the serial
module:
-module(serial_tests). -include_lib("eunit/include/eunit.hrl"). -import(serial, [treeToList/1, listToTree/1 tree0/0, tree1/0,]). leaf_test() -> ?assertEqual(tree0() , listToTree(treeToList(tree0()))). ... etc ...
In this section, you will learn about the foundations of the EUnit system, with which you can build tests and test sets.
The basic building block of EUnit is a single test, given by a
..._test()
function. On the right
side of the earlier code examples, we used assertEqual
and
assertError
to check values and
exceptions. Other assert macros include the following:
assert(BoolExpr)
Can be used not only in tests, but anywhere in a program to check the value of a Boolean expression at that point
assertNot(BoolExpr)
assertMatch(GuardedPattern,
Expr)
Will evaluate the Expr
, and if it fails to match the
guarded pattern, an exception is reported on test()
assertExit(TermPattern,
Expr)
and assertThrow(TermPattern, Expr)
Will test for a program exit or a throw of an exception,
similar to assertError
In the example, we used assertEqual(E,
F)
instead of assert(E =:=
F)
because assertEqual
generates more
informative messages when the test fails.
Beyond a single test, you can define test-generating functions that combine a number of tests into a single function. A test generator returns a representation of a set of tests to be executed by EUnit.
The simplest way to represent a test is as a fun
expression that takes no arguments:
leaf_value_test_
() ->fun () ->
?assertEqual([2,ant] , treeToList(tree0()))end
.
In the preceding code, the differences from the definition of
leaf_value_test
are highlighted. The
macro library allows you to write this more succinctly:
leaf_value_test_
() -> ?_
assertEqual([2,ant] , treeToList(tree0())).
In this code, the _assertEqual
macro plays the same role as assertEqual
, but for test representations
rather than tests.
A test-generating function will, in general, return a set of tests. For instance, the following code encapsulates two tests into a single function:
tree_test_() -> [?_assertEqual(tree0() , listToTree(treeToList(tree0()))), ?_assertEqual(tree1() , listToTree(treeToList(tree1())))].
When EUnit runs this test, all the tests in the list are performed.
EUnit represents tests and test sets in a variety of different ways. Here is a
list of the most useful; a full description is in the EUnit
documentation. A test representation TestRep
is run by calling
eunit:test(TestRep)
:
The simplest test object is a nullary fun
, that is, a fun
that takes no arguments. A simple
test object is also given by a pair of atoms, {Module, Function}
, referring to a
function within a module.
Test sets are given by lists and deep lists. A module name (atom) is also used to represent the tests within the module.
The primitives do not contain embedded test sets as
arguments, but instead are descriptions of tests that lie within a
module, as in {module, Module}
;
within a directory, as in {dir,
Path::string()}
; and within an application, a file, and
so forth. Generators are also embedded as {generator, GenFun::(() ->
Tests)}
.
It is possible to control how and where tests are to be executed:
{spawn, Tests}
Will run the tests in a separate subprocess, with the test process waiting until the tests finish
{timeout, Time::number(),
Tests}
Will run the tests for Time
seconds, terminating any
unfinished tests at that time
{inorder,
Tests}
Will run the tests in strict order
{inparallel,
Tests}
Will run the tests in parallel where possible
Fixtures support the setup and cleanup for a particular set of tests to run; we discuss them in more detail in the next section.
To explain this topic, we will go back to the example of the mobile user database. When we introduced the example in Chapter 10, we tested it from the Erlang shell (see the section A Mobile Subscriber Database Example). This section shows how you can incorporate this style of testing into EUnit. We will also look in more detail at the way in which tests are represented.
It is characteristic of state-based systems that you can test particular properties only after you have set up the right configuration; once you’ve done this, the test can take place, but after that, you need to clean up the system to prepare for any further tests.
The first test we’ll write will test a lookup on an empty database
after the tables are created with create_tables("UsrTabFile")
:
?_assertMatch({error,instance}, lookup_id(1))
After the test is run, we clean up by removing the file UsrTabFile. This is implemented by writing a fixture: a test description that allows setup and cleanup. The simplest fixtures have the following form:
{setup, Setup, Tests} {setup, Setup, Cleanup, Tests}
where, for some type T
:
Setup :: (() -> T) Cleanup :: ((T) -> any())
the Setup
function is executed
before the Tests
, returning a value
X
of type T
. After the tests, Cleanup(X)
is performed; this allows
information from the setup—for example, about pids or tables—to be
communicated to the cleanup phase.
For our example, we can write the following:
setup1_test_() -> {spawn, {setup, fun () -> create_tables("UsrTabFile") end, % setup fun (_) -> ?cmd("rm UsrTabFile") end, % cleanup ?_assertMatch({error,instance}, lookup_id(1)) } }.
Note in the cleanup that we can call an external Unix command to
remove the file using the ?cmd
macro,
and that the test is executed by a call to eunit:test/1
. To test
that the database is functioning correctly, we need a rather more
elaborate setup; on termination of the spawn
ed process, the ETS tables constructed by
the program will be destroyed:
setup2_test_() -> {spawn, {setup, fun () -> create_tables("UsrTabFile"), Seq = lists:seq(1,100000), Add = fun(Id) -> add_usr(#usr{msisdn = 700000000 + Id, id = Id, plan = prepay, services = [data, sms, lbs]}) end, lists:foreach(Add, Seq) end, fun (_) -> ?cmd("rm UsrTabFile") end, ?_assertMatch({ok, #usr{status = enabled}} , lookup_msisdn(700000001) ) } }.
We can run these two tests by calling Mod:test()
or eunit:test(Mod)
, where Mod
is the name of the module containing the
tests.
When a program consists of a number of objects evolving concurrently, it is harder to see how a unit-testing framework such as EUnit can directly help. Although EUnit provides the scaffolding for an expression to be applied at a particular point during system evolution, it is harder to monitor the evolution of the system itself. The biggest challenges in testing concurrent systems are race conditions. You run your test, and you can easily reproduce your error. You turn on the trace on these processes, and all of a sudden everything works as expected, as the extra I/O causes your processes to execute in a different order.
Some EUnit facilities can be useful; it is possible to ?assert
a property at any point in the program,
and so to monitor when a pre- or post-condition or system invariant is
broken. EUnit also provides support for debugging, with messages reported to the Erlang console
rather than to standard output. These macros include the following:
debugVal(Expr)
Will print the source code and current value of Expr
. The result is always the value of
Expr
, and so the macro can be
written around any expression in the program without affecting its
functionality.
debugTime(Text, Expr)
Will print the Text
followed by the (elapsed) execution time of the Expr
.
If the NODEBUG
macro is set (to
true
) before the eunit
header file is included in the module, the
macros have no effect in that module.
Other systems that support the testing of concurrent programs include Quviq QuickCheck,[46] which implements property-based random testing for Erlang; McErlang,[47] a model checker for Erlang written in Erlang; and Common Test, a systems-testing framework based on the OTP Test Server application, and part of the standard Erlang distribution.
Revisit the exercises in Chapter 3, and devise EUnit tests for your solutions: do your solutions pass all the tests? Is that because of faults in the solutions or in the way you have defined the tests?
Devise tests within EUnit for the echo
process introduced in Chapter 4.
How do you have to change the tests when the implementation is modified
to register the process?
How would you define EUnit tests for a software upgrade in the
db_server
example given in Chapter 8?
Incorporate the test examples given in Chapter 12 into the EUnit framework.
Devise EUnit tests for the solutions to the exercises in Chapter 12.
[46] http://www.quviq.com/; an earlier version of the tool, and its application in testing telecom software, is discussed at http://doi.acm.org/10.1145/1159789.1159792.
[47] https://babel.ls.fi.upm.es/trac/McErlang/; an early version of the tool is described at http://doi.acm.org/10.1145/1291220.1291171.