Difference between revisions of "Tests"

From Octave
Jump to navigation Jump to search
(much expansion of text, added a lot more examples, no text left from previous)
Line 1: Line 1:
Writing tests for function is an important thing that is usually overlooked. It helps a lot in preventing regression. There's a section in the Octave manual on [http://www.gnu.org/software/octave/doc/interpreter/Test-Functions.html#Test-Functions Test Functions].
+
Having a thorough [http://en.wikipedia.org/wiki/Test_suite test suite] is
 +
something very important which is usually overlooked. It is an incredible
 +
help in preventing regression bugs and quickly assess the status of old code.
 +
For example, many packages in Octave Forge become deprecated after losing
 +
their maintainer simply because they have no test suite.
 +
 
 +
GNU Octave has multiple tools that help in creating a comprehensive test
 +
suite, accessible to both developers and end-users, as detailed on the
 +
[http://www.gnu.org/software/octave/doc/interpreter/Test-Functions.html Octave manual].
 +
Basically, test blocks are {{codeline|%!test}} comment blocks, typically at the
 +
end of a source file, which are ignored by the Octave interpreter and only
 +
read by the {{codeline|test}} function.
 +
 
 +
== Running tests ==
 +
 
 +
To run all the tests of a specific function, simply use the {{codeline|test}}
 +
command at the Octave prompt.  For example, to run the tests in
 +
{{codeline|mean()}}:
 +
 
 +
octave-cli-3.8.2> test mean
 +
PASSES 17 out of 17 tests
 +
 
 +
These tests are written in the Octave language at the bottom of the
 +
[http://hg.savannah.gnu.org/hgweb/octave/file/6443693a176f/scripts/statistics/base/mean.m#l130 m file]
 +
which defines {{codeline|mean()}}.  It is important that these tests are
 +
also available for the end users so they can test the status of their
 +
installation.  The whole Octave test suite can be ran with:
 +
 
 +
octave-cli-3.8.2> __run_test_suite__
 +
 +
Integrated test scripts:
 +
 +
[...]
 +
 +
Summary:
 +
 +
  PASS    11556
 +
  FAIL        3
 +
  XFAIL        6
 +
  SKIPPED    38
 +
 +
See the file test/fntests.log for additional details.
 +
 
 +
To run tests in a specific file, one can simply specify the path instead of
 +
a function name:
 +
 
 +
  test /full/path/to/file.cc
 +
 
  
 
== Writing tests ==
 
== Writing tests ==
  
Octave provides for robust testing of functionsThe manual includes a section on the [http://www.gnu.org/software/octave/doc/interpreter/Test-Functions.html#Test-Functions Test Functions].  Several examples are included.
+
Tests appear as {{codeline|%!}} blocks at the bottom of the source file,
 +
together with {{codeline|%!demo}} blocksA typical m function file, will
 +
have the following structure:
  
===Declaring Functions Inside a Test Block===
+
## Copyright
 +
##
 +
## A block with the copyright notice
 +
 +
## -*- texinfo -*-
 +
##
 +
## A block with the help text
 +
 +
function [x, y, z] = foo (bar)
 +
  ## here's some amazing code
 +
endfunction
 +
 +
%!assert (foo (1))
 +
%!assert (foo (1:10))
 +
%!assert (foo ("on"), "off")
 +
%!error <must be positive integer> foo (-1)
 +
%!error <must be positive integer> foo (1.5)
 +
 +
%!demo
 +
%! ## see how cool foo() is:
 +
%! foo([1:100])
  
Octave's ''test'' function automatically inserts the contents of each test block into a function. Although it is admittedly hacky/ugly, this implies that addition functions may be declared within a test block. For example, the empty function below named {{Codeline|experience}} is accompanied by a single test block which includes two function definitions.
+
Tests can be added to oct functions in the C++ sources just as easily, see
 +
[http://hg.savannah.gnu.org/hgweb/octave/file/f5ad7470d957/libinterp/corefcn/find.cc#l566 find.cc]
 +
for example. The syntax is exactly the same, but done within C comment blocks.
 +
During installation, these lines are automatically extracted from the sources
 +
and special test scripts are generated.  A typical C++ source file has the
 +
following structure:
 +
 
 +
// Copyright
 +
//
 +
// A block with the copyright notice
 +
 +
  DEFUN_DLD (foo, args, ,
 +
"-*- texinfo -*-\n\
 +
A block with the help text")
 +
{
 +
  \\ here's some amazing code
 +
}
 +
function [x, y, z] = foo (bar)
 +
  ## here's some amazing code
 +
endfunction
 +
 +
/*
 +
%!assert (foo (1))
 +
%!assert (foo (1:10))
 +
%!assert (foo ("on"), "off")
 +
%!error <must be positive integer> foo (-1)
 +
%!error <must be positive integer> foo (1.5)
 +
*/
 +
 
 +
 
 +
=== Assert ===
 +
 
 +
{{codeline|%!assert}} lines are simplest tests to write and also the most
 +
common:
 +
 
 +
%!assert (foo (bar))      # test fails if "foo (bar)" returns false
 +
%!assert (foo (bar), qux) # test fails if "foo (bar)" is different from "qux
 +
 
 +
These are actually a shorthand version of
 +
{{codeline|%!test assert (foo (bar))}}, and {{codeline|assert}} is simply
 +
an Octave function that throws errors when two arguments fail to compare.
 +
 
 +
=== Test ===
 +
 
 +
While single line {{codeline|%!assert}}s are the most common test used,
 +
{{codeline|%!test}} blocks are the ultimate, most useful, and flexible.
 +
The code within such block is simply processed through the Octave interpreter
 +
and if the code generates an error, then the test is said to fail. These
 +
often end with a call to {{codeline|assert}}:
  
function experience ()
 
 
  %!test
 
  %!test
  %! experience_design_mat
+
  %! a = [0 1 0 0 3 0 0 5 0 2 1];
%! experience_obs_eqs
+
  %! b = [2 5 8 10 11];
%! assert (experience_design_mat == pi);
+
  %! for i = 1:5
  %! assert (experience_obs_eqs == exp(1));
+
  %!   assert (find (a, i), b(1:i))
  %!
+
  %! endfor
%! endfunction  % this is a trick.
 
%! % now we can declare functions to be used by the test above.
 
%!
 
%! function a = experience_design_mat
 
  %!     a = pi;
 
%! endfunction
 
%!
 
%! function b = experience_obs_eqs
 
%!    b = exp(1);
 
  %! % endfunction: don't add it here. Let test() do it.
 
  
== Running Tests ==
+
==== Test for no failure ====
  
Tests are run tests in m-files by using the {{Codeline|test}} function/command.  For example, to test the {{Codeline|experience}} above use the command below.
+
In a few cases, there is the situation where a function returns nothing,
 +
and the only thing to test is that it causes no error. This can be tested
 +
simply with:
  
  test experience
+
%!test foo (bar)
  
Or alternatively, use the functional form
+
=== Error ===
  
  test ("experience")
+
It is also important to test that a function performs its checks correctly
 +
and throws errors when it receives garbage.  This can be done with
 +
{{codeline|error}} blocks:
  
To run tests in .cc files, the path to the file must be provided.
+
%!error foo ()  # test that causes any error
 +
%!error <BAR must be a positive integer> foo (-1.5)  # test that throws specific error
  
  test /full/path/to/file.cc
 
  
[[Category:Testing]]
+
 
 +
=== Shared functions ===
 +
 
 +
It is often useful to share a function between multiple test.  Sometimes
 +
these are only small helper functions, but more often these are just simpler
 +
low performance implementations of the function being tested.  These are
 +
created in {{codeline|%!function}} blocks:
 +
 
 +
 
 +
%!function x = slow_foo (bar)
 +
%!  ## a simple implementation of foo, definitely correct, but
 +
%!  ## unfortunately too slow for anything other than tests.
 +
%!endfunction
 +
 +
%!assert (foo (bar), slow_foo (bar))
 +
 +
%!test
 +
%! for i = -100:100
 +
%!  bar = qux (i);
 +
%!  assert (foo (bar), slow_foo (bar))
 +
%! endfor
 +
 
 +
 
 +
== Code coverage ==
 +
 
 +
[[Category:Development]]

Revision as of 16:55, 5 October 2014

Having a thorough test suite is something very important which is usually overlooked. It is an incredible help in preventing regression bugs and quickly assess the status of old code. For example, many packages in Octave Forge become deprecated after losing their maintainer simply because they have no test suite.

GNU Octave has multiple tools that help in creating a comprehensive test suite, accessible to both developers and end-users, as detailed on the Octave manual. Basically, test blocks are %!test comment blocks, typically at the end of a source file, which are ignored by the Octave interpreter and only read by the test function.

Running tests

To run all the tests of a specific function, simply use the test command at the Octave prompt. For example, to run the tests in mean():

octave-cli-3.8.2> test mean
PASSES 17 out of 17 tests

These tests are written in the Octave language at the bottom of the m file which defines mean(). It is important that these tests are also available for the end users so they can test the status of their installation. The whole Octave test suite can be ran with:

octave-cli-3.8.2> __run_test_suite__

Integrated test scripts:

[...]

Summary:

  PASS     11556
  FAIL         3
  XFAIL        6
  SKIPPED     38

See the file test/fntests.log for additional details.

To run tests in a specific file, one can simply specify the path instead of a function name:

 test /full/path/to/file.cc


Writing tests

Tests appear as %! blocks at the bottom of the source file, together with %!demo blocks. A typical m function file, will have the following structure:

## Copyright
##
## A block with the copyright notice

## -*- texinfo -*-
##
## A block with the help text

function [x, y, z] = foo (bar)
  ## here's some amazing code
endfunction

%!assert (foo (1))
%!assert (foo (1:10))
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (-1)
%!error <must be positive integer> foo (1.5)

%!demo
%! ## see how cool foo() is:
%! foo([1:100])

Tests can be added to oct functions in the C++ sources just as easily, see find.cc for example. The syntax is exactly the same, but done within C comment blocks. During installation, these lines are automatically extracted from the sources and special test scripts are generated. A typical C++ source file has the following structure:

// Copyright
//
// A block with the copyright notice

DEFUN_DLD (foo, args, ,
"-*- texinfo -*-\n\
A block with the help text")
{
 \\ here's some amazing code
}
function [x, y, z] = foo (bar)
  ## here's some amazing code
endfunction

/*
%!assert (foo (1))
%!assert (foo (1:10))
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (-1)
%!error <must be positive integer> foo (1.5)
*/


Assert

%!assert lines are simplest tests to write and also the most common:

%!assert (foo (bar))      # test fails if "foo (bar)" returns false
%!assert (foo (bar), qux) # test fails if "foo (bar)" is different from "qux

These are actually a shorthand version of %!test assert (foo (bar)), and assert is simply an Octave function that throws errors when two arguments fail to compare.

Test

While single line %!asserts are the most common test used, %!test blocks are the ultimate, most useful, and flexible. The code within such block is simply processed through the Octave interpreter and if the code generates an error, then the test is said to fail. These often end with a call to assert:

%!test
%! a = [0 1 0 0 3 0 0 5 0 2 1];
%! b = [2 5 8 10 11];
%! for i = 1:5
%!   assert (find (a, i), b(1:i))
%! endfor

Test for no failure

In a few cases, there is the situation where a function returns nothing, and the only thing to test is that it causes no error. This can be tested simply with:

%!test foo (bar)

Error

It is also important to test that a function performs its checks correctly and throws errors when it receives garbage. This can be done with error blocks:

%!error foo ()  # test that causes any error
%!error <BAR must be a positive integer> foo (-1.5)  # test that throws specific error


Shared functions

It is often useful to share a function between multiple test. Sometimes these are only small helper functions, but more often these are just simpler low performance implementations of the function being tested. These are created in %!function blocks:


%!function x = slow_foo (bar)
%!  ## a simple implementation of foo, definitely correct, but
%!  ## unfortunately too slow for anything other than tests.
%!endfunction

%!assert (foo (bar), slow_foo (bar))

%!test
%! for i = -100:100
%!   bar = qux (i);
%!   assert (foo (bar), slow_foo (bar))
%! endfor


Code coverage