Difference between revisions of "Tests"

From Octave
Jump to navigation Jump to search
m (Minor text revision and corrections, add category Category:Testing.)
(→‎Writing tests: prettify using syntaxhighlighting.)
Line 50: Line 50:
 
== Writing tests ==
 
== Writing tests ==
  
Tests appear as {{codeline|%!}} blocks at the bottom of the source file,
+
Tests appear as <code>%!</code> blocks at the bottom of the source file, together with <code>%!demo</code> blocks.  A typical m function file, will have the following structure:
together with {{codeline|%!demo}} blocks.  A typical m function file, will
 
have the following structure:
 
  
## Copyright
+
<syntaxhighlight lang="Octave">
##
+
## Copyright
## A block with the copyright notice
+
##
+
## A block with the copyright notice
## -*- texinfo -*-
+
 
##
+
## -*- texinfo -*-
## A block with the help text
+
##
+
## A block with the help text
function [x, y, z] = foo (bar)
+
 
  ## here's some amazing code
+
function [x, y, z] = foo (bar)
endfunction
+
  ## some amazing code
+
endfunction
%!assert (foo (1))
+
 
%!assert (foo (1:10))
+
%!assert (foo (1))
%!assert (foo ("on"), "off")
+
%!assert (foo (1:10))
%!error <must be positive integer> foo (-1)
+
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (1.5)
+
%!error <must be positive integer> foo (-1)
+
%!error <must be positive integer> foo (1.5)
%!demo
+
 
%! ## see how cool foo() is:
+
%!demo
%! foo([1:100])
+
%! ## see how cool foo() is:
 +
%! foo([1:100])
 +
</syntaxhighlight>
  
 
Tests can be added to oct functions in the C++ sources just as easily, see
 
Tests can be added to oct functions in the C++ sources just as easily, see
Line 83: Line 83:
 
following structure:
 
following structure:
  
// Copyright
+
<syntaxhighlight lang="c++">
//
+
// Copyright
// A block with the copyright notice
+
//
 +
// A block with the copyright notice
 
   
 
   
DEFUN_DLD (foo, args, ,
+
DEFUN_DLD (foo, args, ,
"-*- texinfo -*-\n\
+
"-*- texinfo -*-\n\
A block with the help text")
+
A block with the help text")
{
+
{
   \\ here's some amazing code
+
   // some amazing code
}
+
}
function [x, y, z] = foo (bar)
 
  ## here's some amazing code
 
endfunction
 
 
   
 
   
/*
+
/*
%!assert (foo (1))
+
%!assert (foo (1))
%!assert (foo (1:10))
+
%!assert (foo (1:10))
%!assert (foo ("on"), "off")
+
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (-1)
+
%!error <must be positive integer> foo (-1)
%!error <must be positive integer> foo (1.5)
+
%!error <must be positive integer> foo (1.5)
*/
+
*/
 
+
</syntaxhighlight>
  
 
=== Assert ===
 
=== Assert ===
Line 111: Line 109:
 
common:
 
common:
  
%!assert (foo (bar))      # test fails if "foo (bar)" returns false
+
<syntaxhighlight lang="Octave">
%!assert (foo (bar), qux) # test fails if "foo (bar)" is different from "qux"
+
%!assert (foo (bar))      # test fails if "foo (bar)" returns false
 +
%!assert (foo (bar), qux) # test fails if "foo (bar)" is different from "qux"
 +
</syntaxhighlight>
  
 
These are actually a shorthand version of
 
These are actually a shorthand version of
Line 122: Line 122:
 
While single {{codeline|%!assert}} lines are the most common used tests, {{codeline|%!test}} blocks offer more features and flexibility.  The code within {{codeline|%!test}} blocks is simply processed through the Octave interpreter.  If the code generates an error, the test is said to fail.  Often {{codeline|%!test}} blocks end with a call to {{codeline|assert}}:
 
While single {{codeline|%!assert}} lines are the most common used tests, {{codeline|%!test}} blocks offer more features and flexibility.  The code within {{codeline|%!test}} blocks is simply processed through the Octave interpreter.  If the code generates an error, the test is said to fail.  Often {{codeline|%!test}} blocks end with a call to {{codeline|assert}}:
  
%!test
+
<syntaxhighlight lang="Octave">
%! a = [0 1 0 0 3 0 0 5 0 2 1];
+
%!test
%! b = [2 5 8 10 11];
+
%! a = [0 1 0 0 3 0 0 5 0 2 1];
%! for i = 1:5
+
%! b = [2 5 8 10 11];
%!  assert (find (a, i), b(1:i))
+
%! for i = 1:5
%! endfor
+
%!  assert (find (a, i), b(1:i))
 +
%! endfor
 +
</syntaxhighlight>
  
 
==== Test for no failure ====
 
==== Test for no failure ====
Line 135: Line 137:
 
simply with:
 
simply with:
  
%!test foo (bar)
+
<syntaxhighlight lang="Octave">
 +
%!test foo (bar)
 +
</syntaxhighlight>
  
 
=== Error / Warning ===
 
=== Error / Warning ===
Line 143: Line 147:
 
{{codeline|error}} (or {{codeline|warning}}) blocks:
 
{{codeline|error}} (or {{codeline|warning}}) blocks:
  
%!error foo ()  # test that causes any error
+
<syntaxhighlight lang="Octave">
%!error <BAR must be a positive integer> foo (-1.5)  # test that throws specific error message
+
%!error foo ()  # test that causes any error
%!error id=Octave:invalid-fun-call foo ()  # test that throws specific error id
+
%!error <BAR must be a positive integer> foo (-1.5)  # test that throws specific error message
 +
%!error id=Octave:invalid-fun-call foo ()  # test that throws specific error id
  
%!warning foo ()  # test that causes any warning
+
%!warning foo ()  # test that causes any warning
%!warning <negative values might give inaccurate results> foo (-1.5)  # test that triggers a specific warning message
+
%!warning <negative values might give inaccurate results> foo (-1.5)  # test that triggers a specific warning message
%!warning id=BAR:possibly-inaccurate-result foo (-1.5)  # test that triggers a specific warning id
+
%!warning id=BAR:possibly-inaccurate-result foo (-1.5)  # test that triggers a specific warning id
 +
</syntaxhighlight>
  
 
=== Shared functions ===
 
=== Shared functions ===
Line 158: Line 164:
 
created in {{codeline|%!function}} blocks:
 
created in {{codeline|%!function}} blocks:
  
 +
<syntaxhighlight lang="Octave">
 +
%!function x = slow_foo (bar)
 +
%!  ## a simple implementation of foo, definitely correct, but
 +
%!  ## unfortunately too slow for anything other than tests.
 +
%!endfunction
  
%!function x = slow_foo (bar)
+
%!assert (foo (bar), slow_foo (bar))
%!  ## a simple implementation of foo, definitely correct, but
 
%!  ## unfortunately too slow for anything other than tests.
 
%!endfunction
 
 
%!assert (foo (bar), slow_foo (bar))
 
 
%!test
 
%! for i = -100:100
 
%!  bar = qux (i);
 
%!  assert (foo (bar), slow_foo (bar))
 
%! endfor
 
  
 +
%!test
 +
%! for i = -100:100
 +
%!  bar = qux (i);
 +
%!  assert (foo (bar), slow_foo (bar))
 +
%! endfor
 +
</syntaxhighlight>
  
 
[[Category:Testing]]
 
[[Category:Testing]]
 
[[Category:Development]]
 
[[Category:Development]]

Revision as of 05:05, 31 July 2017

Having a thorough test suite is something very important which is usually overlooked. It is an incredible help in preventing regression bugs and quickly assess the status of old code. For example, many packages in Octave Forge become deprecated after losing their maintainer simply because they have no test suite.

GNU Octave has multiple tools that help in creating a comprehensive test suite, accessible to both developers and end-users, as detailed on the Octave manual. Basically, test blocks are %!test comment blocks, typically at the end of a source file, which are ignored by the Octave interpreter and only read by the test function.

Running tests

To run all the tests of a specific function, simply use the test command at the Octave prompt. For example, to run the tests in mean():

>> test mean
PASSES 17 out of 17 tests

These tests are written in the Octave language at the bottom of the m file which defines mean(). It is important that these tests are also available for the end users so they can test the status of their installation. The whole Octave test suite can be ran with:

>> __run_test_suite__

Integrated test scripts:

[...]

Summary:

  PASS     11556
  FAIL         3
  XFAIL        6
  SKIPPED     38

See the file test/fntests.log for additional details.

To run tests in a specific file, one can simply specify the path instead of a function name:

 test /full/path/to/file.cc


Writing tests

Tests appear as %! blocks at the bottom of the source file, together with %!demo blocks. A typical m function file, will have the following structure:

## Copyright
##
## A block with the copyright notice

## -*- texinfo -*-
##
## A block with the help text

function [x, y, z] = foo (bar)
  ## some amazing code
endfunction

%!assert (foo (1))
%!assert (foo (1:10))
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (-1)
%!error <must be positive integer> foo (1.5)

%!demo
%! ## see how cool foo() is:
%! foo([1:100])

Tests can be added to oct functions in the C++ sources just as easily, see find.cc for example. The syntax is exactly the same, but done within C comment blocks. During installation, these lines are automatically extracted from the sources and special test scripts are generated. A typical C++ source file has the following structure:

// Copyright
//
// A block with the copyright notice
 
DEFUN_DLD (foo, args, ,
"-*- texinfo -*-\n\
A block with the help text")
{
  // some amazing code
}
 
/*
%!assert (foo (1))
%!assert (foo (1:10))
%!assert (foo ("on"), "off")
%!error <must be positive integer> foo (-1)
%!error <must be positive integer> foo (1.5)
*/

Assert

%!assert lines are the simplest tests to write and also the most common:

%!assert (foo (bar))      # test fails if "foo (bar)" returns false
%!assert (foo (bar), qux) # test fails if "foo (bar)" is different from "qux"

These are actually a shorthand version of %!test assert (foo (bar)), and assert is simply an Octave function that throws an error when two arguments fail to compare.

Test

While single %!assert lines are the most common used tests, %!test blocks offer more features and flexibility. The code within %!test blocks is simply processed through the Octave interpreter. If the code generates an error, the test is said to fail. Often %!test blocks end with a call to assert:

%!test
%! a = [0 1 0 0 3 0 0 5 0 2 1];
%! b = [2 5 8 10 11];
%! for i = 1:5
%!   assert (find (a, i), b(1:i))
%! endfor

Test for no failure

In a few cases, there is the situation where a function returns nothing, and the only thing to test is that it causes no error. This can be tested simply with:

%!test foo (bar)

Error / Warning

It is also important to test that a function performs its checks correctly and throws errors (or warnings) when it receives garbage. This can be done with error (or warning) blocks:

%!error foo ()  # test that causes any error
%!error <BAR must be a positive integer> foo (-1.5)  # test that throws specific error message
%!error id=Octave:invalid-fun-call foo ()  # test that throws specific error id

%!warning foo ()  # test that causes any warning
%!warning <negative values might give inaccurate results> foo (-1.5)  # test that triggers a specific warning message
%!warning id=BAR:possibly-inaccurate-result foo (-1.5)  # test that triggers a specific warning id

Shared functions

It is often useful to share a function between multiple test. Sometimes these are only small helper functions, but more often these are just simpler low performance implementations of the function being tested. These are created in %!function blocks:

%!function x = slow_foo (bar)
%!  ## a simple implementation of foo, definitely correct, but
%!  ## unfortunately too slow for anything other than tests.
%!endfunction

%!assert (foo (bar), slow_foo (bar))

%!test
%! for i = -100:100
%!   bar = qux (i);
%!   assert (foo (bar), slow_foo (bar))
%! endfor