Tests: Difference between revisions

Tests (edit)

Revision as of 19:17, 27 January 2022

7,489 bytes added , 27 January 2022

→‎Writing tests: added known failures and best practices section

Nrjank

154

edits

@@ Line 34: / Line 34: @@
 == Writing tests ==
+(Please see the latest version of the [https://octave.org/doc/latest/index.html Octave manual] for complete documentation on [https://octave.org/doc/v6.4.0/Test-and-Demo-Functions.html Test and Demo Functions].)
 Tests appear as <code>%!</code> blocks at the bottom of the source file, together with <code>%!demo</code> blocks.  A typical m function file, will have the following structure:
@@ Line 183: / Line 185: @@
 %! endfor
 </syntaxhighlight>
+=== Expected Failures and Known Bugs ===
+It is often the case that a test is developed that should pass for proper function, but is known to fail and cannot be immediately fixed.  In this case, the test can be included to provide documentation of the failure and expected behavior for future correction using either {{codeline|%!test}} or {{codeline|%!xtest}} as described below.
+An {{codeline|%!xtest}} block is a test that is expected to fail.  It is written and interpreted identically as a {{codeline|%!test}} block, with all of the same options available, but failure does not interrupt testing as a failing test would.  Total {{codeline|%!xtest}} failures are counted in the XFAIL total of the test summary as shown above.
+The following test block:
+<syntaxhighlight lang="Octave">
+%!test
+%!  assert (1+1, 2)
+%!xtest
+%!  assert (1+1, 3)
+</syntaxhighlight>
+will evaluate as:
+<syntaxhighlight lang="Octave">
+***** xtest
+ assert(1+1, 3)
+!!!!! known failure
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+PASSES 1 out of 2 test (1 known failure)</syntaxhighlight>
+An optional message can be added after {{codeline|%!test}} and {{codeline|%!xtest}} blocks that will be displayed if the test fails.  For example:
+<syntaxhighlight lang="Octave">
+%!test <good math>
+%!  assert (1+1, 2)
+%!xtest <bad math>
+%!  assert (1+1, 3)</syntaxhighlight>
+which can also be written as single line tests as:
+<syntaxhighlight lang="Octave">
+%!test <good math> assert (1+1, 2)
+%!xtest <bad math> assert (1+1, 3)</syntaxhighlight>
+replaces the message:
+<syntaxhighlight lang="Octave">
+!!!!! known failure </syntaxhighlight>
+with the message:
+<syntaxhighlight lang="Octave">
+!!!!! known bug: bad math</syntaxhighlight>
+A {{codeline|%!test}} block followed by a message will only be displayed if that test fails:
+<syntaxhighlight lang="Octave">
+***** test <good math> assert (1+1, 3)
+!!!!! known bug: good math
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+***** xtest <bad math> assert (1+1, 3)
+!!!!! known bug: bad math
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+PASSES 0 out of 2 tests (2 known bugs)
+</syntaxhighlight>
+If the <em><Message></em> is just a integer, Octave interprets this as a bug report id number that is expected to fail, and that {{codeline|%!test}} block is treated the same as an {{codeline|%!xtest}} block:
+<syntaxhighlight lang="Octave">
+%!test <12345> assert (1+1, 3)
+%!xtest <12345> assert (1+1, 3)
+</syntaxhighlight>
+produces:
+<syntaxhighlight lang="Octave">
+***** test <12345> assert (1+1, 3)
+!!!!! known bug: https://octave.org/testfailure/?12345
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+***** xtest <12345> assert (1+1, 3)
+!!!!! known bug: https://octave.org/testfailure/?12345
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+PASSES 0 out of 2 tests (2 known bugs)
+</syntaxhighlight>
+A {{codeline|%!test}} block with a <em><Message></em> that is an integer preceded by an asterisk (*) is interpreted as a bug report id number where the bug has been fixed.  Octave's build process automatically checks the status of bug reports and adds the "*" in all source files that contain tests tagged with bug numbers. Such blocks failing on later tests are flagged as regressions:
+<syntaxhighlight lang="Octave">
+%!test <*12345> assert (1+1, 3)
+</syntaxhighlight>
+produces:
+<syntaxhighlight lang="Octave">
+***** test <*12345> assert (1+1, 3)
+!!!!! regression: https://octave.org/testfailure/?12345
+ASSERT errors for:  assert (1 + 1,3)
+  Location  |  Observed  |  Expected  |  Reason
+     ()           2            3         Abs err 1 exceeds tol 0 by 1
+PASSES 0 out of 1 test
+</syntaxhighlight>
+=== Self-test Developer Best Practices ===
+* "Too many tests" is rarely a problem.
+* Input Validation:
+** When developing a function, it is good practice to include an "Input Validation tests" section that includes {{codeline|%!error}} tests for every input combination expected to produce an error, including the expected <error message> when possible.
+** Input validation tests should include number of inputs/outputs, input type/class, and any special handling (valid option names, values, etc.), including the possibility of "empty" and "NaN" inputs.
+* Error coverage: Ideally every call to error() in the function would include a test to ensure it is correctly reached and executed when the condition occurs.
+* Code path coverage - Include tests for every primary code function and major combination of inputs to reduce the number of future bugs reported by users when unexpected input combinations produce erroneous results.
+* Tests should verify proper function output (or appropriately informative error message) produced by:
+** different input shapes - scalars, vectors, arrays, or empty ([], ones(0,1), and ones(1,0) are not necessarily the same!)
+** types - numbers, booleans, strings, cells, multi-level cells, cell strings, structs, etc.
+** contents - real or complex, NaN, Inf, etc.
+* Functions that primarily rely on calling other functions with their own tests do not necessarily need to repeat all of the same tests. However, it may still be beneficial to do so if changes to the called function might produce errors that would not otherwise be caught by other tests.
+* Floating point calculations can cause failure of {{codeline|%!assert}} tests expecting exact equality.  Adding a tolerance on the order of {{codeline|%!eps}} or {{codeline|%!eps(variable)}} should rarely be a problem and is often sufficient to circumvent the issue. Depending on mathematical operations, it make be appropriate to use a tolerance several orders of magnitude larger than eps, but care should be taken in setting arbitrarily large tolerances that could hide acutal calculation errors.
+* Tests that ensure code compatibility with Matlab are very valuable for reducing future bug reports for incompatible behavior.  Because future behavior of Matlab functions can and do often change and those changes often go unnoticed until user bug reports appear, it can be useful to note with a comment which version of Matlab the test was verified against (or what the latest release was if the compatibility test is based on the public facing documentation).  As always, please ensure that any test code avoids the use of any copyrighted material.
+* Because of the automatic processing and regression tracking, {{codeline|xtest}} should only be used when there is an expected failure that has no related bug report.  It is preferred that all known bugs that cause function/test failures be reported so that a bug ID# is generated and a {{codeline|%!test <12345>}} format block can be used instead. Tests added to confirm bug fixes should use a {{codeline|%!test <*12345>}} block for automated regression tracking.
 [[Category:Testing]]
 [[Category:Development]]