153
edits
Tests: Difference between revisions
Jump to navigation
Jump to search
→Self-test Developer Best Practices: typo/grammar fixes
m (→Assert: added tolerance info) |
(→Self-test Developer Best Practices: typo/grammar fixes) |
||
| Line 292: | Line 292: | ||
* Input Validation: | * Input Validation: | ||
** When developing a function, it is good practice to include an "Input Validation tests" section that includes {{codeline|%!error}} tests for every input combination expected to produce an error, including the expected <error message> when possible. | ** When developing a function, it is good practice to include an "Input Validation tests" section that includes {{codeline|%!error}} tests for every input combination expected to produce an error, including the expected <error message> when possible. | ||
** Input validation | ** Input validation should test the number of inputs/outputs, input type/class, and any special handling (valid option names, values, etc.), including the possibility of "empty" and "NaN" inputs. | ||
* Error coverage: Ideally every call to error() in the function would include a test to ensure it is correctly reached and executed when the condition occurs. | * Error coverage: Ideally every call to error() in the function would include a test to ensure it is correctly reached and executed when the condition occurs. | ||
* Code path coverage - Include tests for every primary code function and major combination of inputs to reduce the number of future bugs reported by users | * Code path coverage - Include tests for every primary code function and major combination of inputs to reduce the number of future bugs reported by users. | ||
* Tests should verify proper function output (or appropriately informative error message) produced by: | * Tests should verify proper function output (or appropriately informative error message) produced by: | ||
** different input shapes - scalars, vectors, arrays, or empty ([], ones(0,1), and ones(1,0) are not necessarily the same!) | ** different input shapes - scalars, vectors, arrays, or empty ([], ones(0,1), and ones(1,0) are not necessarily the same!) | ||
** types - numbers, booleans, strings, cells, multi-level cells, cell strings, structs, etc. | ** types - numbers, booleans, strings, cells, multi-level cells, cell strings, structs, etc. | ||
** contents - real or complex, NaN, Inf, etc. | ** contents - real or complex, NaN, Inf, etc. | ||
* Functions that primarily rely on calling other functions with their own tests do not necessarily need to repeat all of the same tests. However, it may still be beneficial to do so if changes to the called function | * Functions that primarily rely on calling other functions with their own tests do not necessarily need to repeat all of the same tests. However, it may still be beneficial to do so if changes to the called function could produce errors that would not otherwise be caught by other tests. | ||
* Floating point calculations can cause failure of {{codeline|%!assert}} tests expecting exact equality. Adding a tolerance on the order of {{codeline|%!eps}} or {{codeline|%!eps(variable)}} should rarely be a problem and is often sufficient to circumvent the issue. Depending on mathematical operations, it | * Floating point calculations can cause failure of {{codeline|%!assert}} tests expecting exact equality. Adding a tolerance on the order of {{codeline|%!eps}} or {{codeline|%!eps(variable)}} should rarely be a problem and is often sufficient to circumvent the issue. Depending on mathematical operations, it may be appropriate to use a tolerance several orders of magnitude larger than eps, but care should be taken in setting arbitrarily large tolerances that could hide actual calculation errors. | ||
* Tests that ensure code compatibility with Matlab are very valuable for reducing future bug reports for incompatible behavior. Because future behavior of Matlab functions can and do often change and those changes often go unnoticed until user bug reports appear, it can be useful to note with a comment which version of Matlab the test was verified against (or what the latest release was if the compatibility test is based on the public facing documentation). As always, please ensure that | * Tests that ensure code compatibility with Matlab are very valuable for reducing future bug reports for incompatible behavior. Because future behavior of Matlab functions can and do often change, and those changes often go unnoticed until user bug reports appear, it can be useful to note with a comment which version of Matlab the test was verified against (or what the latest release was if the compatibility test is based on the public facing documentation). As always, please ensure that test code avoids the use of any copyrighted material. | ||
* Because of the automatic processing and regression tracking, {{codeline|xtest}} should only be used when there is an expected failure that has no related bug report. It is preferred that all known bugs that cause function/test failures be reported so that a bug ID# is generated and a {{codeline|%!test <12345>}} format block can be used instead. Tests added to confirm bug fixes should use a {{codeline|%!test <*12345>}} block for automated regression tracking. | * Because of the automatic processing and regression tracking, {{codeline|xtest}} should only be used when there is an expected failure that has no related bug report. It is preferred that all known bugs that cause function/test failures be reported at [https://bugs.octave.org bugs.octave.org] so that a bug ID# is generated and a {{codeline|%!test <12345>}} format block can be used instead. Tests added to confirm bug fixes should use a {{codeline|%!test <*12345>}} block for automated regression tracking. | ||
[[Category:Testing]] | [[Category:Testing]] | ||
[[Category:Development]] | [[Category:Development]] | ||