One of the big problems with maintaining a test plan is developers need to create a test plan document which is separate from the test scripts.
What, inevitably, ends up happening is that as the product evolves, bugs get fixed, etc, more test cases are added, but the test plan is not updated.
Even more problematic is that the design diverges and the HLD and/or requirements are not updated.
The LUTF provides a way to resolve this issue.
Each test script should include a documentation block formatted as follows:
""" @PRIMARY: sample_01 @SECONDARY: sample_01 @DESCRIPTION: Simple Hello Lustre test """
It's enclosed in """ .
- @PRIMARY: is the primary requirement this script fulfills
- @SECONDARY: is the secondary requirement, if any, this script fulfills
- @DESCRIPTION: Is a detailed description of the test case
By specifying these, the LUTF can generate automatic documentation for the test scripts. We no longer need to maintain a separate test plan. Our test scripts become our test plan.
This method provides the glue between the code and the documentation. As bugs are fixed or the feature updated, the developer should create a test case to cover the changes made.
Each test case should have a documentation block.
The LUTF then provides a method to extract the documentation block and create a table which can then be imported to the wiki. In this way it becomes much easier for the developer to maintain appropriate documentation without much overhead.
One way of working would be to create a skeleton of the scripts which just include the above comment block. Then run the create_docs() command on the suite
suites['samples'].create_docs("samples.csv")
This will generate a csv file with all the documentation.
This can then be imported directly into the Confluence wiki.
Once the test plan is reviewed, updated and approved, the script logic can be written in a separate phase.
Let's run through this process to show how easy it is.