Risk-minimising,
requirements-based testing of software systems (ranTEST)
sponsored by the BMBF in the context of the research initiative Software Engineering 2006 |
 |
Goals
The main goal of the ranTEST project is the development and validation of techniques for determining test cases that are required to check that different quality attributes (non-functional requirements) are satisfied. This especially includes their prioritisation with respect to different criteria. Testing the most relevant test cases first minimises the risk of not having adequately checked important quality attributes.
Besides a static prioritisation of test cases, dynamic approaches are developed to allow for a further optimisation of the test case selection. In the latter approach, test case priorities are adjusted depending on the results of previously executed test cases.
Overview
The general idea of the ranTEST project is depicted in figure 1.
Figure 1: Risk-minimising, requirements-based testing - ranTEST
Starting point for the ranTEST techniques are use cases, which describe user system interactions. These use cases are supplemented with quality requirements. The importance of these quality requirements is rated with respect to different criteria. These are (among others):
- severity of violating a quality attribute
- frequency of the use case occurrence
- complexity of the use case
- estimated test effort
- error history
From the use case scenarios, test cases for system testing are derived. First, test case scenarios are determined that additionally contain steps to guarantee a test case's precondition. Existing approaches to test case scenario derivation from requirements models are augmented by quality properties. Test case scenarios consist of a sequence of user inputs and the corresponding system responses. They have to be described in a way that is suitable for automated processing.
Test cases are derived based on the test case scenarios by determining concrete inputs and outputs. In particular, how often a test case must be executed has to be estimated, because many requirements can only be quantified statistically.
The derived test cases are prioritised with respect to the evaluation criteria of the corresponding scenarios. It is taken into account how often a test case must be executed in order to obtain a defined level of trust. Therefore, it can be beneficial to execute a test case less often than would be needed in an optimal case, because then resources are saved for test cases that otherwise would not be tested at all. The effort to execute a test case has to be estimated and taken into account during test case prioritisation. Test cases are executed according to their prioritisation. Therefore, on the one hand, a trade-off between test effort and remaining risk can be achieved and test resources can be estimated based on a desired system quality. On the other hand, if test resources are predefined it is possible to estimate the remaining risk not to meet a quality requirement.
Besides this static reduction of test cases before running tests, approaches to dynamically prioritise tests are developed. Such a dynamic prioritisation can allow for the increase or decrease of test coverage of certain parts of the software system, depending on prior test results.
Results
In phase 1 of the project, ranTEST was initialised successfully. The state of the art was determined with a literature research on the project’s topics. Furthermore, the state of the practice was evaluated with interviews concerning development and test of software systems. Thus, concrete research questions and requirements for the solutions that will be developed within the project have been derived. As a result, a deliverable describing the state of the art and the state of the practice regarding risk-based testing of quality attributes was completed.
A further result of the first phase of the project is a consortium-wide glossary supporting the common understanding of terms used within the ranTEST project. Elicitation of relevant terms was first performed separately by the project partners, followed by a consolidation during an internal workshop. The glossary is provided as a project internal Wiki allowing all partners to read and update the terms when it becomes necessary, e.g. due to advances in the ranTEST project.
In phase 2 of the project, a technique for the documentation of quality requirements and evaluation criteria suitable for industry was developed. After an identification of existing techniques for the documentation of quality requirements and evaluation criteria, these techniques were evaluated within a usability study together with the industry partners. On the basis of these results a technique for the documentation of quality requirements and evaluation criteria was specified. By developing a framework, the basis for prototypical tool development in the project was established and first tool prototypes have been developed. In this context, also a risk assessment tool was developed that provides an early and metric-based risk assessment.
In parallel, project phase 4 has developed a technique for the static prioritisation of test cases. In this project phase a survey for the test case derivation collecting existing approaches was developed. Moreover examples for the validation in project phase 5 were developed and documented.
Partners
- University of Duisburg-Essen, Software Systems Engineering
- Fraunhofer IESE
- market maker Software AG
- Siemens AG, Transportation Systems
Official Project Website
www.rantest.de (in German)
Person in Charge
Prof. Dr. Klaus Pohl
Contact
Heiko Stallbaum
Dr. Andreas Metzger
Publications
A list of all SSE publications can be found here.
© 2003-2009 - Software Systems Engineering
Disclaimer
Disclaimer
