Test Plan Specification
Meta-Information
Origin: Gustavo Garcia Padilla / Hella, Sadri Hakki / Hella
Written: May 2019
Purpose: Specification of a test plan for a test project/test sub-process using a generic template based on the Standard for Software Test Documentation ISO/IEC/IEEE 29119-3. The generic template includes in the same document the specification of test design, test data, test cases and test procedures.
Context/Pre-Conditions: This pattern is usable to generate a test plan for the functional and non functional requirements of the System under Test (SuT). Depending on the testing levels, objectives and strategies different different XiL instantiations of the test platform (e.g. MiL, SiL, HiL) and testing techniques can be considered. The pattern was applied by HAGL for the testing of ADAS functions and tools in the automotive domain and is being applied by the UC10 partners for the description of how tests in the Maritime Co-Simulation Platform validation development project are planned to be carried out.
To consider: The template is a proposal for a fast, simple and complete specification of a test plan and has to be adapted for the specific the test project/test sub-process. Expert knowledge in testing and expert knowledge about the SuT is required. The order of appearance of the blocks in the document can vary, e.g. the glossary can be put at the end of the document.
Structure
Note: yellow boxes describe activities to establish/achieve the noted artefacts.
Refinement of upper process:
.
Participants and Important Artefacts
Test Expert: Responsible person for preparing the test plan with expert knowledge in testing.
Requirement Specification: Document defining the functional and non functional requirements of the SuT.
Use Case Specification: Document describing the use case of the SuT.
Risk Analysis: A table with risk levels (not safety related failure consequence) of individual SW and system parts. This document is the result from the not safety related related risk analysis and is used to assign priorities to the test cases.
Safety Analysis: A table with the Safety Class assignement (e.g. ASIL in the automotive domain) of individual SW and system parts, to be used to assign more or less test cases and the test types to the corresponding safety requirements of the individual SW or system part (e.g. following the recommendation of ISO26262 ). Can be used in combination with the risk analysis to assign priorities to the use cases.
Actions and collaborations
(1) Introduction: Unic document name and purpose of the document, change service, definitions/glossary, references.
(2) Objectives: A table defining the goals of the test project adressed in this test plan against the SuT, e.g. correctness, stability, time response, error handling.
(3) General Framework: List of documents building the general framework for planning, specification and execution of the tests (e.g. Milestone Plan), Description of the Test Objects, List of documents necessary to specify the test procedure and the test cases for this test project (Test Basis), List of Features/Requirements to test.
(4) Test Strategy: A table refining the goals of the test project for the features/requirements to test including the logical test description, test automation, criteria for pass/fail and end of test, risk evaluation (e.g. ASIL assignement) of individual SW and system parts.
(5) Test Organization: Test Team, Test Schedule, Traceability information, References to the test work products Test-Design / Test case specification / Test procedure specification / Test data/ Test report.
(6) Refinement of the Test Approach: The test goals of chapter 4.1 “Achievement of the test goals” will be refined in this chapter: a table is given how the individual test goals are tested (test procedure) and how many test cases are necessary.
(7) Test Case Specification: Only the definition of the necessary test case attributes and a test case example should be given in the test plan document. The test cases should be specified in a separated document as a table (e.g. Excel table) with the test cases in the rows and the attributes in the columns. The filtering features of the Excel table are very helpful. The technical test procedure specification is defined as attribute of a test case and is part of the test case specification.
Discussion
Benefits: The pattern is based on the standard ISO/IEC/IEEE 29119 with the corresponding advantages of systematic consideration of best experiences, efficiency, cost reduction, common understanding and continuous updating. Each activity of the test plan can be reviewed and freezed to avoid rework at a later activity. For example if the test strategy (4) is fixed then the refinement of the test approach (6) can start. If (6) or a part is finished or freezed then the corresponding part of the test case specification (7) can start. The pattern helps to avoid the specifing too much or too less test cases. The test automation is also a very important factor to consider for reduction of costs and test time.
Limitations: The application of the pattern for the specific test project requires expert testing knowledge and expert knowlege about the SuT.
Application Examples
ENABLE-S3 Use Case 6 “Valet Parking”: The pattern is used by the partners for the preparation of a test plan for the functional safety requirements of the SuT “Valet Parking Function”.
ENABLE-S3 Use Case 10 “Shore-based Bridge”: The pattern is applied for the description of how tests in the Maritime Co-Simulation Platform validation development project are planned to be carried out.
The pattern was further applied by HAGL to test ADAS functions and tools in the automotive domain.
.
Relations to other Patterns
As this is a general solution pattern, it is related to basically all other patterns by putting them into the overall test framework.