Closed-Loop Testing

Meta-Information

Origin: Wolfgang Herzner / AIT, David Gonzalez-Arjona / GMV

Written: April 2019

Purpose: Validate the SUT (System under Test) in several closed-loop testing steps. In closed-loop testing, SUT‘s output is fed back into the testing environment, to simulate the influence of a SUT‘s behavior onto its real environment. As this technique can be applied on several levels of abstraction, it is also named as ‘XiL’ (X in the loop).

Context/Pre-Conditions: SUT and environment need to be available as model as well as real system.

To consider: This pattern may be considered for the construction of a SW & HW target.

Structure

Participants and Important Artefacts

MiL (Model in the Loop), SiL (Software in the Loop), HiL (Hardware in the Loop): sub-patterns where the actual work is carried out.

Engineer (expert): In charge of the quality surveillance of the different steps.

SUT Model: Formal description of the system to be tested, specifying all its characteristics (structure, behavior, …) relevant for the intended V&V steps.

Environment Model: formal description of all environmental aspects that allow for calculating the impact of the SUT’s behavior (output) on the environment and feeding back results to the SUT(-model). If the SUT is component of a larger system, relevant aspects of the remaining parts of that system are also modelled.

SUT (Target): real SUT.

Environment Simulation Hardware (HiL System): physical items simulating relevant environmental aspects in real time, in particular their reaction on SUT’s behavior. If the SUT is component of a larger system, the remaining parts of that system are also simulated.

Uncovered requirements / test cases: in general, during test plan specification (also a V&V-pattern) it is decided which SUT features are tested where. However, it may turn out during MiL or SiL that certain proves or tests planned for that level cannot be achieved there, e.g. due to insufficient computing power. These have then to be propagated to next lower level.

Actions/Collaborations

MiL: Execution of the SUT model against the environment model, purely virtual, not real-time.

SiL: Execution of the target SUT against the environment model, at least of SUT‘s software. Not necessarily real-time.

HiL: Execution of the target SUT in a hardware-based environment that allows real-time.

Evaluate Results: Assessing whether results of the respective XiL execution meets the requirements. If not, usually the SUT is modified (corrected) according to results and the step repeated. If the test environment is inappropriate to achieve the required measures, but can be improved, this is done and the step repeated. Otherwise, the uncovered aspects are propagated to next lower level.

Discussion

Benefits:

– Gradual platform migration from Models to final target.

– Quality surveillance during system construction.

Limitations: Not all the steps could be suitable for every SUT. In particular, for hardware-intensive SUTs SiL is usually applicable only to its software components. Or, hardware components are either simulated or interfaced with environment model in a way that bridges potential not real-time / real-time gaps.

Variants: The described categorization into three distinct levels mainly aims at simplification. In practice, many intermediate forms are possible (see application examples).

Application Example

ENABLE-S3 UC8 “Reconfigurable Video Processors for Space”:

– Several integration steps in the TSY

– Use of PC, SW, Real-Time Processor and FPGA

.

Relations to other Patterns

Pattern Name	Relation*
MiL (Model in the Loop)	Description of MiL. Is sub-pattern of this pattern
SiL (Software in the Loop)	Description of SiL. Is sub-pattern of this pattern
HiL (Hardware in the Loop)	Description of HiL. Is sub-pattern of this pattern
Test Plan Specification	Supports allocation of requirements to XiL-levels
Scenario-based V&V Process	Supports allocation of test scenarios / cases to XiL-level

* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern