(Semi-)virtual Testing

Meta-Information

Written: March 2019

Purpose: Estimation of the risk that a given SUT* violates a given catalog of requirements.

To consider: The quality of the estimation depends on the quality of the used simulation environment and the abstract scenarios catalog.

Structure

Participants and Important Artefacts

Test Field: Recorded/Measured data from the real world in the deployment context of the SUT.

Abstract Scenarios DB: A catalog of abstract scenarios which shall be qualified to be a proxy for the real world, e.g. left turns on two lane city road intersection.

Set of Synthetic Scenarios: A set of concrete scenarios that has been derived by randomly instantiating abstract scenarios.

Requirements DB: A set of requirements under which the abstract scenario catalog will be compared to the real world scenarios.

Criticality Indicators DB: A set of indicators and causes for requirement violations (like accidents or near-accident situations). These indicators can be used to improve scenario selection and simulation guiding during the (semi-)virtual testing.

Estimated Requirement Satisfaction: Estimated probability that a (random) (semi-)virtual run (of the system-under-test in the virtual world) will satisfy the requirements.

Actions/Collaborations

Generate Synthetic Scenarios: Generate a set of synthetic scenarios by randomly (i.i.d.*) instantiating abstract scenarios from the abstract scenario catalog.

Setup of Physical Components: Setup of the physical components that are required for executing the test cases corresponding to the set of synthetic scenarios.

Setup of Virtual Components: Setup of the physical components that are required for executing the test cases corresponding to the set of synthetic scenarios.

Guided Simulation with Requirement Monitoring: Execute the test cases corresponding to the set of synthetic scenarios and observe the satisfaction of the requirements to estimate the probability that the SUT violates the requirements in a random scenario (i.i.d.).

* independently and identically distributed

Discussion

Benefits:

– The (semi-) virtual testing allows testing of components in isolation with known-to-work reference components as well as reduces the effort for setup and execution of the test.

– The (semi-) virtual testing allows testing of critical scenarios in which human and property are at risk.

Comment: As this is a hypothesis-test the result is influenced by parameters characterizing the statistical power of the test (sensitivity, significance).

Application Examples

The pattern was applied for estimating the risk that the SUT (an autonomous driving function) will collide with pedestrians in following use cases:

ENABLE-S3 UC2 “Intersection Crossing using ACPS”

ENABLE-S3 UC6 “Valet Parking”:

Relations to other Patterns

Pattern Name	Relation*
Scenario-based V&V Process	This pattern uses the scenarios catalog developed with that pattern

* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern