Scenario-Representativeness Checking

Meta-Information

Origin: Eike Möhlmann / OFFIS, Sebastian Gerwinn / OFFIS, Gustavo Garcia Padilla / Hella

Written: May 2019

Purpose: Quantification how well a set of abstract scenarios represents the real world. It determines the degree to which this set covers situations which could be observed in the real world and vice-versa. Specifically, it is a hypothesis-test whether the given set can be used as a proxy for the real world (representativeness) regarding tests against a given set of requirements.

Context/Pre-Conditions: Level of abstraction of collected real word data and scenario catalog has to match, such that it enables a later mapping.

To consider: Representativeness indicates how good a catalog of abstract scenarios matches a DB of recorded real world data. Hence, it directly states how good results obtain using one catalog can be transferred to the other. The better the representativeness the closer will be the results.

Structure

Participants and Important Artefacts

Real World Data DB: Recorded/Measured data from the real world in the deployment context of the SUT.

Abstract Scenarios DB: A catalog of abstract scenarios which shall be qualified to be a proxy for the real world, e.g. “left turns on two lane city road intersection“.

Set of Synthetic Scenarios: A set of concrete scenarios that has been derived by randomly instantiating abstract scenarios.

Requirements DB: A set of requirements under which the abstract scenario catalog will be compared to the real world scenarios.

Representativeness measure / Confidence in the coverage: Estimated probability that a (random) real world data point can be represented with an abstract scenario. This estimate is valid up to a known confidence. As the recorded data is used as a proxy for the real world, the estimated probability measures the likelihood relative to this recording process.

Actions/Collaborations

Synthetic Scenarios Generation: Synthetically generates a set of concrete scenario instances from a catalog of abstract scenarios.

Bi-directional Mapping of Scenarios regarding Requirements: Given a set real world scenarios and a set of synthetic scenarios, the mapping tries to identify whether the two sets are obtained from different processes. The mapping compares the scenarios only regarding the given set of requirements.

Discussion

Benefits: Once a scenarios catalog/DB has been shown to be sufficiently representative, the catalog can be re-used for a continuous design process including the (semi-)virtual testing as a proxy for physical testing. Hence, physical tests can be replaced by virtual tests.

Comment: As this is a hypothesis-test the result is influenced by parameters characterizing the statistical power of the test (sensitivity, significance).

Application Examples

This pattern was applied for qualification of a scenarios catalog for which also the quality (in terms of representativeness) is estimated. While the coverage estimation method itself is domain-independent, the scenario-database is specific for the domain, in following use cases:

ENABLE-S3 UC2 “Intersection Crossing using ACPS”

ENABLE-S3 UC6 “Valet Parking”

Relations to other Patterns

Pattern Name	Relation*
Requirement and Scenario Elicitation	This pattern evaluates relevance of scenarios and requirements coverage for its super-pattern “Requirement and Scenario Elicitation”
KPI-Models based Validation	This pattern can be used to establish representativeness measures for the “KPI-Models based Validation” pattern

* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern