Scenario-based V&V Process

Meta-Information

Origin: Eike Möhlmann / OFFIS, Sebastian Gerwinn / OFFIS

Written: May 2019

Purpose: Scenario-based Semi-virtual verification and validation of a SUT. This pattern covers the whole process from operational scenario mining over requirement elicitation, abstract scenario design to verification of a given SUT.

Structure

Participants and Important Artefacts

Requirements DB: A set of requirements on the system-under-test which have been derived from analyzing e.g. operational scenarios, laws, regulations.

Abstract Scenarios DB: A set or catalog of abstract scenarios assumed to be a proxy for the real world, e.g. “left turns on two lane city road intersection”.

Deployment Context: A description of the context in which the SUT will be deployed. This restricts the environment to the relevant operation conditions (like the highway for the highway chauffeur).

Criticality Indicator DB: A set of indicators and causes for requirement violations (like accidents or near-accident situations). These indicators can be used to improve scenario selection and simulation guiding during the (semi-)virtual testing.

Representativeness measure / Confidence in the coverage: Estimated probability that a (random) real world data point can be represented with an abstract scenario and vice versa. This estimate is valid up to a known confidence. As the recorded data is used as a proxy for the real world, the estimated probability measures the likelihood relative to this recording process.

Estimated Requirement Satisfaction: Estimated probability that a (random) (semi-)virtual run (system-under-test in the virtual world) will satisfy the requirements.

Estimated Remaining Risk: Estimated probability that the requirements are not satisfied (with a confidence level).

Actions/Collaborations

Requirements and Scenarios Elicitation: Combined elicitation of requirements and the scenario in which they have to be satisfied. See also that sub-pattern..

Criticality Analysis: Analyze the abstract scenarios regarding the requirements to identify indicators and causes of requirement violation (e.g. accidents or near-accidents).

Semi-virtual Testing: Estimation of the risk that a given SUT* violates a given catalog of requirements. See also that sub-pattern.

Establish (Overall) Verification Argument: Given an estimation on the satisfaction of the requirements and the representativeness measure, estimate the remaining risk that the requirements will be violated in the real world.

Discussion

Benefits: Once a scenario catalog and criticality indicators have been shown to be sufficiently representative for a set of requirements, the set of scenarios can be re-used for a continuous design/development process together with a (semi-)virtual verification and validation.

Comments:

– As this process favors hypothesis-testing the result is influenced by parameters characterizing the statistical power of the tests (sensitivity, significance).

– The result of the final risk assessment is relative to the quality of the recorded data.

Application Examples

It is applied as overall methodology for scenario-based V&V in e.g. ENABLE-S3 Use Case 6 “Valet Parking”.

Relations to other Patterns

Pattern Name	Relation*
Requirements and Scenarios Elicitation	This pattern is super-pattern of the “Requirements and Scenarios Elicitation” pattern
Semi-Virtual Testing	This pattern is super-pattern of the “Semi-Virtual Testing” pattern
Test Plan Specification	This pattern describes a certain way to realize activities (4)…(7) in that pattern

* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern