Scenario-based V&V Process
Origin: Eike Möhlmann / OFFIS, Sebastian Gerwinn / OFFIS
Written: May 2019
Purpose: Scenario-based Semi-virtual verification and validation of a SUT. This pattern covers the whole process from operational scenario mining over requirement elicitation, abstract scenario design to verification of a given SUT.
Participants and Important Artefacts
Abstract Scenarios DB: A set or catalog of abstract scenarios assumed to be a proxy for the real world, e.g. “left turns on two lane city road intersection”.
Criticality Indicator DB: A set of indicators and causes for requirement violations (like accidents or near-accident situations). These indicators can be used to improve scenario selection and simulation guiding during the (semi-)virtual testing.
Representativeness measure / Confidence in the coverage: Estimated probability that a (random) real world data point can be represented with an abstract scenario and vice versa. This estimate is valid up to a known confidence. As the recorded data is used as a proxy for the real world, the estimated probability measures the likelihood relative to this recording process.
Estimated Remaining Risk: Estimated probability that the requirements are not satisfied (with a confidence level).
Requirements and Scenarios Elicitation: Combined elicitation of requirements and the scenario in which they have to be satisfied. See also that sub-pattern..
Criticality Analysis: Analyze the abstract scenarios regarding the requirements to identify indicators and causes of requirement violation (e.g. accidents or near-accidents).
Semi-virtual Testing: Estimation of the risk that a given SUT* violates a given catalog of requirements. See also that sub-pattern.
Establish (Overall) Verification Argument: Given an estimation on the satisfaction of the requirements and the representativeness measure, estimate the remaining risk that the requirements will be violated in the real world.
Benefits: Once a scenario catalog and criticality indicators have been shown to be sufficiently representative for a set of requirements, the set of scenarios can be re-used for a continuous design/development process together with a (semi-)virtual verification and validation.
– As this process favors hypothesis-testing the result is influenced by parameters characterizing the statistical power of the tests (sensitivity, significance).
– The result of the final risk assessment is relative to the quality of the recorded data.
It is applied as overall methodology for scenario-based V&V in e.g. ENABLE-S3 Use Case 6 “Valet Parking”.
Relations to other Patterns
|Requirements and Scenarios Elicitation
|This pattern is super-pattern of the “Requirements and Scenarios Elicitation” pattern
|This pattern is super-pattern of the “Semi-Virtual Testing” pattern
|Test Plan Specification
|This pattern describes a certain way to realize activities (4)…(7) in that pattern
* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern