Runtime Monitoring

Meta-Information

Origin: Dejan Nickovic /AIT, Udo Brockmeyer /BTC, Tino Teige / BTC

Written: April 2019

Purpose: Monitor at test environment runtime whether the SUT satisfies its specification.

Context/Pre-Conditions: an executable SUT with an observable interface.

To consider: Runtime monitoring is a versatile black-box technology that can be applied at all design stages and levels of abstraction (MiL, SiL, HiL, ViL, real system). Runtime monitoring does not address the problem of coverage and generation of stimuli for the SUT. It is often used in conjunction with coverage-based test generation methods.

Structure

Participants and Important Artefacts

Informal description of a requirement: : typically comes from the requirements document.

Formal requirement language: syntax and semantics, understandable by the runtime monitor generation.

Formal requirement: a valid sentence in the formal requirement language.

Runtime monitor desciption: executable artefact (compiled or interpreted SW, Verilog/VHDL code, etc.) that implements the monitoring algorithm from the formal requirement.

SUT desciption: executable artefact (Simulink model, compiled or interpreted SW, Verilog/VHDL code, etc.) that implements the SUT.

Trace: sequence of time-stamped values either provided as stimulus to the SUT (input) or generated by the SUT (output).

Verdict: evaluation result that indicates (the degree of) satisfaction/violation of the formal requirement by the observed trace. It can be a {pass, fail} verdict, but also a quantitative measure.

Actions/Collaborations

(1) Requirement formalization: typically manual process of translating informal into formal requirements. It can be sometimes supported by tools for elicitating requirements.

(2) Runtime monitor generation: translation of arbitrary requirements given in a formal requirement language into runtime monitor description.

(3a) Runtime monitor execution: activity that executes the runtime monitor and checks whether traces generated by the SUT satisfy or violate the formal requirement, resulting in a verdict.

(3b) SUT execution environment: activity that executes the SUT and environment model, transforming input traces into output traces according to the SUT function.

Discussion

Benefits: Runtime monitoring is a black-box technique that does not require a model of the system and can be applied at many levels of abstraction. The technology is also highly reusable across the product lifecycle.

Limitations: The analysis is done on individual traces, hence it cannot be used alone to gain the confidence about the system correctness.

Application hints: the observable interface must contain all signals which are referenced in the formal requirements.

Application Example

ENABLE-S3 Use Case 6 “Valet Parking”:

Examples for values in traces are
xxxInput Trace: positions, speed, and moving directions of other moving objects in valet parking zone,
xxxOutput Trace: position, speed, and turning angle of ego car (SuT).

 

Relations to other Patterns

Pattern Name	Relation*
Formalized Analysis and Verification	Both patterns need formalized requirement presentation (called “dependability property” there)
KPI Definition	That pattern can help to select relevant observables
Scenario-Representativeness Checking	That pattern can help to select relevant scenarios for runtime monitoring

* “this pattern” denotes the pattern described here, “that pattern” denotes the related pattern