Terms and Definitions

The Relation between Test Case and Scenario

Test case specifications and scenario descriptions are important document types in automotive testing across all different kinds of testing methodologies (from requirements based to interfaces testing, fault injection, resource usage, and so on).

For example, in requirements-based testing methodology, the use of test case descriptions has proven successful in specifying requirements, in test bench configuration, and in defining test steps and expected result of test runs. As another example, scenarios have become increasingly important in the context of scenario-based testing using simulation (also known as virtual testing) and other relevant test environments for the homologation of ADAS/AD (reference to the test matrix in the report). The scenario-based testing methodology for ADS is evolving, and different standards and regulations are being formed with respect to it. Virtual testing environments based on scenario specifications have been demonstrated by few AV-focused companies.

In scenario-based testing, one or more scenarios are referenced from a test case specification. Scenarios contain the behavioral specifications for vehicles, pedestrians, etc. acting in the (virtual) world. Scenarios may include scenario validity criteria that may impact the test results. At the end, there is a wide spectrum of potential mutual relationships between the test and the scenarios. Below are several examples of such relationships:

• The test may affect scenario parametrization by posing additional constraints on it
• The scenario may result in events that are used by the test

And many others.

• There is always a trade-off between the amount of information flowing back and forth between the test and the scenario and the reusability of different scenarios across different tests. That is, independent of the choice of scenario language and test language, the ability to exchange information and reuse tests and scenarios should be allowed as much as possible. This leads to a need for a standardized interface between the test and scenario that clearly specifies the interaction rules.

Standardized Interface
Due to the heterogeneous architecture of both test instances and simulation environments in the industrial environment, both document types must be formulated generically with respect to the underlying hardware and software architecture. This is necessary to enable the consistent use of test cases and scenarios across different test instances and test environments throughout the development process and within homologation processes.

In scenario-based testing, one or more scenarios are referenced from a test case specification. This requires an interface within the test case description to implement higher-level concepts (e.g. parametrization) consistently in tests and scenarios. To enable the use of different scenario description languages (from the ASAM Open Standards), this interface must be designed in an abstract and modular manner. Standardizing the interface simplifies the synchronization of the developments and reduces integration risks.

Minimum requirements need to be defined regarding what should be handled between test case and scenario.

• Reference to scenarios (from within test cases)
• Parametrization and metrics of scenarios and test cases (e.g. changing the initial velocity of a vehicle) shall be consistent
• Specification of interfaces for runtime interaction within test cases
• with scenarios (access to language elements, e.g. events and success criteria)
• with environment simulation and the test environment (interface examples: see image [tracetronic]) e.g. evaluation and modification of numeric values during test execution (model parameters, ECU values, bus data, etc.)
• Interfaces should be bidirectional