SBT4ADS

Being capable of identifying significant safety shortcomings, search-based methods are a core tool for testing automated driving system (ADS) technologies. In this domain, Frenetic has proven to be a popular and very effective tool, searching and identifying diverse sets of roads that point out potentially faulty ADS behaviour. This paper presents Frenetic-lib, a Python library that captures Frenetic’s novel combination of road representation and genetic algorithm, and makes it generally available in a customisable way. Next to the capacity to integrate additional ADS simulators, Frenetic-lib further creates new research opportunities on search-based road testing, novel road representations and mutation operators.

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.

FreneticV is a search-based testing tool based on an evolutionary approach that generates roads where an automated driving agent possibly fails the lane-keeping task. It uses a curvature-based road representation and, compared to its predecessor Frenetic, considers the validity of the generated roads. In particular, it tries to avoid generating roads with overly sharp turns, detects self-intersecting roads, and can rotate and relocate roads to fit them in a given map.

This paper presents the first evaluation of k-nearest neighbours-Averaging (kNN-Avg) on a real-world case study. kNN-Avg is a novel technique that tackles the challenges of noisy multi-objective optimisation (MOO). Existing studies suggest the use of repetition to overcome noise. In contrast, kNN-Avg approximates these repetitions and exploits previous executions, thereby avoiding the cost of re-running. We use kNN-Avg for the scenario generation of a real-world autonomous driving system (ADS) and show that it is better than the noisy baseline. Furthermore, we compare it to the repetition-method and outline indicators as to which approach to choose in which situations.

Currently, most existing approaches for the design of Automated Driving Systems (ADS) scenarios focus on the description at one particular abstraction level — typically the most detailed one. This practice often removes information at higher levels, such that this data has to be re-synthesized if needed. As the abstraction granularity should be adapted to the task at hand, however, engineers currently have the choice between re-calculating the needed data or operating on the wrong level of abstraction. For instance, the search in a scenario database for a driving scenario with a map of a given road-shape should abstract over the lane markings, adjacent vegetation, or weather situation. Often though, the general road shape has to be synthesized (e.g. interpolated) from the precise GPS information of road boundaries. This paper outlines our vision for multi-level ADS scenario models that facilitate scenario search, generation, and design. Our concept is based on the common modelling philosophy to interact with scenarios at the most appropriate abstraction level. We identify different abstraction levels of ADS scenarios and suggest a template abstraction hierarchy. Our vision enables seamless traversal to such a most suitable granularity level for any given scenario, search and modelling task. We envision that this approach to ADS scenario modelling will have a lasting impact on the way we store, search, design, and generate ADS scenarios, allowing for a more strategic verification of autonomous vehicles in the long run.

Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimise the out of bound distance, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.