Ontology-based test generation for automated and autonomous driving functions

Ontology-based Test Generation for Automated and Autonomous Driving Functions

Journal Pre-proof

Ontology-based Test Generation for Automated and Autonomous Driving Functions Yihao Li, Jianbo Tao, Franz Wotawa PII: DOI: Reference:

S0950-5849(18)30227-1 https://doi.org/10.1016/j.infsof.2019.106200 INFSOF 106200

To appear in:

Information and Software Technology

Received date: Revised date: Accepted date:

1 February 2019 9 October 2019 11 October 2019

Please cite this article as: Yihao Li, Jianbo Tao, Franz Wotawa, Ontology-based Test Generation for Automated and Autonomous Driving Functions, Information and Software Technology (2019), doi: https://doi.org/10.1016/j.infsof.2019.106200

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier B.V.

Highlights • An automated approach for testing automated driving functions and autonomous vehicles. • Discussing two algorithms for automatically generating combinatorial test suites from ontologies. describing the environment of the system under test. • Empirical comparison of two algorithms for generating combinatorial test suites from environmental ontologies. • Experimental results based on a industrial case study indicating the practical usability of the proposed approach.

1

Ontology-based Test Generation for Automated and Autonomous Driving Functions Yihao Li1 , Jianbo Tao, Franz Wotawa∗ Institute for Software Technology, Graz University of Technology, Inffeldgasse 16b, 8010 Graz, Austria AVL List GmbH, Hans-List-Platz 1, 8020 Graz, Austria CD Laboratory for Quality Assurance Methodologies for Autonomous Cyber-Physical Systems, Institute for Software Technology, Graz University of Technology, Inffeldgasse 16b, 8010 Graz, Austria

Abstract Context: Ontologies are known as a formal and explicit conceptualization of entities, their interfaces, behaviors, and relationships. They have been applied in various application domains such as autonomous driving where ontologies are used for decision making, traffic description, auto-pilot etc. It has always been a challenge to test the corresponding safety-critical software systems in autonomous driving that have been playing an increasingly important role in our daily routines. Objective: Failures in these systems potentially not only cause great financial loss but also the loss of lives. Therefore, it is vital to obtain and cover as many as critical driving scenarios during auto drive testing to ensure that the system can always reach a fail-safe state under different circumstances. Method: We outline a general framework for testing, verification, and validation for automated and autonomous driving functions. The introduced method makes use of ontologies for describing the environment of autonomous vehicles and convert them to input models for combinatorial testing. The combinatorial test suite comprises abstract test cases that are mapped to concrete test cases that can be executed using simulation environments. Results: We discuss in detail on how to automatically convert ontologies to the corresponding combinatorial testing input models. Specifically, we present two conversion algorithms and compare their applicability using ontologies with different ∗ 1

Franz Wotawa is the corresponding author, email: [email protected] Yihao Li is the first author, email: [email protected]

Preprint submitted to Information and Software Technology

October 12, 2019

sizes. We also carried out a case study to further demonstrate the practical value of applying ontology-based test generation in industrial settings. Conclusion: The proposed approach for testing autonomous driving takes ontologies describing the environment of autonomous vehicles, and automatically converts it to test cases that are used in a simulation environment to verify automated driving functions. The conversion relies on combinatorial testing. The first experimental results relying on an example from the automotive industry indicates that the approach can be used in practice. Keywords: combinatorial testing, ontologies, test case generation, autonomous driving, input models 1. Introduction Due to the rapid development of artificial intelligence, its application to automobiles has drawn wide attention in the world. Having automated or autonomous cars driving on our streets is no longer a fantasy that only appears in sci-fi movies, but a real product, we can purchase from car manufacturers like Tesla Inc. Although, there might be restrictions of autonomy, such cars offer more and more autonomous driving functionality. With the increase of autonomy of cars the great challenge to provide reliable and sufficient testing on the related artificial intelligence software systems developed for autonomous cars becomes more and more important. Failures during operation will not only lead to significant financial loss but also, more severe, loss of lives. Therefore, the system itself must assure reaching a fail-safe state when faults occur under all circumstances. However, such a safety-critical requirement is difficult to meet. For example, it would most likely to cause a fatal crash when a moving car is programmed to stop immediately when a fault occurs on a highway having cars behind. Consequently, stopping the car alone is not considered as reaching a safe state. To tackle the challenge, research has been conducted on testing for autonomous cars. Koopman and Wagner [1] pointed out that a major problem in this area is how to ensure a “good enough” test suite on such auto-drive systems. According to Kalra and Paddock [2], an autonomous car should reach a certain amount of mileage to test the system. The number of miles without any accident for an autonomous car should be much higher than a car steered by man. Kalra and Paddock further argued that 275 million miles would be necessary for a sufficient testing based on the motor vehicle fatality rate in the US. Apparently, this is impossible to achieve as it will easily take 400 years for 100 autonomous cars driving simultaneously and nonstop with a speed of 25 miles per hour. As a result, more sophisticated approaches 3

are strongly needed. One way to alleviate the scalability issue aforementioned is to consider critical driving scenarios when testing autonomous cars [3]. But, how to automatically obtain such critical scenarios still remains unclear. To achieve this, approaches combining domain ontologies modeling the environment of autonomous cars with combinatorial testing techniques have been proposed, e.g., [4], [5], [6]. Ontologies are a formal conceptualization of entities, their interfaces and behaviors, and relationships that have been used for various purposes in the domain of autonomous driving such as decision making [7], traffic description [8], and auto pilot [9]. In the context of software testing, ontologies provide a common shared understanding of knowledge (i.e., entities and their relationships) that is easy to organize, maintain and update in a machine processable format [10] from which test cases can be extracted. Let us take the simplified Road Section ontology which comprises the concepts of RoadSection, Lane, V ehicle, Car and Cycle as an example. As shown in Fig. 1, an instance of a RoadSection is a sequence of lanes (i.e., instances of Lane followed by instances of V ehicle and instances of V ehicle followed by Car and Line). A test case can be an instance of RoadSection comprising its parts specified in the ontology. If we want to use such test cases to verify a program like a driving simulator, we might be looking for particular combinations of vehicles that form a lane as well as particular combinations of car and cycle that form a vehicle in order to test the simulator under different scenarios. To come up with such combinations, one possible solution is the use of combinatorial testing that has been proved useful and effective for software verification in different application domains [11]. Combinatorial testing (CT) takes a set of n variables together with their domains as input model and generates test cases that includes all combinations for any size k (k 6 n) of the variables, where k is called the strength of test suite generated from the input model. It is worth noting that before Wotawa and colleagues [4], [5], [6] only a few have focused on ontology-based software testing using CT in the domain of autonomous driving (e.g., taking an ontology as input and returns a CT input model or a combinatorial test suite of strength k as output). To pave the path to ontology-based test generation for automated and autonomous driving functions, in this paper, we will first outline a general framework for testing, verification and validation for automated and autonomous driving functions in Section 3. Then, we discuss in detail on how to automatically convert ontologies to the corresponding combinatorial testing input models, which can be used in the autonomous driving domain in Section 4. Specifically, we discuss two existing conversion algorithms (i.e., CTONT Union and CTONT Flatten), and the results of an experimental study we have conducted to evaluate their practicability in Section 5. 4

Figure 1: A simplified road section ontology

5

In addition, another case study is conducted in Section 6 to further demonstrate the industrial applicability of ontology-based test generation. Finally, we conclude the paper in Section 7. Compared to our previously published conference paper [5], the major extensions/contributions of this paper can be summarized as follows: • An integrated summary of previous research conducted in [5] and [6] • A detailed comparison between CTONT Union and CTONT Flatten • A real-life case study of applying ontology-based test generation in industrial settings (i.e., an autonomous emergency braking system function from AVL List GmbH) • A preliminary report on the experiment results obtained from the industrial case study

2. Related Work In combinatorial testing, input model construction is a very fundamental and important activity for combinatorial testing. The effectiveness and efficiency of CT directly depend on the quality of input model used. There are different kinds of models, for instance, finite state machine, state charts, activity diagrams, sequence diagram, Markov chain and grammars. Grochtmann et al. [12] introduced the classification tree method (CTM) as approach to set up input model and this method and tool have been tried out successfully on actual examples. A systematic method called as the category partition method (CPM) was introduced by Thomas J Ostrand et.al [13] in order to generate functional tests from the function specifications. The author decomposes the functional specifications into different testable units. The decomposition process involves identifying the parameters and environment conditions, partitioning the parameters and conditions into categories and dividing the categories into distinct choices. AETG [14] presented the technique using AETGspec notation to represent the input space model. M. Grindal and J. Offutt [15] presented an 8-step structured method for the input parameter models creation. Two approaches for input parameter modelling (IPM) were suggested, the interface-based IPM and functionality-based IPM. The author compared the strength and weakness of both two approaches, results indicate that functionality Based-IPM includes more semantic information and useful for subsequent test case generation. M. Spichkova et al. [16] presented a human-centered methodology into input modelling and testing 6

of cyber-physical systems using rule-based framework. Schuldt et al. [17] combined equivalence class partitioning and boundary value analysis with combinatorial testing to generate test cases for driver assistance systems. Satish and his team proposed a rule-based approach to derive CT parameters and values from use case specification and use case diagrams [18], UML activity diagrams [19] and sequence diagrams [20], respectively. Regarding the use of ontology in software testing and auto drive, Moser et al. [21] emphasized that the feasibility of the ontology-based test case generation approach and the cost-benefit for a constant number of parameters and for expanding the number of parameters are critical to an ontology-based approach. Studies [22, 23, 24, 25, 26] designed ontologies for software testing process and the corresponding activities and artifacts. In autonomous driving domain ontology has been used for different purposes. Xiong et al. [27] developed an ontology for abstract driving context representation and reasoning. Xiong et al. [28] used ontologies to create a formal standardized description for driving scenario in order to generate intentional interactions with simulated drivers. Geyer et al. [29] introduced an ontology built from a metaphor-based terminology for cooperative vehicle guidance. Bagschik et al. [30] discussed in detail how ontologies have been used for various applications in the field of automated driving. Li et al. [31] developed an ontology to generate user-centric GUI test cases. Nguyen et al. [32] proposed an ontology-based test generation framework for multi-agent systems. Huelsen et al. [33] used ontologies to create a generic situation description for advanced driver assistance systems using logic reasoning on a traffic situation. Armand et al. [34] presented an ontology about the vehicle, perceived entities and map information context in order to provide a conceptual description of all the street entities and their interaction. The author uses the ontology based approach to infer how perceived entities are expected to behave and to indicate the consequences of these behaviors on the subject vehicle. 3. A General Framework for Testing, Verification, and Validation for Automated and Autonomous Driving Functions To conduct a sufficient and reliable testing process for automated and autonomous driving functions, it is critical to follow a rational and standardized procedure from domain ontology construction, test scenario generation, and test execution. Fig. 2 shows a simplified overview on our testing framework based on the above three phases. Phase 1– Domain Ontology Construction. Feilmayr and W¨oß [35] stated that “an ontology is a formal, explicit specification of a shared conceptualization that is characterized by high semantic expressiveness required for increased complexity ”. Therefore, 7

Figure 2: A general framework for testing, verification, and validation for automated and autonomous driving functions taken from [6]

8

ontologies should be able to depict concepts as well as their relationships in a formal way. In our previous work [5], we introduced a simplified definition of ontologies based on concepts restricting relations to composition and inheritance. Specifically, an ontology is a tuple (C, A, D, ω, R, τ, ψ) where C is a finite set of concepts, A is a finite set of attributes (i.e., properties that characterize concepts), D is a finite set of domain elements. Concepts can have attributes. An attribute has its own type such as string or date. However, for the ease of building CT input models, we only rely on enumeration types meaning that elements of the enumeration that corresponds to D an attribute need to be specified. ω : C 7→ 2A×2 is a function mapping concepts to a set of tuples specifying the attribute and its domain elements, which is a subset of D. R is a finite set of tuples from C × C stating that two concepts are related. The function τ : R × R 7→ {c, i} assigns a type to each relation using c for composition and i for inheritance. We use these two relations to formalize knowledge if a concept is related with other concepts or one concept is more general than another. Furthermore, ψ : R 7→ IN0 × IN0 is a function mapping relationships solely of type c to its minimum and maximum arity. The arity is for specifying how many concepts a particular concept may comprise and ranges from 0 to any arbitrary natural number. Note that in the graphical representation the arity of the concept C1 of the composition relation is always 1. The minimum and maximum arity of concept C is given as follows: In case of * the minimum arity is 0 and the maximum arity is any predefined value. Otherwise, we have a 1..* indicating a minimum arity of 1. In addition, this definition of ontologies assures that there is at the maximum one relation between two concepts. Hence, it is not possible to state that one concept is a sub-concept of another concept and that there is a compositional relationship between them as well. For simplicity, we further introduce the function dom : C × A 7→ 2D that we will use later in this paper. The function dom returns the domain for an attribute a ∈ A of a given concept c ∈ C, and is defined as follows: dom(c, a) = d ↔ (a, d) ∈ ω(c). As outlined in detail in our previous work [5], we assume ontologies to be wellformed, i.e., having exactly one root concept, no cycles in the corresponding directed graph of the relations, and attributes in the leaf concepts. A root concept c is a concept from C where we do not have any (c0 , c, x) ∈ R for an arbitrary concept c0 inC and x ∈ {c, i}. A leaf concept c is a concept from C having no element (c, c0 , x) ∈ R for a c0 ∈ C and x ∈ {c, i}. Back to our road section ontology example shown in Fig 1 where we use the concepts of RoadSection, Lane, V ehicle, Car, and Cycle to construct an ontology for a certain road section (i.e., C = {RoadSection, Lane, V ehicle, Car, Cycle} with concepts Lane, Car, and Cycle having one attribute A = {id}, respectively). In 9

this ontology, a road section consists of 2 that are either curve or straight. A lane consists of 2 vehicles. Concepts RoadSection and Vehicle do not have any attributes. Therefore, we have: R = {(RoadSection, Lane), (Lane, V ehicle), (V ehicle, Car), (V ehicle, Cycle)} where τ (RoadSection, Lane) = c, τ (Lane, V ehicle) = c, τ (V ehicle, Car) = i, τ (V ehicle, Cycle) = i Let’s further assume that the domain of Car includes two elements which are GM and FORD. The domain of Cycle includes two elements which are YAMAHA and SUZUKI. And the domain of Lane includes two elements which are curve and straight. Thus, we have: D = {0 straight0 ,0 curve0 ,0 Y AM AHA0 ,0 SU ZU KI 0 ,0 F ORD0 ,0 GM 0 }. For ω, we have: ω(RoadSection) = {}, ω(Lane) = {(id, {0 straight0 ,0 curve0 })}, ω(Car) = {(id, {0 F ORD0 ,0 GM 0 })}, ω(Cycle) = {(id, {0 Y AM AHA0 ,0 SU ZU KI 0 })}, ω(V ehicle) = {}. For ψ, we have: ψ(RoadSection, Lane) = (1, 2), ψ(Lane, V ehicle) = (1, 2). An exemplary road section generated from this ontology could be a GM and a YAMAYA driving on a curve lane while a FORD and SUZUKI driving on a straight lane. ¨ According to The Austrian Traffic Association (OAMTC) [36], 43% percent of all accidents on motorways and highways were rear-end collisions. The major cause of rear-end collisions is tailgating. Therefore, a function that enables automatic emergency braking (AEB) can significantly improve the safety of self-driving cars in such cases. However, if AEB does not trigger the brake at due time, severe road accidents could happen. In our defense, the test scenarios generated via our road section ontology can be used to identify driving situations where the function does not operate as designed. Although there are many other factors (e.g., subject vehicle speed, visibility, distance, etc) that need to be included into the current exemplary ontology to make it more practical, the proposed ontology provides valuable insight into the potential difficulties or unexpected situations the corresponding function 10

Figure 3: Basic concepts, their attributes, and relationships that form an ontology of a particular application domain

may encounter. The domain ontology constructed in this phase is to collect and structure relevant parameters and their values for building the corresponding CT input model. The gathered information is generally presented in a hierarchical structure where different parameters can represent formal subsets of domain-related influence factors, which addresses specific environmental domains the system under test is interacting with. The ontology provides the basis for obtaining valid inputs for simulation. Phase 2 – Test Scenario Generation. In this phase, ontologies formed in Phase 1 will be automatically transferred into the corresponding CT input models which will later be used for test generation. Specifically, all concepts that are formally described in an ontology represent instances of a particular input model with attributes becoming the parameters and domain elements becoming the values for their parameters. Once an input model is built, a particular strategy for n-way software testing is applied for test case generation. The obtained test cases represent different test scenarios to be used in Phase 3. The details of the mapping algorithms will be thoroughly discussed in Section 4. Phase 3 – Test Execution. In this phase, the automated or autonomous driving functions will be tested under different scenarios generated in Phase 2 under a virtual 11

testing environment tailored for test execution. Based on the feedback from the execution, potential faults will be detected and a machine learning-based approach (e.g., [37] and [38]) can be applied in Phase 2 to iteratively identify and generate more critical test scenarios in order to find more faults until no more failed test executions (e.g., the autonomous vehicle reveals an unintended behavior) in Phase 3. With respect to test oracle, an option is to check if an execution leads to some formalized safety violations such as hitting another car, a pedestrian, or any other object. 4. From Ontologies to Combinatorial Testing Input Models We now focus on how to convert an ontology into a CT input model. For each concept in the ontology, a CT input model (M CT ) comprising its variables (V CT ) with their corresponding domains (DOM CT ) and constraints (CON S CT ) will be formed (i.e., M CT = (V CT , DOM CT , CON S CT )). A combinatorial testing algorithm is denoted as CT (M, k) where M is a combinatorial testing input model and k the combinatorial strength, returning a test suite. Precisely, two conversion algorithms, CTONT Union (from [5]) and CTONT Flatten (from [6]) will be discussed in the following. CTONT Union. The conversion inside CTONT Union consists of three cases: (1) converting concepts with attributes, (2) converting concepts with inheritance relations, and (3) converting concepts with composition relations. For (1), let us consider a concept c ∈ C with attributes a1 , . . . , an ∈ A, n ≥ 1. This is the simplest case where we only have to construct input variables for combinatorial testing for each attribute of the concept. The domain of each combinatorial testing input variable is equivalent to the domain of its corresponding attribute, and there are no further constraints to add. I.e.: V CT = {c a1 , . . . , c an }, ∀ni=1 : DOM CT (c ai ) = dom(c, ai ), and CON S CT = {}. For (2), let us assume a concept c ∈ C having n ≥ 1 sub-concepts c1 , . . . , cn ∈ C such that ∀ni=1 : (c, ci ) ∈ R. In this case, for c we have only one variable on side of combinatorial testing, and all its values come from the different values obtained when using the models from its sub-concepts and generating their combinatorial test suites. Let us assume MiCT to be the combinatorial testing model ci , Sn of sub-concept CT CT CT CT then M of c is given as follows: V = {c}, DOM (c) = i=1 CT (Mi , t), and CON S CT = {}, where CT is an algorithm computing a combinatorial test suite of strength t. For (3), let us assume the case of two concepts c1 and c2 from C and a relation (c1 , c2 ) ∈ R of type τ (c1 , c2 ) = c in our ontology. Depending on the minimum arity 12

of the relation we have to consider two cases. If the minimum arity is zero, then there need not to be any instance of c2 attached to c1 . In case the minimum arity is one, we have at least one individual of c2 . Algorithm 1 TC GEN (O,k) Require: A well-formed ontology O and a combinatorial strength k. Ensure: A combinatorial test suite for the root concept of the ontology O. 1: Let r be the root concept of ontology O. 2: Call CTONT Union(c,O,k) and store the result in (V CT , DOM CT , CON S CT ). 3: return CT ((V CT , DOM CT , CON S CT ), k) Algorithm TC GEN makes use of CTONT Union that recursively computes the combinatorial input models for the different concepts starting with the root note down to the leaves. In CTONT Union, we assume that the global variable m stores the number of instances that should be generated for composition relations. Let’s take root concept RoadSection as an example. It is straightforward to obtain the following information: M CT (Car) : V CT (Car) = {Car id} DOM CT (Car) = {0 F ORD0 ,0 GM 0 } M CT (Cycle) : V CT (Cycle) = {Cycle id} DOM CT (Cycle) = {0 Y AM AHA0 ,0 SU ZU KI 0 } Therefore, for M CT (V ehicle) we have: V CT (V ehicle) = {Car id, Cycle id} DOM CT (V ehicle) = {(0 GM 0 ,0 Y AM AHA0 ), (0 GM 0 ,0 SU ZU KI 0 ), (0 F ORD0 ,0 Y AM AHA0 ), (0 F ORD0 ,0 SU ZU KI 0 )} T heref ore, f orMˆCT(Lane)wehave : V CT (Lane) = {Lane id, V ehicle1, V ehicle2} DOM CT (Lane) = M CT (V ehicle, 2) = {0 curve0 ,0 straight0 , ((0 GM 0 ,0 Y AM AHA0 ), (0 GM 0 ,0 SU ZU KI 0 ), (0 F ORD0 ,0 Y AM AHA0 ), (0 F ORD0 ,0 SU ZU KI 0 )), 13

Algorithm 2 CTONT Union (c,O,k) Require: A concept c of a well-formed ontology O, and a combinatorial strength k. Ensure: A combinatorial input model for n. 1: Let V CT and CON S CT be empty sets. 2: for all attributes a ∈ ω(c) do 3: Add c a to V CT . 4: Let DOM CT (c a) be dom(c, a). 5: end for 6: if c is not a leaf concept then 7: Let tmp be the empty set. 8: for all relations (c, c0 ) ∈ R with type τ (c, c0 ) = i do 9: Add CT (CTONT Union(c0 , O, k), k) to tmp 10: end for 11: if tmp is not empty then 12: Add c to V CT . 13: Let DOM CT (c) be tmp. 14: end if 15: for all relations (c, c0 ) ∈ R with type τ (c, c0 ) = c do 16: Add variables c0 1 to c0 m to V CT . 17: for i = 1 to m do 18: Let DOM CT (c0 i) be CT (CTONT Union(c0 , O, k), k) ∪ {}. 19: end for 20: if ψ(c, c0 ) W = (1, x) then 21: Add i∈{1,...,m} c0 i 6=  to CON S CT . 22: end if 23: end for 24: end if 25: return (V CT , DOM CT , CON S CT )

14

Table 1: A 2-way test suite for Lane using CTONT Union

Lane id straight curve straight curve curve straight curve straight curve straight straight straight curve straight curve straight

Vehicle1 Vehicle2 (GM,YAMAHA) (GM,YAMAHA) (GM,YAMAHA) (GM,SUZUKI) (GM,YAMAHA) (FORD,YAMAHA) (GM,YAMAHA) (FORD,SUZUKI) (GM,SUZUKI) (GM,YAMAHA) (GM,SUZUKI) (GM,SUZUKI) (GM,SUZUKI) (FORD,YAMAHA) (GM,SUZUKI) (FORD,SUZUKI) (FORD,YAMAHA) (GM,YAMAHA) (FORD,YAMAHA) (GM,SUZUKI) (FORD,YAMAHA) (FORD,YAMAHA) (FORD,YAMAHA) (FORD,SUZUKI) (FORD,SUZUKI) (GM,YAMAHA) (FORD,SUZUKI) (GM,SUZUKI) (FORD,SUZUKI) (FORD,YAMAHA) (FORD,SUZUKI) (FORD,SUZUKI)

((0 GM 0 ,0 Y AM AHA0 ), (0 GM 0 ,0 SU ZU KI 0 ), (0 F ORD0 ,0 Y AM AHA0 ), (0 F ORD0 ,0 SU ZU KI 0 ))} Table 1 presents a 2-way test suite for Lane where 16 test cases are generated in total. Finally, for M CT (RoadSection) we have: V CT (RoadSection) = {RoadSection Lane1, RoadSection Lane2} DOM CT (RoadSection Lane1) = M CT (Lane, 2) DOM CT (RoadSection Lane2) = M CT (Lane, 2) Note that the size of 2-way test suite for Lane is 16 and RoadSection has two attributes RoadSection Lane1 and RoadSection Lane2, thus, the size of the 2-way test suite for RoadSection is 16 × 16 = 256 as for RoadSection the number of attributes equals to the combinatorial strength in this case. In general, CTONT Union terminates because we only consider finite and wellformed ontologies where there are no cycles and a finite number of concepts and relations with exactly one root concept and all leaf concepts have attributes. The 15

algorithm traverses the whole graph and has therefore a computational complexity in the size of the graph when ignoring the computational complexity of combinatorial test suites during the traversal. The drawback of CTONT Union is that the number of computed test cases can increase significantly and the depth of the ontology has to be small in order not to be still feasible. (This will be detailedly discussed later). CTONT Flatten. The motivation behind CTONT Flatten is straightforward, to control the size of target test suite. Different from CTONT Union, which uses the entire CT-based test suite generated for a lower level (or composee) concept (e.g., concept Lane) as the domain for its direct related higher level (or composer) concept (e.g., concept RoadSection), when applying CTONT Flatten as shown in Algorithm CTONT Flatten the domains, variables, and constraints of a lower level (or composee) concept cumulatively form that of its direct related higher level (or composer) concept. Let us refer to Fig. 3 to distinguish different conversion cases. For inheritance relations, the variables, domains, and constraints of subconcepts C2 and C3 will be integrated into the variables, domains, and constraints of C1. For composition relations, the variables, domains, and constraints of C1 consists of the variables, domains, and constraints of n (depending on the value of *) instances of C2. Algorithm 3 TC GEN (r,O,k) Require: A well-formed ontology O and a combinatorial strength k. Ensure: A combinatorial test suite for the root concept of the ontology O. 1: Let r be the root concept of ontology O. 2: Call CTONT Flatten(r, O, k) 3: return CT ((V CT , DOM CT , CON S CT ), k) Again, we take root concept Road Section as an example. Therefore, for M CT (Lane) we have: DOM CT (Car) = {0 F ORD0 ,0 GM 0 } DOM CT (Cycle) = {0 Y AM AHA0 ,0 SU ZU KI 0 } V CT (V ehicle) = {Car id, Cycle id} Therefore: V CT (Lane) = = {Lane id, V ehicle1 Car id, V ehicle1 Cycle id, V ehicle2 Car id, V ehicle2 Cycle id} 16

Algorithm 4 CTONT Flatten (c,O,k) Require: A concept c of a well-formed ontology O, and a combinatorial strength k. Ensure: A combinatorial input model for n. 1: Let V CT and CON S CT be empty sets. 2: for all attributes a ∈ ω(c) do 3: Add c a to V CT . 4: Let DOM CT (c a) be dom(c, a). 5: end for 6: if c is not a leaf concept then 7: for all relations (c, c0 ) ∈ R with type τ (c, c0 ) = i do 8: Add c to V CT 9: Add  to dom(c) 10: Add dom(c) to DOM CT (c) 11: Call CTONT Flatten(c, O, k) 12: end for 13: for all relations (c, c0 ) ∈ R with type τ (c, c0 ) = c do 14: Add variables c0 1 to c0 m to V CT . 15: for i = 1 to m do 16: Add  to dom(c0 i) 17: Add dom(c0 i) to DOM CT (c) 18: Add CON S CT (c’ i) to CON S CT (c) 19: Call CTONT Flatten(c0 i, O, k) 20: end for 21: end for 22: end if 23: return (V CT , DOM CT , CON S CT )

17

Table 2: A 2-way test suite for Road Section using CTONT Flatten - Part 1 Lane1 id curve curve straight straight straight straight curve straight straight curve

Lane1 Vehicle1 Car id GM GM GM FORD GM FORD GM FORD GM FORD

Lane1 Vehicle1 Cycle id Lane1 Vehicle2 Car id SUZUKI FORD FORD YAMAHA YAMAHA FORD SUZUKI GM SUZUKI GM YAMAHA FORD SUZUKI FORD YAMAHA GM YAMAHA FORD SUZUKI FORD

Lane1 Vehicle2 Cycle id SUZUKI GM YAMAHA SUZUKI YAMAHA SUZUKI SUZUKI YAMAHA YAMAHA SUZUKI

Table 3: A 2-way test suite for Road Section using CTONT Flatten - Part 2 Lane2 id straight curve straight curve straight curve curve straight straight straight

Lane2 Vehicle1 Car id FORD YAMAHA GM FORD GM FORD GM FORD FORD GM

Lane2 Vehicle1 Cycle id Lane2 Vehicle2 Car id SUZUKI FORD GM GM SUZUKI GM YAMAHA FORD YAMAHA FORD SUZUKI GM YAMAHA GM SUZUKI FORD YAMAHA GM SUZUKI GM

Lane2 Vehicle2 Cycle id SUZUKI YAMAHA SUZUKI YAMAHA SUZUKI YAMAHA SUZUKI YAMAHA YAMAHA SUZUKI

Accordingly, for RoadSection we have: V CT (RoadSection) = {Lane1 id, Lane1 V ehicle1 Car id, Lane1 V ehicle1 Cycle id, Lane1 V ehicle2 Car id, Lane1 V ehicle2 Cycle id, Lane2 id, Lane2 V ehicle1 Car id, Lane2 V ehicle1 Cycle id, Lane2 V ehicle2 Car id, Lane2 V ehicle2 Cycle id} Altogether, we have 10 test cases generated for RoadSection as shown in Table 2 and Table 3. 5. Experiments In this section, we will further investigate the test generation applicability of CTONT Union and CTONT Flatten with respect to different ontology size/complexity and combinatorial strength. Specifically, two ontologies, simple (i.e., Fig. 4) and complex (i.e., Fig. 5), are used for the comparison. As observed from the two figures, the size and complexity of the simple ontology is similar to that of the road section ontology used in previous sections which has 5 concepts with 4 composition 18

Figure 4: Simple ontology

relations. The complex ontology is about 3 times the size of the simple ontology with 12 relations including 9 composition relations and 3 inheritance relations. In addition to the ontology size, the complex ontology also has deeper and flatter hierarchical structure than the simple ontology making the conversion more challenging. For viewing convenience, Table 4 gives a list of abbreviations and their descriptions which are used to describe the results of our experiments. The working environment used to carry all experiments is a Dell XPS Laptop with 2.6GHz, Intel Core i7, 16 GB memory using Windows 10. Table 5 shows the case study results on the simple ontology. We compare CTONT Union and CTONT Flatten in terms of NP, MNVPP, and NTC with combinatorial strength k varing from 1 to 7 for the root concept (i.e., concept Root). For example, when using CTONT Union to converse the simple ontology with k = 1 the resultant combinatorial testing input model has 2 parameters (i.e., from left to right, the top element in row 2, column 3) and the maximum number of values in

19

Figure 5: Complex ontology

Table 4: Abbreviations and descriptions Abbreviation Description NP Number of parameters MNVPP Maximum number of values per parameter NTC Number of test cases generated Null Test generation fails as k > N P Overflow Ontology conversion or test generation fails due to stack overflow Over 30 mins Ontology conversion is forced to terminate as the execution time exceeds a predefined upper time limit (i.e., 30 minutes)

20

a parameter is 3 (i.e., the middle element in row 2, column 3). The number of test cases generated from this input model is 3 (i.e., the bottom element in row 2, column 3) using ACTS 3.1 [39] and IPOG [40]. When using CTONT Flatten for ontology conversion under the same setting, the resultant input model has 8 parameters (i.e., the top element in row 2, column 4) and the maximum number of values in a parameter is 3 (i.e., the middle element in row 2, column 4). The size of generated test cases is as same as using CTONT Union which is 3 (i.e., the bottom element in row 2, column 4). For another example, when using CTONT Union the input model fails to generate test cases for when k = 4 (i.e., the bottom element in row 5, column 3) because the number of parameters of the model is 2 (i.e., the top element in row 5, column 3) which is smaller than the used combinatorial strength 4 (i.e., row 5, column 1). In addition, when the strength increases from 4 to 5 for CTONT Union, the ontology conversion is forced to terminate as the execution time exceeds the predefined upper time limit (i.e., 30 minutes). As a result, the value of MNVPP (i.e., the middle element in row 6, column 3) cannot be obtained. Table 6 shows the case study results on the root concept (i.e., concept Root) of the complex ontology. The most distinct difference compared to Table 5 is there are a lot of reported ontology conversion or test generation failures due to stack overflow when using CTONT Union even with a combinatorial strength of 2. However, this is not the case for CTONT Flatten. On the contrary, CTONT Flatten is able to successfully convert both ontologies into the corresponding CT input models and then successfully generate test cases with different strength (except when k = 6 and k = 7 the test generation is forced to terminate because of the execution overtime). In addition, when using CTONT Union the MNVPP increases dramatically from 4 to 6561 with k increasing slightly from 1 to 2. Furthermore, the MNVPP of the input model converted by CTONT Flatten is smaller than that of the input model converted by CTONT Union for both ontologies under different combinatorial strength except when k = 1. Also, for both ontologies when using CTONT Union, NP of the root concept is always equal to the maximum arity of concept B. In general, when using CTONT Union we have the following observations: (1) the NP of a concept in the ontology directly depends on the maximum arity of its directly connected concepts that the target concept consists of (e.g., in the simple ontology, the NP of concept Root depends on the maximum arity of concept B); (2) the MNVPP of a concept could increase significantly with the increase of combinatorial strength k or/and with the increase of the size of ontology (e.g., scalability issue); (3) the MNVPP directly depends on the maximum test suite size generated from its directly connected concepts that the target concept consists of (e.g., in the complex ontology, the NP of concept Root depends on the size of M CT (B, k)); and (4) the 21

Table 5: Case study on root concept of the simple ontology. See Table 4 for more details about the used abbreviations.

Simple Ontology CTONT Union CTONT Flatten NP 2 8 k = 1 MNVPP 3 3 NTC 3 3 NP 2 8 k = 2 MNVPP 9 3 NP 81 15 NP 2 8 k = 3 MNVPP 27 3 NTC Null 48 NP 2 8 k = 4 MNVPP 54 3 NTC Null 141 NP 2 8 k = 5 MNVPP Over 30 mins 3 NTC Null 366 NP 2 8 k = 6 MNVPP Over 30 mins 3 NTC Null 744 NP 2 8 k = 7 MNVPP Over 30 mins 3 NTC Null 1458

22

Table 6: Case study on root concept of the complex ontology. See Table 4 for more details about the used abbreviations.

Complex Ontology CTONT Union CTONT Flatten NP 4 58 k = 1 MNVPP 4 4 NTC 4 4 NP 4 58 k = 2 MNVPP 6561 4 NP Overflow 29 NP 4 58 k = 3 MNVPP Overflow 4 NTC Overflow 143 NP 4 58 k = 4 MNVPP Overflow 4 NTC Overflow 654 NP 4 58 k = 5 MNVPP Overflow 4 NTC Null 2726 NP 4 58 k = 6 MNVPP Overflow 4 NTC Null Over 30 mins NP 4 58 k = 7 MNVPP Overflow 4 NTC Null Over 30 mins

23

NTC could increase significantly with the increase of combinatorial strength k or/and with the increase of the size of ontology (e.g., scalability issue). With respect to CTONT Flatten we have the following observations: (1) the NP of a concept in the ontology depends on the maximum arity of its directly connected concepts that the target concept consists of as well as the maximum arity of concepts that indirectly connects to the target concept (e.g., in the complex ontology, the NP of concept F jointly depends on maximum arity of concepts J, F, and L); (2) The NP and MNVPP will not change as k increases; and (3) the NTC greatly depends on the k used. Compared to CTONT Flatten, CTONT Union takes all combinations of different composee’s while CTONT Flatten determines k-way coverage only after composee’s are embedded into the composer. On one hand, CTONT Flatten corresponds well to one would expect from the model transformation: it flattens the ontology to collect all relevant variables and their choice ranges. On the other hand, CTONT Union recursively builds the test cases, with the effect that the sizes multiply and global CT optimization is hampered. As a result, CTONT Flatten better scales CTONT Union in converting ontologies to CT input models and then generating test cases from them.

24

Figure 6: An industry-level ontology

Table 7: Case study on an industry-level ontology

Complex Ontology CTONT Flatten NP 523 k = 1 MNVPP 4 NTC 4 NP 523 k = 2 MNVPP 4 NTC 34 NP 523 k = 3 MNVPP 4 NTC 190 NP 523 k = 4 MNVPP 4 NTC Over 30 mins

Knowing that CTONT Flatten has better scalability than CTONT Union, we then further apply CTONT Flatten on a third ontology whose size and complexity are equal to a reallife ontology designed to describe test scenarios for a driving autonomous vehicle from an automotive consulting firm in Austria. As shown in Fig. 6, this ontology has 53 concepts and 52 relations (41 composition relations and 11 inheritance relations), which is 4 times the size of Complex ontology (see Fig. 5) with much deeper and flatter hierarchical structure. Table 7 shows the corresponding test generation results with k varying from 1 to 4. The number of parameters has reached 523 due to the significantly increased number of concepts and the number of relations compared to Complex ontology. However, the size of generated test suite based on each k is quite reasonable. Once again, CTONT Flatten is applicable to converting ontologies with flexible sizes which is essential for ontology conversion in industrial application domain. 6. An Industrial Application of Ontology-based Test Generation for Autonomous Driving Function In this section an industrial level case study of applying ontology-based test generation for autonomous driving function testing is presented.

26

Figure 7: The process overview: from ontology to simulation

The Process Overview. Specifically, we will construct an ontology for a prototype Autonomous Emergency Braking System Function (AEB) from AVL List GmbH (AVL) and convert the ontology to its corresponding CT input model with well-defined constraints, parameters and their values using CTONT Flatten. The generated test cases from CT will later be used for the virtual simulation. Fig. 7 shows the overview of the entire process. The Construction of AEB function ontology. Autonomous Emergency Braking (AEB) systems detect potential collision situations using sensors around a vehicle. When a critical scenario is encountered, AEB will take control and react to minimize the damage caused by the incoming collision if the driver does not respond in a timely manner. As a well-known organization for car safety performance assessment, the European New Car Assessment Programme (Euro NCAP) [41] has become synonymous with crash testing and safety ratings. Specifically, Euro NCAP has introduced the test protocol for AEB Car2Car (C2C) [42] as well as for AEB Vulnerable road users VRU[43], respectively. The AEB C2C specifies the most frequent accidents scenarios of Car-to-Car on the road as shown in Fig. 8. In the figure, Car-to-Car Rear Stationary (CCRs) stands refers to a scenario that in which a vehicle travels forwards toward another stationary vehicle and the frontal structure of the vehicle strikes the rear structure of the other. CCRm stands for Car-to-Car Rear Moving, where the other vehicle is moving. CCRb means Car-to-Car Rear Braking, which 27

Figure 8: AEB scenarios in EuroNcap protocol

the other vehicle is braking. With respect to the scenarios with pedestrian, CPFA means Car-to-Pedestrian Farside Adult; CPNA means Car-to-Pedestrian Nearside Adult; CPNC means Car-to-Pedestrian Nearside Child; and CPLA means Car-toPedestrian Longitudinal Adult. It is worth noting that all the variables/parameters in the above scenarios are clearly described in the protocol. For example, Fig. 10 shows the test case values of the CCRs scenario. According to the protocol the CCRs scenario is a combination of speed and overlap with 5km/h incremental steps in speed and 25% in overlap within the ranges as shown in this figure. These values are later considered in the AEB function ontology. Based on this information we are able to create an ontology for AEB function where all the scenarios aforementioned are covered. As shown in Fig. 11, the AEB function ontology has 4 concepts and 3 composition relations. For example, the root concept “ScenarioDynamicPart” has two attributes, the number of vehicle players (i.e., NumberOfVehiclePlayer)and the number of pedestrians (i.e., NumberOfPed). In this case, the number of vehicle players can vary from 1 to 3 (including the parking vehicles in a scenario like CPNC). The values for the number of pedestrians include 1, 2, and null where null refers to zero pedestrian as there are no pedestrian in the 28

Figure 9: Illustration of constructed AEB ontology based on EuroNCAP scenarios

Figure 10: Example of test case values in EuroNCAP protocol

29

Figure 11: Constructed AEB ontology based on EuroNCAP scenarios using UML

car-to-car scenarios. The remaining three concepts are Ego Vehicle, Vehicle Players, and Pedestrian Objects, respectively. The definitions regarding the parameters of AEB function ontology are illustrated in Fig. 9. Again, the parameter value range or the representative values for each parameter in Fig. 12 is extracted based on the corresponding values defined in the protocol. Ontology Conversion, Test Generation, and Simulation. Once the AEB ontology is built, CTONT Flatten is applied to converting the ontology to the corresponding CT input model. Later, the constraints are manually added into the generated input model. Constraints are used to avoid impossible combinations. For example one of the constraints is depicted as if number of vehicle is 1, the parameters of other two vehicles should be “null”. The writing is shown below.

30

Figure 12: Parameter values of constructed AEB ontology

31

(N umberOf V ehicleP layer = 1) => (V ehicles P layers2 Start speed = “null00 && V ehicles P layers2 V ehicleT ype = “null00 && V ehicles P layers2 Of f set s = “null00 && V ehicles P layers2 Of f set t = “null00 && V ehicles P layers2 Rate = “null00 && V ehicles P layers2 T arget Speed = 0&& V ehicles P layers2 DriverT ype = “null00 && V ehicles P layers3 Start speed = “null00 && V ehicles P layers3 V ehicleT ype = “null00 && V ehicles P layers3 Of f set s = “null00 && V ehicles P layers3 Of f set t = “null00 && V ehicles P layers3 Rate = “null00 && V ehicles P layers3 T arget Speed = 0&& V ehicles P layers3 DriverT ype = “null00 ) In total, 39 constraints are written into the input model. With the implementation of ACTS 3.1 with IPOG, the 2-way test suite results in 319 test cases generated. Later, the generated test suite is reformatted in the form of SpenScenario xml that fits the specification of with OpenScenario [44], an open file format for the description of dynamic contents in driving simulation applications. Fig. 13 shows virtual displayed scenario of two generated scenarios in VTD [45]. The picture on the left displays the 3D view while the picture on the right displays the 2D view of the scenario in the simulation platform. As observed from Fig. 13, test case #13 is one test case with 2 parking vehicles and one pedestrian and test case #166 refers to the scenario with two pedestrians with one from the left side and the other from the right side. Preliminary Findings. From the simulation results, we identified 9 crash cases and 30 critical cases out of the 319 test cases. We use time-to-collision (TTC) as our oracle. TTC is a well-known time-based safety indicator for detecting potential rearend collision. It is the time separation between two vehicles if both continue at their actual speed along their respective trajectories [46]. Therefore, the higher a TTC value, the safer the situation. In our study, the crash cases mean that a physical collision occurred in the simulation where TTC is less than or equal to 0 second. For the critical cases, a physical collision is about to happen but does not actually happen. According to [47], encounters with a TTC less than 1.5s are considered as critical situations in reality. However, since we are looking at a fully automated 32

emergency braking system, the driver’s reaction time is neglected. As a result, a case with a TTC between 0 and 1 seconds is considered as a critical case. The remaining cases where TTC is larger than 1 second belong to safe cases. In Figure 14, we show an example of a crash case from the view of the ego vehicle where pedestrians are involved crossing the street. In this scenario number of vehicle is three and number of pedestrians is two. There are three vehicles one driving in front and two parking, and two pedestrians. As shown in the figure one pedestrian is crossing from left and another from the right side. The front vehicle suddenly hit the brake when two pedestrians were crossing the street at the same time. The ego vehicle reacted while the front vehicle braked and the first pedestrian crossed the lane successfully from left, but the collision happened with the second pedestrian crossed the street, which is obviously a faulty behavior. The detected crash information was reported to the AEB function developers for further debugging activities. The 9 crashes do not indicate the actual number of bugs in the function. In our case, we found out that the failures origin in the system architecture as well as in the implementation. Regarding 30 critical cases, although the crash does not happen, but their evaluations are on borderline and close to the crash limit, therefore these cases are also important for the function verification. Any deviations of influencing factors which caused by the differences between simulation and reality could lead a critical case to a crash case. In the following we performed data analysis using JRip to study the interactions between entities and parameters, important indicators, rules, combinations or scenarios which result in a crash, critical, or safe case. JRip implements Repeated Incremental Pruning to Produce Error Reduction, aka RIPPER, a propositional rule learner proposed by William W. Cohen as an optimized version of IREP [48]. JRip is based in association rules with reduced error pruning (REP). In REP for rules algorithms, the training data is split into a growing set and a pruning set. First, an initial heuristic-based rule set is formed that over the growing set. Then, the overlarge rule set is repeatedly simplified by applying one of a set of pruning operators to delete any single condition or any single rule. During each iteration the pruning operator which yields the greatest reduction of error on the pruning set will be chosen. The simplification loop ends if the chosen pruning operator increases error on the pruning set. The derived rules that lead to crash, critical or safe cases are shown as below: (P edestrains Objects1 Start speed ≥ 1.6)and (V ehicles P layers1 Of f set t ≥ 65)and (EgoV ehicle1 Start speed ≥ 23.611111)and (V ehicles P layers3 Start speed ≤ 8)and (EgoV ehicle1 Of f set s ≤ 1) 33

Figure 13: 3D and 2D views of generated scenarios in simulation platform

34

Figure 14: Example of collision scenario in view of ego-vehicle

⇒ Result = crash(4.0/0.0)) (V ehicles P layers2 Start speed ≥ 41)and(P edestrains Objects1 Rate ≥ 6)and (EgoV ehicle1 Of f set s ≥ 0) ⇒ Result = crash(2.0/0.0) (EgoV ehicle1 Start speed ≥ 36.111111)and(EgoV ehicle1 Rate ≤ 8) ⇒ Result = critical(7.0/0.0) (EgoV ehicle1 Start speed ≤ 0) ⇒ Result = critical(5.0/0.0) (EgoV ehicle1 T arget Speed ≤ 14)and(EgoV ehicle1 Rate ≤ 8) ⇒ Result = critical(3.0/0.0) (EgoV ehicle1 Start speed ≤ 5.555556)and(EgoV ehicle1 Start speed ≥ 5.555556)and (V ehicles P layers1 Start speed ≥ 11) ⇒ Result = critical(4.0/0.0) (EgoV ehicle1 Start speed ≥ 23.611111)and (P edestrains Objects1 Start speed ≤ 0.56)and (P edestrains Objects2 Of f set t ≥ 18)and(EgoV ehicle1 Of f set s ≥ 1) ⇒ Result = critical(3.0/0.0) (EgoV ehicle1 Rate ≥ 10)and(P edestrains Objects1 Rate ≤ 0)and 35

(V ehicles P layers1 Rate ≥ 8) ⇒ Result = saf e(280.0/4.0) For example, the binary split combination (P edestrains Objcts1 Start speed ≥ 1.6)and(V ehicles P layers1 Of f set t ≥ 65)and(EgoV ehicle1 Start Speed ≥ 23.61111(m/s))and(V ehicle P layer3 Start Speed ≤ 8(m/s))and (EgoV ehicles1 Of f set s ≤ 1)) has appeared 4 times during virtual test simulation and results in a crash case every single time. Therefore, JRip makes this combination as one of the rules. Another example, the binary split combination (EgoV ehicles1 Rate ≥ 10)and(P edestrains Object1 Rate ≤ 0)and (V ehicle P layers1 Rate ≥ 8) has appeared 280 times during virtual test simulation and results in a safe case very single time. The rules derived from this analysis can provide useful insight into the identification of significant failure-causing interactions as well as the direction for further test case optimization. 7. Conclusion and Future Work Testing automated and autonomous vehicle has always been challenging due to the fact that it is safety-critical and is lack of standardized procedures to identify its boundaries and prerequisites. Therefore, answers to the questions about when to stop testing and how to tell the system is sufficiently tested can hardly be given. Instead driving millions of miles to test a self-driving vehicle, ontologies which describes the environment of an autonomous vehicle or an automated driving function combined with combinatorial testing techniques should be adopted to generate different test scenarios that would reveal a fault in the system. For this purpose, in this paper we first outline a general framework for testing, verification, and verification of automated and autonomous vehicles. The framework includes three phases: ontology construction, test scenario generation, and test execution. Later, two algorithms that convert an ontology to a combinatorial testing input model, namely CTONT Union and CTONT Flatten, are introduced followed by case studies on three ontologies greatly different in size and complexity to further evaluate their practical applicability. The results indicate that CTONT Flatten better scales than CTONT Union. To the best of our knowledge, CTONT Union and CTONT Flatten are the first algorithms allowing to obtain combinatorial test suites from ontologies featured by UML class diagrams. In addition, an industrial level case study is further conducted to demonstrate the practical value of the proposed ontology-based technique on testing autonomous driving functions. In the future, we plan to do the following: (1) investigate the impact of k on failure detection as k increases; (2) 36

extend current AEB ontology by integrating static parts such as roads, weather, and so on; and (3) develop a mechanism to automate constraints extraction. Acknowledgment The research was supported by ECSEL JU under the project H2020 737469 AutoDrive - Advancing fail-aware, fail-safe, and fail-operational electronic components, systems, and architectures for fully automated driving to make future mobility safer, affordable, and end-user acceptable. AutoDrive is funded by the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) under the program “ICT of the Future” between May 2017 and April 2020. More information can be . The financial support by retrieved from https://iktderzukunft.at/en/ the Austrian Federal Ministry for Digital and Economic Affairs and the National Foundation for Research, Technology and Development is gratefully acknowledged. References [1] P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation,” SAE Int. J. Trans. Safety, vol. 4, pp. 15–24, 04 2016. [Online]. Available: https://doi.org/10.4271/2016-01-0128 [2] N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” Transportation Research Part A: Policy and Practice, vol. 94, pp. 182 – 193, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/ S0965856416302129 [3] T. Menzel, G. Bagschik, and M. Maurer, “Scenarios for development, test and validation of automated vehicles,” in arXiv:1801.08598, 2018, accepted at the 2018 IEEE Intelligent Vehicles Symposium. [Online]. Available: https://arxiv.org/abs/1801.08598 [4] F. Wotawa, “Testing autonomous and highly configurable systems: Challenges and feasible solutions,” in Automated Driving, D. Watzenig and M. Horn, Eds. Springer International Publishing Switzerland, 2017. [5] F. Wotawa and Y. Li, “From ontologies to input models for combinatorial testing,” in Testing Software and Systems, M. G. Medina-Bulo, Inmaculadaand Merayo and R. Hierons, Eds. Springer International Publishing, 2018, pp. 155–170.

37

[6] F. Kl¨ uck, Y. Li, M. Nica, J. Tao, and F. Wotawa, “Using ontologies for test suites generation for automated and autonomous driving functions,” in Proceedings of the 29th IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW2018), Memphis, TN, USA, October 2018. [7] R. Regele, “Using ontology-based traffic models for more efficient decision making of autonomous vehicles,” in Fourth International Conference on Autonomic and Autonomous Systems (ICAS’08), March 2008, pp. 94–99. [8] M. H¨ ulsen, J. M. Z¨ollner, and C. Weiss, “Traffic intersection situation description ontology for advanced driver assistance,” in 2011 IEEE Intelligent Vehicles Symposium (IV), June 2011, pp. 993–999. [9] C. Schlenoff, S. Balakirsky, M. Uschold, R. Provine, and S. Smith, “Using ontologies to aid navigation planning in autonomous vehicles,” The Knowledge Engineering Review, vol. 18, no. 3, pp. 243–255, 2003. [10] H. Naseer and A. Rauf, “Validation of ontology based test case generation for graphical user interface,” in 15th International Multitopic Conference, 2012. [11] D. Kuhn, R. Kacker, Y. Lei, and J. Hunter, “Combinatorial software testing,” Computer, pp. 94–96, August 2009. [12] M. Grochtmann and D. benz Ag, “Test case design using classification trees,” Proceedings of STAR, vol. 94, pp. 93–117, 1994. [13] T. J. Ostrand and M. J. Balcer, “The category-partition method for specifying and generating fuctional tests,” Communications of the ACM, vol. 31, no. 6, pp. 676–686, 1988. [14] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The aetg system: An approach to testing based on combinatorial design,” IEEE Transactions on Software Engineering, vol. 23, no. 7, pp. 437–444, 1997. [15] M. Grindal and J. Offutt, “Input parameter modeling for combination strategies,” in Proceedings of the 25th conference on IASTED International MultiConference: Software Engineering. ACTA Press, 2007, pp. 255–260. [16] M. Spichkova, A. Zamansky, and E. Farchi, “Towards a human-centred approach in modelling and testing of cyber-physical systems,” in 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 2015, pp. 847–851. 38

[17] F. Schuldt, A. Reschka, and M. Maurer, “A method for an efficient, systematic test case generation for advanced driver assistance systems in virtual environments,” in Automotive Systems Engineering II, H. Winner, G. Prokop, and M. Maurer, Eds. Springer International Publishing AG, 2018. [18] P. Satish, M. Basavaraja, M. S. Narayan, and K. Rangarajan, “Building combinatorial test input model from use case artefacts,” in 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, 2017, pp. 220–228. [19] P. Satish, K. Sheeba, and K. Rangarajan, “Deriving combinatorial test design model from uml activity diagram,” in 6th IEEE International Conference on Software Testing, Verification and Validation Workshops, 2013, pp. 331–337. [20] P. Satish, A. Paul, and K. Rangarajan, “Extracting the combinatorial test parameters and values from uml sequence diagrams,” in 7th IEEE International Conference on Software Testing, Verification, and Validation Workshops, 2014, pp. 88–97. [21] T. Moser, G. D¨ urr, and S. Biffl, “Ontology-based test case generation for simulating complex production automation systems,” in 22nd International Conference on Software Engineering & Knowledge Engineering, 2010, pp. 478–483. [22] S. Vasanthapriyan, J. Tian, D. Zhao, S. Xiong, and J. Xiang, “An ontologybased knowledge sharing portal for software testing,” in 17th IEEE International Conference on Software Quality, Reliability and Security (Companion Volume), 2017, pp. 472–479. [23] E. F. Souza, R. A. Falbo, and N. L. Vijaykumar, “Using ontology patterns for building a reference software testing ontology,” in 17th IEEE International Enterprise Distributed Object Computing Conference Workshops, 2013, pp. 21– 30. [24] E. F. de Souza, “Knowledge management applied to software testing: An ontology based framework,” Ph.D. dissertation, National Institute for Space Research, 2014. [25] V. H. Nasser, “Ontology-based unit test generation,” Master’s thesis, Amirkabir University of Technology, 2007.

39

[26] E. Blomqvist, A. S. Sepour, and V. Presutti, “Ontology testing-methodology and tool,” in 18th International Conference on Knowledge Engineering and Knowledge Management, 2012, pp. 216–226. [27] Z. Xiong, V. V. Dixit, and S. T. Waller, “The development of an ontology for driving context modelling and reasoning,” in 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Nov 2016, pp. 13–18. [28] Z. Xiong, H. Jamson, A. G. Cohn, and O. Carsten, Ontology for Scenario Orchestration (OSO): A Standardised Scenario Description in Driving Simulation. ASCE Library, 2013. [Online]. Available: https: //ascelibrary.org/doi/abs/10.1061/9780784413036.210 [29] S. Geyer, M. Baltzer, B. Franz, S. Hakuli, M. Kauer, M. Kienle, S. Meier, T. Weissgerber, K. Bengler, R. Bruder, F. Flemisch, and H. Winner, “Concept and development of a unified ontology for generating test and use-case catalogues for assisted and automated vehicle guidance,” IET Intelligent Transport Systems, vol. 8, no. 3, pp. 183–189, May 2014. [30] G. Bagschik, T. Menzel, and M. Maurer, “Ontology based scene creation for the development of automated vehicles,” CoRR, vol. abs/1704.01006, 2017. [Online]. Available: http://arxiv.org/abs/1704.01006 [31] H. Li, H. Guo, F. Chen, H. Yang, and Y. Yang, “Using ontology to generate test cases for gui testing,” International Journal of Computer Applications in Technology, vol. 42, no. 2/3, pp. 213–224, February 2011. [32] C. D. Nguyen, A. Perini, and P. Tonella, “Ontology-based test generation for multiagent systems,” in 7th International Conference on Autonomous Agents and Multiagent Systems, May 2008, pp. 1315–1318. [33] M. H¨ ulsen, J. M. Z¨ollner, and C. Weiss, “Traffic intersection situation description ontology for advanced driver assistance,” in 2011 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2011, pp. 993–999. [34] A. Armand, D. Filliat, and J. Iba˜ nez-Guzman, “Ontology-based context awareness for driving assistance systems,” in 2014 IEEE intelligent vehicles symposium proceedings. IEEE, 2014, pp. 227–233. [35] C. Feilmayr and W. W¨oß, “An analysis of ontologies and their success factors for application to business,” Data & Knowledge Engineering, pp. 1–23, 2016. 40

¨ [36] OAMTC, “Every second highway accident is a rear-end collision.” [Online]. Available: https://www.oeamtc.at/thema/panne-unfall/ jeder-zweite-autobahnunfall-ist-ein-auffahrunfall-19777709 [37] Q. Xia, J. Duan, F. Gao, T. Chen, and C. Yang, “Automatic generation method of test scenario for adas based on complexity,” in SAE Technical Paper. SAE International, 09 2017. [Online]. Available: https://doi.org/10.4271/2017-01-1992 [38] R. B. Abdessalem, S. Nejati, L. Briand, and T. Stifter, “Testing vision-based control systems using learnable evolutionary algorithms,” in IEEE/ACM International Conference on Software Engineering, 2018, pp. 1016–1026. [39] L. Yu, Y. Lei, R. Kacker, and D. Kuhn, “Acts: A combinatorial test generation tool,” in Software Testing, Verification and Validation (ICST), 2013 IEEE Sixth International Conference on, 2013, pp. 370–375. [40] “Ipog: A general strategy for t-way software testing.” [Online]. Available: https://ws680.nist.gov/publication/get pdf.cfm?pub id=50944 [41] “The official site of the european new car assessment programme.” [Online]. Available: https://www.euroncap.com/en [42] N. Euro, “test protocolaeb systems,” Brussels, Belgium: Eur. New Car Assess. Programme (Euro NCAP), 2017. [43] N. Euro and A. V. T. Protocol, “v2. 0.2,” 2017. [44] “Openscenario.” [Online]. Available: http://www.openscenario.org/ [45] “Vtd-vires virtual test drive.” [Online]. Available: vtd-vires-virtual-test-drive/

https://vires.com/

[46] K. Vogel, “A comparison of headway and time to collision as safety indicators,” Accident analysis & prevention, vol. 35, no. 3, pp. 427–433, 2003. [47] R. Van Der Horst and J. Hogema, “Time-to-collision and collision avoidance systems,” in Proceedings of the 6th ICTCT workshop, Salzburg, Austria, January 1994. [48] W. W. Cohen, “Fast effective rule induction,” in Twelfth International Conference on Machine Learning. Morgan Kaufmann, 1995, pp. 115–123. 41

Declaration of interests ☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. ☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: