Publications

Edited Volumes

1. Software Engineering im Unterricht der Hochschulen, (SEUH 2024)

   Proceedings. Axel Schmolitzky, and Stefan Klikovits   editors.

   Feb 2024

   Bib Website

   @proceedings{DBLP:conf/seuh/2024,
     editor = {Schmolitzky, Axel and Klikovits, Stefan},
     title = {Software Engineering im Unterricht der Hochschulen, ({SEUH} 2024)},
     series = {{LNI}},
     volume = {{P-346}},
     publisher = {Gesellschaft f{\"{u}}r Informatik e.V.},
     year = {2024},
     month = feb,
     url = {https://dl.gi.de/handle/20.500.12116/43785},
     isbn = {978-3-88579-740-1},
   }

Book Chapters

1. CREST — a DSML for hybrid CPS modelling

   Stefan Klikovits, and Didier Buchs

   In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021

   Abs DOI: 10.1016/B978-0-12-819105-7.00014-3 Bib

   Hybrid systems modelling remains a very popular topic within the modelling and simulation community. Its expressiveness allows for the definition of highly complex systems that merge discrete-state-based transitions systems with continuous value-evolutions for variables, so that cyber-physical systems can be modelled in all their intricacies. This expressive power, however, comes at a downside of complex models and undecidable verification problems even for small systems. In this chapter we present CREST, a novel modelling language for the definition of hybrid systems. CREST merges features from various formalisms and languages such as hybrid automata, dataflow programming and internal domain-specific modelling language (DSML) designs to create simple yet powerful language for the modelling of resource flows within small-scale CPS such as automated gardening applications and smart homes. The language provides an easy to learn graphical interface and is supported by a Python-based tool implementation that allows for the efficient modelling, simulation and verification of CPS models.

   @incollection{Klikovits:2021:mpma4cps:ch9,
     title = {{CREST --- a DSML for hybrid CPS modelling}},
     editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco},
     booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems},
     publisher = {Academic Press},
     chapter = {9},
     pages = {221-242},
     year = {2021},
     isbn = {978-0-12-819105-7},
     doi = {https://doi.org/10.1016/B978-0-12-819105-7.00014-3},
     url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000143},
     author = {Klikovits, Stefan and Buchs, Didier},
     keywords = {peerreviewed}
   }

2. An ontology for multi-paradigm modelling

   Holger Giese, Dominique Blouin, Rima Al-Ali, Hana Mkaouar, Soumyadip Bandyopadhyay, and 4 more authors

   In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021

   Abs DOI: 10.1016/B978-0-12-819105-7.00009-X Bib

   This chapter presents the ontology for Multi-Paradigm Modelling (MPM) specified using the Web Ontology Language (OWL) as introduced in Chapter 2. A thorough state of the art on MPM’s core notions including multi-formalism and model management approaches, languages and tools is presented. In particular, model management approaches have been characterized according to their modularity and incremental execution properties as required to scale for the large complex CPSs we face today. Based on this state of the art, an outline of the MPM ontology is developed by introducing its main classes and properties. The validation of the ontology is presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2.

   @incollection{Klikovits:2021:mpma4cps:ch4,
     title = {{An ontology for multi-paradigm modelling}},
     editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco},
     booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems},
     publisher = {Academic Press},
     chapter = {4},
     pages = {67-122},
     year = {2021},
     isbn = {978-0-12-819105-7},
     doi = {https://doi.org/10.1016/B978-0-12-819105-7.00009-X},
     url = {https://www.sciencedirect.com/science/article/pii/B978012819105700009X},
     author = {Giese, Holger and Blouin, Dominique and Al-Ali, Rima and Mkaouar, Hana and Bandyopadhyay, Soumyadip and Iacono, Mauro and Amrani, Moussa and Klikovits, Stefan and Erata, Ferhat},
     keywords = {peerreviewed}
   }

3. An integrated ontology for multi-paradigm modelling for cyber-physical systems

   Dominique Blouin, Rima Al-Ali, Holger Giese, Stefan Klikovits, Soumyadip Bandyopadhyay, and 2 more authors

   In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021

   Abs DOI: 10.1016/B978-0-12-819105-7.00010-6 Bib

   This chapter presents the Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS) ontology. This ontology integrates the Shared, MPM and CPS ontologies respectively introduced in Chapters 2, 3 and 4. It includes cross-cutting notions such as viewpoints, model-based development processes and modelling paradigms that together relate the formalisms and workflows (and their paradigms) to the part of CPSs developed with these formalisms. A brief state of the art on these notions is first presented, on which the MPM4CPS ontology builds. An overview of the ontology is then developed by introducing its main classes and properties. The validation of the ontology is finally presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2. The chapter also discusses perspectives and future work on this integrated ontological framework, which can serve as a basis to develop model management solutions to relate and combine modelling languages and tools, in order to better develop cyber-physical systems with appropriate formalisms and workflows.

   @incollection{Klikovits:2021:mpma4cps:ch5,
     title = {{An integrated ontology for multi-paradigm modelling for cyber-physical systems}},
     author = {Blouin, Dominique and Al-Ali, Rima and Giese, Holger and Klikovits, Stefan and Bandyopadhyay, Soumyadip and Barišić, Ankica and Erata, Ferhat},
     booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems},
     editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco},
     chapter = {5},
     pages = {123-145},
     year = {2021},
     doi = {https://doi.org/10.1016/B978-0-12-819105-7.00010-6},
     url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000106},
     isbn = {978-0-12-819105-7},
     publisher = {Academic Press},
     keywords = {peerreviewed}
   }

4. Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event Systems

   Didier Buchs, Stefan Klikovits, and Alban Linard

   In Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems, Feb 2020

   Abs DOI: 10.1007/978-3-030-43946-0 Bib

   The study of concurrent and parallel systems has been a challenging research domain within cyber-physical systems community. This chapter provides a pragmatic introduction to the creation and analysis of such system models using the popular Petri nets formalism. Petri nets is a formalism that convinces through its simplicity and applicability. We offer an overview of the most important Petri nets concepts, analysis techniques and model checking approaches. Finally, we show the use of so-called High-level Petri nets for the representation of complex data structures and functionality and present a novel research approach that allows the use of Petri nets inside Functional Mock-up Units and cyber-physical system models.

   @incollection{Klikovits:2020:mpm4cps:foundationsbook,
     author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban},
     title = {{Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event Systems}},
     booktitle = {{Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems}},
     editor = {Carreira, P. and Amaral, V. and Vangheluwe, H.},
     publisher = {Springer},
     doi = {10.1007/978-3-030-43946-0},
     isbn = {978-3-030-43946-0},
     year = {2020},
     keywords = {peerreviewed}
   }

Journal Articles

1. A Method for Template-based Architecture Modeling and its Application to Digital Twins

   Daniel Lehner, Jérôme Pfeiffer, Stefan Klikovits, Andreas Wortmann, and Manuel Wimmer

   Journal of Object Technology, Jul 2024

   The 20th European Conference on Modelling Foundations and Applications (ECMFA 2024)

   Abs DOI: 10.5381/jot.2024.23.3.a8 Bib

   Digital Twins (DTs) have recently emerged to support domain experts in engineering and operating Cyber-Physical Systems (CPSs). As a result, software vendors started to create DT services offering advanced functionality to support the development and operation of DTs in the industry. However, the integration of services into a DT architecture is challenging. Services typically rely on specific software and modeling languages that are often not interoperable with other services. Hence, they have to be manually integrated which requires a significant, repetitive effort. Thus, currently, it is tedious to extend the DT’s underlying architectures with new services or exchange individual service implementations. In this paper, we propose a tool-supported method for architecture modeling and its application for digital twins. The presented method provides several steps to manage the complexity of current DT architectures. First, DT reference architectures are assembled by connecting DT templates, which provide an abstraction for a set of similar DT services. Second, dedicated DT modules are used to wrap existing services which provide concrete realizations of the DT templates. Third, a product-line-based generator supports the configuration of reference architectures into concrete architectures by selecting an appropriate set of modules for the used templates which are finally used in the derived integration solution. The transition from reference architecture modeling to the product-line-based configuration is supported by a dedicated model transformation. Our evaluation shows that the proposed DT templates enable the efficient modeling of different DT reference architectures and integration of new DT services into already existing architectures.

   @article{JOT:issue_2024_03/article8,
     author = {Lehner, Daniel and Pfeiffer, Jérôme and Klikovits, Stefan and Wortmann, Andreas and Wimmer, Manuel},
     title = {A Method for Template-based Architecture Modeling and its Application to Digital Twins},
     journal = {Journal of Object Technology},
     volume = {23},
     number = {3},
     issn = {1660-1769},
     year = {2024},
     month = jul,
     note = {The 20th European Conference on Modelling Foundations and Applications (ECMFA 2024)},
     doi = {10.5381/jot.2024.23.3.a8},
     url = {http://www.jot.fm/contents/issue_2024_03/article8.html},
   }

2. Digital Transformation in Tunneling–A Project Report on TransIT

   Thomas Bednar, Robert Galler, Christian Huemer, Marco Huymajer, Stefan Klikovits, and 5 more authors

   Zeitschrift für Hochschulentwicklung, Jul 2024

   Abs DOI: 10.21240/zfhe/SH-A/09 Bib

   The digital transformation has significantly affected many industries. Recently, it has gained momentum in the Architecture, Engineering, and Construction (AEC) sector in general and tunneling in particular. To support these developments, TransIT represents an inter-university and interdisciplinary research initiative with the goal of advancing digitalization in tunneling. This article discusses the challenges connected with the digital transformation of the tunneling sector. We examine this topic with different use cases demonstrating solutions to those challenges.

   @article{bednar2024digital,
     title = {{Digital Transformation in Tunneling--A Project Report on TransIT}},
     author = {Bednar, Thomas and Galler, Robert and Huemer, Christian and Huymajer, Marco and Klikovits, Stefan and Melnyk, Oleksandr and Paskaleva, Galina and Steiner, Bernhard and Wenighofer, Robert and Wimmer, Manuel},
     journal = {Zeitschrift f{\"u}r Hochschulentwicklung},
     volume = {19},
     number = {Sonderheft Administration},
     pages = {143--163},
     year = {2024},
     doi = {10.21240/zfhe/SH-A/09},
     keywords = {peerreviewed}
   }

3. Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for ADS Testing

   Stefan Klikovits, Ezequiel Castellano, Ahmet Cetinkaya, and Paolo Arcaini

   Science of Computer Programming, Jul 2023

   Abs DOI: 10.1016/j.scico.2023.102996 Bib

   Being capable of identifying significant safety shortcomings, search-based methods are a core tool for testing automated driving system (ADS) technologies. In this domain, Frenetic has proven to be a popular and very effective tool, searching and identifying diverse sets of roads that point out potentially faulty ADS behaviour. This paper presents Frenetic-lib, a Python library that captures Frenetic’s novel combination of road representation and genetic algorithm, and makes it generally available in a customisable way. Next to the capacity to integrate additional ADS simulators, Frenetic-lib further creates new research opportunities on search-based road testing, novel road representations and mutation operators.

   @article{Klikovits:freneticlib:2023,
     title = {{Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for {ADS} Testing}},
     journal = {Science of Computer Programming},
     volume = {230},
     pages = {102996},
     date = {2023-07},
     year = {2023},
     month = jul,
     issn = {0167-6423},
     doi = {10.1016/j.scico.2023.102996},
     url = {https://www.sciencedirect.com/science/article/pii/S0167642323000783},
     author = {Klikovits, Stefan and Castellano, Ezequiel and Cetinkaya, Ahmet and Arcaini, Paolo},
     keywords = {peerreviewed},
   }

4. Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty

   Thomas Laurent, Stefan Klikovits, Paolo Arcaini, Fuyuki Ishikawa, and Anthony Ventresque

   ACM Transactions on Software Engineering Methodology (TOSEM), Apr 2023

   Abs DOI: 10.1145/3550270 Bib

   Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.

   @article{10.1145/3550270,
     author = {Laurent, Thomas and Klikovits, Stefan and Arcaini, Paolo and Ishikawa, Fuyuki and Ventresque, Anthony},
     title = {Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty},
     year = {2023},
     issue_date = {July 2023},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {32},
     number = {3},
     issn = {1049-331X},
     url = {https://doi.org/10.1145/3550270},
     doi = {10.1145/3550270},
     journal = {ACM Transactions on Software Engineering Methodology (TOSEM)},
     month = apr,
     articleno = {58},
     numpages = {31},
     keywords = {peerreviewed},
   }

5. Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averaging

   Stefan Klikovits, Cédric Ho Thanh, Ahmet Cetinkaya, and Paolo Arcaini

   Applied Soft Computing, Apr 2023

   Abs DOI: 10.1016/j.asoc.2023.110631 Bib

   Multi-objective optimisation (MOO) is a popular approach for finding solutions to many types of complex problems with large search spaces and conflicting search objectives. In the past, MOO algorithms have been shown to reliably produce good optimisation results. With the rise of cyber–physical systems, however, emerges the new challenge of non-deterministic system executions, caused by e.g. imperfect sensor readings or synchronisation in multi-process/multi-agent system architectures. These systems produce varying output on each execution, causing the algorithms’ observations to be noisy. Naturally, MOO algorithms favour the fittest solutions, which may have been measured with great inaccuracy. The end results are therefore not trustworthy. In this paper, we propose kNN-averaging, a new method to address this issue by identifying the k-nearest neighbours (kNN) of a solution, and using their weighted average as an estimate for its true fitness. Our experiments demonstrate the viability of kNN-averaging on 40 synthetic benchmark problems and on a real-world case study system. In the process, we compare kNN-averaging to the noisy baseline as well as two resampling-based methods and one spectral sampling approach on a range of algorithm settings. The results show that kNN-averaging approximates the fitness of solutions more accurately than the noisy baseline, leading to more trustworthy results.

   @article{Klikovits:2023:2023110631,
     title = {{Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averaging}},
     journal = {Applied Soft Computing},
     volume = {146},
     pages = {110631},
     date = {2023-10},
     year = {2023},
     issn = {1568-4946},
     doi = {10.1016/j.asoc.2023.110631},
     url = {https://www.sciencedirect.com/science/article/pii/S156849462300649X},
     author = {Klikovits, Stefan and {Ho Thanh}, Cédric and Cetinkaya, Ahmet and Arcaini, Paolo},
     keywords = {peerreviewed},
   }

6. Pragmatic Reuse for DSML Development

   Stefan Klikovits, and Didier Buchs

   Software and Systems Modeling (SoSyM), Apr 2021

   Abs DOI: 10.1007/s10270-020-00831-4 Bib

   By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.

   @article{Klikovits:2020:sosym:crest,
     author = {Klikovits, Stefan and Buchs, Didier},
     title = {{Pragmatic Reuse for DSML Development}},
     journal = {{Software and Systems Modeling (SoSyM)}},
     doi = {10.1007/s10270-020-00831-4},
     year = {2021},
     volume = {20},
     pages = {837-866},
     keywords = {peerreviewed}
   }

Conference Workshop

1. Leveraging Software Product Lines for Testing Automated Driving Systems

   Stefan Klikovits, Alessio Gambi, Deepak Dhungana, and Rick Rabiser

   In 18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024), Bern, Switzerland, Feb 2024

   Abs DOI: 10.1145/3634713.3634720 Bib PDF

   Extensive testing of Automated Driving Systems (ADS), such as Advanced Driver Assistance Systems and Autonomous Vehicles, is commonly conducted using simulators programmed to implement various driving scenarios, a technique known as scenario-based testing. ADS scenario-based testing using simulations is challenging because it requires identifying scenarios that can effectively test ADS functionalities while ensuring that driving simulators’ features match the driving scenarios’ requirements. This short paper discusses the main challenges of systematically conducting simulation-based testing and proposes leveraging Software Product Line techniques to address them. Specifically, we argue that variability models can be used to support testers in generating test scenarios by effectively capturing and relating the variability in driving simulators, testing scenarios, and ADS implementations. We conclude by outlining an agenda for future research in this important area.

   @inproceedings{Klikovits:2024:VAMOS,
     author = {Klikovits, Stefan and Gambi, Alessio and Dhungana, Deepak and Rabiser, Rick},
     title = {{Leveraging Software Product Lines for Testing Automated Driving Systems}},
     booktitle = {18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024)},
     year = {2024},
     month = feb,
     doi = {10.1145/3634713.3634720},
     location = {Bern, Switzerland},
     keywords = {peerreviewed}
   }

2. Model-Driven Optimization for Quantum Program Synthesis with MOMoT

   Felix Gemeinhardt, Martin Eisenberg, Stefan Klikovits, and Manuel Wimmer

   In 5th Workshop on Artificial Intelligence and Model-driven Engineering, Väster\aas, Sweden, Oct 2023

   Abs DOI: 10.1109/MODELS-C59198.2023.00100 Bib PDF

   In the realm of classical software engineering, model-driven optimization has been widely used for different problems such as (re)modularization of software systems. In this paper, we investigate how techniques from model-driven optimization can be applied in the context of quantum software engineering. In quantum computing, creating executable quantum programs is a highly non-trivial task which requires significant expert knowledge in quantum information theory and linear algebra. Although different approaches for automated quantum program synthesis exist—e.g., based on reinforcement learning and genetic programming—these approaches represent tailor-made solutions requiring dedicated encodings for quantum programs. This paper applies the existing model-driven optimization approach MOMoT to the problem of quantum program synthesis. We present the resulting platform for experimenting with quantum program synthesis and present a concrete demonstration for a well-known quantum algorithm.

   @inproceedings{Klikovits:2023:MDEIntelligence,
     author = {Gemeinhardt, Felix and Eisenberg, Martin and Klikovits, Stefan and Wimmer, Manuel},
     title = {{Model-Driven Optimization for Quantum Program Synthesis with MOMoT}},
     booktitle = {5th Workshop on Artificial Intelligence and Model-driven Engineering},
     year = {2023},
     month = oct,
     doi = {10.1109/MODELS-C59198.2023.00100},
     location = {V\"{a}ster\aas, Sweden},
     keywords = {peerreviewed}
   }

3. Towards Generating Model-Driven Speech Interfaces for Digital Twins

   Ramiya Jayaraman, Daniel Lehner, Stefan Klikovits, and Manuel Wimmer

   In 3rd International Workshop on Model-Driven Engineering of Digital Twins, Väster\aas, Sweden, Oct 2023

   Abs DOI: 10.1109/MODELS-C59198.2023.00080 Bib PDF

   The recent decade saw enormous advances of the capabilities of speech synthesis and speech recognition. While specific benefits depend on the individual applications, speech interfaces typically increase accessibility, enable “hands-free” and “no-screen” interaction, and often increase interaction speed and allow for more flexible usage patterns and increased multitasking, leading to higher user satisfaction. This paper presents a method to transfer these powerful benefits to the digital twin (DT) domain by automatically generating speech interfaces for DT applications. Our approach is based on Model Driven Engineering principles, where we automatically deduce command patterns from structural model information as used in, e.g., DT platform specifications. The speech interface generation is highly configurable and extendable, thus it can be applied to different DT models. The concepts behind the generator are generic as well, thus they can be ported to other applications and platforms. We validate our approach by applying it to two DT demonstration cases and provide a detailed description of the sketch interface configuration workflow.

   @inproceedings{Klikovits:2023:MODDIT,
     author = {Jayaraman, Ramiya and Lehner, Daniel and Klikovits, Stefan and Wimmer, Manuel},
     title = {{Towards Generating Model-Driven Speech Interfaces for Digital Twins}},
     booktitle = {3rd International Workshop on Model-Driven Engineering of Digital Twins},
     year = {2023},
     month = oct,
     doi = {10.1109/MODELS-C59198.2023.00080},
     location = {V\"{a}ster\aas, Sweden},
     keywords = {peerreviewed}
   }

4. Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator Discovery

   Felix Gemeinhardt, Stefan Klikovits, and Manuel Wimmer

   In Genetic and Evolutionary Computation Conference Companion (GECCO’23), Lisbon, Portugal, Jul 2023

   Abs DOI: 10.1145/3583133.3590696 Bib

   The processing of quantum information is defined by quantum circuits. For applications on current quantum devices, these are usually parameterized, i.e., they contain operations with variable parameters. The design of such quantum circuits and aggregated higher-level quantum operators is a challenging task which requires significant knowledge in quantum information theory, provided a polynomial-sized solution can be found analytically at all. Moreover, finding an accurate solution with low computational cost represents a significant trade-off, particularly for the current generation of quantum computers. To tackle these challenges, we propose a multi-objective genetic programming approach—hybridized with a numerical parameter optimizer—to automate the synthesis of parameterized quantum operators. To demonstrate the benefits of the proposed approach, it is applied to a quantum circuit of a hybrid quantum-classical algorithm, and then compared to an analytical solution as well as a non-hybrid version. The results show that, compared to the non-hybrid version, our method produces more diverse solutions and more accurate quantum operators which even reach the quality of the analytical baseline.

   @inproceedings{10.1145/3583133.3590696,
     author = {Gemeinhardt, Felix and Klikovits, Stefan and Wimmer, Manuel},
     title = {Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator Discovery},
     booktitle = {Genetic and Evolutionary Computation Conference Companion (GECCO'23)},
     pages = {795–798},
     numpages = {4},
     date = {2023-07},
     year = {2023},
     month = jul,
     publisher = {ACM},
     keywords = {peerreviewed},
     doi = {10.1145/3583133.3590696},
     location = {Lisbon, Portugal},
     series = {GECCO '23 Companion},
   }

5. SBFT Tool Competition 2023 - Cyber-Physical Systems Track

   Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo Riccio

   In 16th IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT 2023, Melbourne, Australia, May 14, 2023, May 2023

   DOI: 10.1109/SBFT59156.2023.00010 Bib Website

   @inproceedings{SBFT-toolcomp23,
     author = {Biagiola, Matteo and Klikovits, Stefan and Peltom\"{a}ki, Jarkko and Riccio, Vincenzo},
     title = {{SBFT} Tool Competition 2023 - Cyber-Physical Systems Track},
     booktitle = {16th {IEEE/ACM} International Workshop on Search-Based and Fuzz Testing, {SBFT} 2023, Melbourne, Australia, May 14, 2023},
     year = {2023},
     month = may,
     pages = {45-48},
     doi = {10.1109/SBFT59156.2023.00010},
   }

6. FreneticV at the SBST 2022 Tool Competition

   Ezequiel Castellano, Stefan Klikovits, Ahmet Cetinkaya, and Paolo Arcaini

   In 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), May 2022

   Abs DOI: 10.1145/3526072.3527532 Bib Video

   FreneticV is a search-based testing tool based on an evolutionary approach that generates roads where an automated driving agent possibly fails the lane-keeping task. It uses a curvature-based road representation and, compared to its predecessor Frenetic, considers the validity of the generated roads. In particular, it tries to avoid generating roads with overly sharp turns, detects self-intersecting roads, and can rotate and relocate roads to fit them in a given map.

   @inproceedings{Castellano2022FreneticVAT,
     title = {{FreneticV} at the {SBST} 2022 Tool Competition},
     author = {Castellano, Ezequiel and Klikovits, Stefan and Cetinkaya, Ahmet and Arcaini, Paolo},
     booktitle = {2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)},
     year = {2022},
     month = may,
     pages = {47-48},
     doi = {10.1145/3526072.3527532},
   }

7. Dynamic Shielding for Reinforcement Learning in Black-Box Environments

   Masaki Waga, Ezequiel Castellano, Sasinee Pruekprasert, Stefan Klikovits, Toru Takisaka, and 1 more author

   In Automated Technology for Verification and Analysis (ATVA’22), May 2022

   Abs DOI: 10.1007/978-3-031-19992-9_2 Bib

   It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.

   @inproceedings{10.1007/978-3-031-19992-9_2,
     author = {Waga, Masaki and Castellano, Ezequiel and Pruekprasert, Sasinee and Klikovits, Stefan and Takisaka, Toru and Hasuo, Ichiro},
     editor = {Bouajjani, Ahmed and Hol{\'i}k, Luk{\'a}{\v{s}} and Wu, Zhilin},
     title = {Dynamic Shielding for Reinforcement Learning in Black-Box Environments},
     booktitle = {Automated Technology for Verification and Analysis (ATVA'22)},
     year = {2022},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {25--41},
     isbn = {978-3-031-19992-9},
     doi = {10.1007/978-3-031-19992-9_2},
     keywords = {peerreviewed}
   }

8. Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems

   Stefan Klikovits, and Paolo Arcaini

   In 26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021), Nov 2021

   Abs DOI: 10.1109/PRDC53464.2021.00011 Bib

   This paper presents the first evaluation of k-nearest neighbours-Averaging (kNN-Avg) on a real-world case study. kNN-Avg is a novel technique that tackles the challenges of noisy multi-objective optimisation (MOO). Existing studies suggest the use of repetition to overcome noise. In contrast, kNN-Avg approximates these repetitions and exploits previous executions, thereby avoiding the cost of re-running. We use kNN-Avg for the scenario generation of a real-world autonomous driving system (ADS) and show that it is better than the noisy baseline. Furthermore, we compare it to the repetition-method and outline indicators as to which approach to choose in which situations.

   @inproceedings{KlikovitsA2021PRDC,
     title = {Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems},
     author = {Klikovits, Stefan and Arcaini, Paolo},
     booktitle = {26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021)},
     publisher = {IEEE},
     year = {2021},
     month = nov,
     doi = {10.1109/PRDC53464.2021.00011},
     keywords = {peerreviewed}
   }

9. On the Need for Multi-Level ADS Scenarios

   Stefan Klikovits, and Paolo Arcaini

   In 3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS’21), Oct 2021

   Abs DOI: 10.1109/MODELS-C53483.2021.00054 Bib

   Currently, most existing approaches for the design of Automated Driving Systems (ADS) scenarios focus on the description at one particular abstraction level — typically the most detailed one. This practice often removes information at higher levels, such that this data has to be re-synthesized if needed. As the abstraction granularity should be adapted to the task at hand, however, engineers currently have the choice between re-calculating the needed data or operating on the wrong level of abstraction. For instance, the search in a scenario database for a driving scenario with a map of a given road-shape should abstract over the lane markings, adjacent vegetation, or weather situation. Often though, the general road shape has to be synthesized (e.g. interpolated) from the precise GPS information of road boundaries. This paper outlines our vision for multi-level ADS scenario models that facilitate scenario search, generation, and design. Our concept is based on the common modelling philosophy to interact with scenarios at the most appropriate abstraction level. We identify different abstraction levels of ADS scenarios and suggest a template abstraction hierarchy. Our vision enables seamless traversal to such a most suitable granularity level for any given scenario, search and modelling task. We envision that this approach to ADS scenario modelling will have a lasting impact on the way we store, search, design, and generate ADS scenarios, allowing for a more strategic verification of autonomous vehicles in the long run.

   @inproceedings{KlikovitsA2021MPM4CPS,
     title = {On the Need for Multi-Level {ADS} Scenarios},
     author = {Klikovits, Stefan and Arcaini, Paolo},
     booktitle = {3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'21)},
     series = {MODELS'21},
     publisher = {ACM},
     year = {2021},
     month = oct,
     pages = {322-326},
     doi = {10.1109/MODELS-C53483.2021.00054},
     keywords = {peerreviewed}
   }

10. KNN-Averaging for Noisy Multi-Objective Optimisation

    Stefan Klikovits, and Paolo Arcaini

    In Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC), Faro, Portugal (Online), Sep 2021

    Abs DOI: 10.1007/978-3-030-85347-1_36 Bib

    Multi-objective optimisation is a popular approach for finding solutions to complex problems with large search spaces that reliably yields good optimisation results. However, with the rise of cyber-physical systems, emerges a new challenge of noisy fitness functions, whose objective value for a given configuration is non-deterministic, producing varying results on each execution. This leads to an optimisation process that is based on stochastically sampled information, ultimately favouring solutions with fitness values that have co-incidentally high outlier noise. In turn, the results are unfaithful due to their large discrepancies between sampled and expectable objective values. Motivated by our work on noisy automated driving systems, we present the results of our ongoing research to counteract the effect of noisy fitness functions without requiring repeated executions of each solution. Our method kNN-Avg identifies the k-nearest neighbours of a solution point and uses the weighted average value as a surrogate for its actually sampled fitness. We demonstrate the viability of kNN-Avg on common benchmark problems and show that it produces comparably good solutions whose fitness values are closer to the expected value.

    @inproceedings{KlikovitsA2021knnAveraging,
      author = {Klikovits, Stefan and Arcaini, Paolo},
      title = {{KNN-Averaging for Noisy Multi-Objective Optimisation}},
      editor = {Paiva, Ana C. R. and Cavalli, Ana Rosa and Ventura Martins, Paula and P{\'e}rez-Castillo, Ricardo},
      booktitle = {Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC)},
      publisher = {Springer International Publishing},
      pages = {503--518},
      series = {Communications in Computer and Information Science (CCIS)},
      volume = {1439},
      isbn = {978-3-030-85347-1},
      location = {Faro, Portugal (Online)},
      doi = {10.1007/978-3-030-85347-1_36},
      year = {2021},
      month = sep,
      keywords = {peerreviewed}
    }

11. Architecture-Guided Test Resource Allocation via Logic

    Clovis Eberhart, Akihisa Yamada, Stefan Klikovits, Shin-ya Katsumata, Tsutomu Kobayashi, and 2 more authors

    In Tests and Proofs (TAP 2021), Jun 2021

    Abs DOI: 10.1007/978-3-030-79379-1_2 arXiv Bib

    We introduce a new logic named Quantitative Confidence Logic (QCL) that quantifies the level of confidence one has in the conclusion of a proof. By translating a fault tree representing a system’s architecture to a proof, we show how to use QCL to give a solution to the test resource allocation problem that takes the given architecture into account. We implemented a tool called Astrahl and compared our results to other testing resource allocation strategies.

    @inproceedings{Eberhart2021,
      author = {Eberhart, Clovis and Yamada, Akihisa and Klikovits, Stefan and Katsumata, Shin-ya and Kobayashi, Tsutomu and Hasuo, Ichiro and Ishikawa, Fuyuki},
      title = {Architecture-Guided Test Resource Allocation via Logic},
      booktitle = {Tests and Proofs (TAP 2021)},
      series = {Lecture Notes in Computer Science (LNCS)},
      volume = {12740},
      doi = {10.1007/978-3-030-79379-1_2},
      year = {2021},
      month = jun,
      publisher = {Springer},
      pages = {22--38},
      keywords = {peerreviewed}
    }

12. Frenetic at the SBST 2021 Tool Competition

    Ezequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, and 1 more author

    In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), May 2021

    Abs DOI: 10.1109/SBST52555.2021.00016 Bib Video

    Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimise the out of bound distance, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.

    @inproceedings{Castellano:2021:SBST,
      author = {Castellano, Ezequiel and Cetinkaya, Ahmet and Thanh, Cédric Ho and Klikovits, Stefan and Zhang, Xiaoyi and Arcaini, Paolo},
      title = {Frenetic at the {SBST} 2021 Tool Competition},
      booktitle = {2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)},
      year = {2021},
      month = may,
      editor = {Zhang, Jie and Fredericks, Erik},
      pages = {36-37},
      publisher = {IEEE},
      keywords = {genetic algorithms, genetic programming},
      doi = {10.1109/SBST52555.2021.00016},
    }

13. ML4CREST: Machine Learning for CPS Models

    Stefan Klikovits, Aurélien Coet, and Didier Buchs

    In 2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT), Oct 2018

    Abs Bib PDF Slides

    Models of small CPS and IoT applications often use approximated values that describe physical system behaviour. Physical resources, such as electricity consumption and heating power, have to be estimated, since many off-the-shelf components lack the required descriptions. Controllers which are based on these approximations can hence use imprecise models, perform misleading simulation, and cause damaged systems and financial loss. In this paper we present ML4CREST, a machine learning approach to automatically calibrate models using sensor measurements. We show that our approach is well-suited for the calibration of the flow rates within an automated watering system, which allows precise simulation and prevents spillage.

    @inproceedings{Klikovits:MDE4IOT:CREST,
      title = {{ML4CREST: Machine Learning for CPS Models}},
      author = {Klikovits, Stefan and Coet, Aur\'{e}lien and Buchs, Didier},
      year = {2018},
      month = oct,
      pages = {515-520},
      url = {http://ceur-ws.org/Vol-2245/mde4iot_paper_4.pdf},
      volume = {2245},
      series = {CEUR Workshop Proceedings},
      booktitle = {2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT)},
      keywords = {peerreviewed}
    }

14. CREST - A DSL for Reactive Cyber-Physical Systems

    Stefan Klikovits, Alban Linard, and Didier Buchs

    In 10th System Analysis and Modeling Conference (SAM2018). Languages, Methods, and Tools for Systems Engineering, Oct 2018

    Abs DOI: 10.1007/978-3-030-01042-3_3 Bib Slides

    This article presents CREST, a novel domain-specific language for the modelling of cyber-physical systems. CREST is designed for the simple and clear modelling, simulation and verification of small-scale systems such as home and office automation, smart gardening systems and similar. The language is designed to model the flow of resources throughout the system. It features synchronous system evolution and reactive behaviour. CREST’s formal semantics allow real-valued time advances and the modelling of timed system evolution. The continuous time concept permits the precise simulation of future system behaviour by automatically calculating next transition times. We present CREST in a practical manner, and elaborate on the Python-based DSL implementation and simulator.

    @inproceedings{Klikovits:SAM18:CREST,
      title = {{{CREST}} - {{A DSL}} for {{Reactive Cyber}}-{{Physical Systems}}},
      isbn = {978-3-030-01042-3},
      booktitle = {10th System {{Analysis}} and {{Modeling}} {{Conference}} ({{SAM2018}}). {{Languages}}, {{Methods}}, and {{Tools}} for {{Systems Engineering}}},
      publisher = {{Springer}},
      series = {Lecture Notes in Computer Science},
      volume = {11150},
      author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier},
      editor = {Khendek, Ferhat and Gotzhein, Reinhard},
      year = {2018},
      month = oct,
      pages = {29-45},
      doi = {10.1007/978-3-030-01042-3_3},
      keywords = {peerreviewed}
    }

15. Petri Sport: A Sport for Petri Netters

    Stefan Klikovits, Alban Linard, Dimitri Racordon, and Didier Buchs

    In Petri Nets and Software Engineering. International Workshop, PNSE’18, Sep 2018

    Abs Bib PDF Slides

    Petri nets are a family of formalisms dedicated to the representation of concurrent systems. Their strength is the compact modeling of complex behaviors using very simple rules. Despite this simplicity, many teachers observe that students often require a lot of exposure and numerous exercises to truly understand the semantics of Petri nets. In order to speed up this learning process and provide a different attack angle, we propose Petri sport, a fun game based upon the Petri net formalism. In Petri sport, players aim to gather points by moving across a Petri net-shaped playing field and "firing" transitions. A clock-based play style supports a structured game advance while at the same time it encourages players to move fast. As the playing field is shaped like a Petri net, it is possible to challenge a player’s movement speed, intellectual capabilities, as well as team coordination and communication. The difficulty level of Petri sport is based on the choice of playing field. This allows for adaptation in order to best fit the competitors’ age, experience and/or physical fitness level.

    @inproceedings{Klikovits:PNSE18:PetriSport,
      title = {{Petri Sport: A Sport for Petri Netters}},
      pages = {35--56},
      author = {Klikovits, Stefan and Linard, Alban and Racordon, Dimitri and Buchs, Didier},
      year = {2018},
      month = sep,
      booktitle = {Petri Nets and Software Engineering. International Workshop, PNSE'18},
      editor = {Moldt, Daniel and Kindler, Ekkart and R{\"o}lke, Heiko},
      series = {CEUR Workshop Proceedings},
      volume = {2138},
      url = {http://ceur-ws.org/Vol-2138/paper2.pdf},
      keywords = {peerreviewed}
    }

16. A Model Checker Collection for the Model Checking Contest Using Docker and Machine Learning

    Didier Buchs, Stefan Klikovits, Alban Linard, Romain Mencattini, and Dimitri Racordon

    In Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, PETRI NETS 2018, Sep 2018

    Abs DOI: 10.1007/978-3-319-91268-4_21 Bib Slides

    This paper introduces mcc4mcc, the Model Checker Collection for the Model Checking Contest, a tool that wraps multiple model checking solutions, and applies the most appropriate one based on the characteristics of the model it is given. It leverages machine learning algorithms to carry out this selection, based on the results gathered from the 2017 edition of the Model Checking Contest, an annual event in which multiple tools compete to verify different properties on a large variety of models. Our approach brings two important contributions. First, our tool offers the opportunity to further investigate on the relation between model characteristics and verification techniques. Second, it lays out the groundwork for a unified way to distribute model checking software using virtual containers.

    @inproceedings{DBLP:conf/apn/BuchsKLMR18,
      author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban and Mencattini, Romain and Racordon, Dimitri},
      title = {{A Model Checker Collection for the Model Checking Contest Using Docker and Machine Learning}},
      booktitle = {Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, {PETRI} {NETS} 2018},
      pages = {385--395},
      year = {2018},
      month = sep,
      series = {Lecture Notes in Computer Science},
      volume = {10877},
      publisher = {Springer},
      doi = {10.1007/978-3-319-91268-4_21},
      url = {https://doi.org/10.1007/978-3-319-91268-4_21},
      keywords = {peerreviewed}
    }

17. CREST - A Continuous, REactive SysTems DSL

    Stefan Klikovits, Alban Linard, and Didier Buchs

    In 5th International Workshop on the Globalization of Modeling Languages (GEMOC 2017), 19 September 2017, Austin, TX, USA, Sep 2017

    Abs Bib Slides

    The advance of cyber-physical systems in everyday life requires powerful modeling capabilities. Existing formalisms often have severe limitations and require complicated notations. In this paper we introduce CREST, a domain-specific language for modeling entity behavior and resource transfers in CPS. CREST aims to support CPS architects through clarity, comprehensiveness and analyzability.

    @inproceedings{DBLP:conf/models/KlikovitsLB17,
      author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier},
      title = {{CREST} - {A} Continuous, REactive SysTems {DSL}},
      booktitle = {5th International Workshop on the {Globalization of Modeling Languages} {(GEMOC} 2017), 19 September 2017, Austin, TX, USA},
      pages = {286--291},
      year = {2017},
      month = sep,
      url = {http://ceur-ws.org/Vol-2019/gemoc_2.pdf},
      volume = {2019},
      series = {CEUR Workshop Proceedings},
      keywords = {peerreviewed}
    }

18. Modeling Frames

    Stefan Klikovits, Joachim Denil, Alexandre Muzy, and Rick Salay

    In 14th Workshop on Model-Driven Engineering, Verification and Validation (MoDeVVa 2017), 19 September 2017, Austin, TX, USA, Sep 2017

    Abs Bib Slides

    Modeling activities such as calibration, verification and validation are often executed in under-specified environments. This hinders reproducibility, reduces re-usability and generally decreases the modeling precision and quality. This paper describes a framework for the definition of model frames. Model frames capture the context an activity/model is executed in and facilitate re-use, replacement, validation and verification of models. We show the application of the frames approach onto a real-world example, introduce several kinds of frames and show their application on this case study.

    @inproceedings{DBLP:conf/models/KlikovitsDMS17,
      author = {Klikovits, Stefan and Denil, Joachim and Muzy, Alexandre and Salay, Rick},
      title = {Modeling Frames},
      booktitle = {14th Workshop on  {Model-Driven Engineering, Verification and Validation} {(MoDeVVa} 2017), 19 September 2017, Austin, TX, USA},
      volume = {2019},
      series = {CEUR Workshop Proceedings},
      pages = {315--320},
      year = {2017},
      month = sep,
      keywords = {peerreviewed}
    }

19. The Experiment Model and Validity Frame in M&S

    Joachim Denil, Stefan Klikovits, Pieter J. Mosterman, Antonio Vallecillo, and Hans Vangheluwe

    In Proceedings of the Symposium on Theory of Modeling & Simulation, Apr 2017

    Abs Bib

    Modelling and Simulation approaches use system models to conduct simulation experiments. Experimental frames have been applied in this context to formally define a system’s context. During the creation of an experimental frame for a simple spring model it becomes clear that experimental frames in their current definition lack certain properties and omit relevant information. Our approach describes the process of capturing the context of models and systems to provide truly reproducible experiment descriptions. The information, captured as experimental setups, can then be used for different purposes and in different scenarios, in particular for checking the validity of a model, the discovery of suitable models for the design of a system, and for calibrating models.

    @inproceedings{denil:2017:tmsdevs,
      author = {Denil, Joachim and Klikovits, Stefan and Mosterman, Pieter J. and Vallecillo, Antonio and Vangheluwe, Hans},
      title = {The Experiment Model and Validity Frame in M\&S},
      year = {2017},
      month = apr,
      booktitle = {Proceedings of the Symposium on Theory of Modeling \& Simulation},
      series = {TMS/DEVS '17},
      pages = {109--120},
      isbn = {ISBN: 978-1-5108-3823-9},
      venue = {Virginia Beach, Virginia, USA},
      url = {http://scs.org/wp-content/uploads/2017/06/27_Final_Manuscript.pdf},
      keywords = {peerreviewed}
    }

20. Towards Language Independent (Dynamic) Symbolic Execution

    Stefan Klikovits, Manuel Gonzalez-Berges, and Didier Buchs

    In Proceedings of the 24th PhD Mini-Symposium, Jan 2017

    Abs DOI: 10.5281/zenodo.291899 Bib Slides

    Symbolic execution is well-known for its capability to produce high-coverage test suites for software source code. So far, most tools are created to support a specific language. This paper elaborates on performing language independent symbolic execution and three ways to achieve it. We describe the use of one approach to perform dynamic symbolic execution using translation of a proprietary language and show the results of the tool execution on a real-life codebase.

    @inproceedings{klikovits:2017:minisy,
      author = {Klikovits, Stefan and Gonzalez{-}Berges, Manuel and Buchs, Didier},
      title = {Towards Language Independent (Dynamic) Symbolic Execution},
      year = {2017},
      month = jan,
      booktitle = {Proceedings of the 24th PhD Mini-Symposium},
      pages = {50--53},
      isbn = {978-963-313-243-2},
      venue = {Budapest, Hungary},
      url = {https://doi.org/10.5281/zenodo.291899},
      publisher = {{Budapest University of Technology and Economics, Department of Measurement and Information Systems}},
      doi = {10.5281/zenodo.291899},
      keywords = {peerreviewed}
    }

21. Automated Test Case Generation for the CTRL Programming Language Using Pex: Lessons Learned

    Stefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and Didier Buchs

    In 8th International Workshop on Software Engineering for Resilient Systems (SERENE), Sep 2016

    Abs DOI: 10.1007/978-3-319-45892-2_9 Bib Slides

    Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. In this paper we first present ITEC, a tool for automated test case generation in CTRL, as well as initial results of test cases executions on one of CERN’s SCADA frameworks. Our tool relies on Microsoft’s Pex for its code exploration. For the purpose of using this existing test generation tool, we have to translate the proprietary CTRL code into C#, one of Pex’s operating languages. Our main contribution lies in detailing a formal foundation for this step through source code decomposition and anonymization. We then propose a quality measure that is used to determine our confidence into the translation and the generated test cases.

    @inproceedings{DBLP:conf/serene/KlikovitsLGB16,
      author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier},
      title = {Automated Test Case Generation for the {CTRL} Programming Language Using Pex: Lessons Learned},
      booktitle = {8th International Workshop on Software Engineering for Resilient Systems (SERENE)},
      pages = {117--132},
      publisher = {Springer},
      series = {Lecture Notes in Computer Science},
      volume = {9823},
      year = {2016},
      month = sep,
      url = {https://doi.org/10.1007/978-3-319-45892-2_9},
      doi = {10.1007/978-3-319-45892-2_9},
      keywords = {peerreviewed}
    }

22. Considering Execution Environment Resilience: A White-Box Approach

    Stefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and Didier Buchs

    In 7th International Workshop on Software Engineering for Resilient Systems (SERENE), Sep 2015

    Abs DOI: 10.1007/978-3-319-23129-7_4 Bib Slides

    Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. Only few of them consider dependencies from outside the Code Under Test’s scope such as global variables, database values and subroutine calls as influences to the execution path. In order to fully test all possible scenarios these dependencies have to be taken into account for the test input generation. This paper introduces ITEC, a tool for automated test case generation to support execution environment resilience in large-scaled, complex systems. One of ITEC’s corner stones is a technique called semi-purification, a source code transformation technique to overcome limitations of existing tools and to set up the required system state for software testing.

    @inproceedings{DBLP:conf/serene/KlikovitsLGB15,
      author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier},
      title = {Considering Execution Environment Resilience: {A} White-Box Approach},
      booktitle = {7th International Workshop on Software Engineering for Resilient Systems (SERENE)},
      pages = {46--61},
      publisher = {Springer},
      series = {Lecture Notes in Computer Science},
      volume = {9274},
      month = sep,
      year = {2015},
      url = {https://doi.org/10.1007/978-3-319-23129-7_4},
      doi = {10.1007/978-3-319-23129-7_4},
      keywords = {peerreviewed}
    }

23. Multi-platform SCADA GUI Regression Testing at CERN

    Paul C. Burkimsher, Manuel Gonzalez-Berges, and Stefan Klikovits

    In Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS), Sep 2011

    Abs Bib PDF Slides

    The JCOP Framework is a toolkit used widely at CERN for the development of industrial control systems in several domains (i.e. experiments, accelerators and technical infrastructure). The software development started 10 years ago and there is now a large base of production systems running it. For the success of the project, it was essential to formalize and automate the quality assurance process. The paper will present the overall testing strategy and will describe in detail mechanisms used for GUI testing. The choice of a commercial tool (Squish) and the architectural features making it appropriate for our multi-platform environment will be described. Practical difficulties encountered when using the tool in the CERN context are discussed as well as how these were addressed. In the light of initial experience, the test code itself has been recently reworked in OO style to facilitate future maintenance and extension. The paper concludes with a description of our initial steps towards incorporation of full-blown Continuous Integration (CI) support.

    @inproceedings{burkimsher2011icalepcs,
      author = {Burkimsher, Paul C. and Gonzalez{-}Berges, Manuel and Klikovits, Stefan},
      title = {Multi-platform {SCADA} {GUI} Regression Testing at {CERN}},
      booktitle = {Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS)},
      year = {2011},
      venue = {Grenoble, France},
      publisher = {JACoW},
      url = {https://accelconf.web.cern.ch/icalepcs2011/papers/thbhmust01.pdf},
      keywords = {peerreviewed}
    }

Theses

1. A Domain-Specific Language Approach To Hybrid CPS Modelling

   Stefan Klikovits

   University of Geneva, Switzerland, Jun 2019

   PhD Thesis

   DOI: 10.13097/archive-ouverte/unige:121355 Bib

   @phdthesis{klikovits_phdthesis_2019,
     author = {Klikovits, Stefan},
     title = {A Domain-Specific Language Approach To Hybrid CPS Modelling},
     year = {2019},
     month = jun,
     school = {University of Geneva, Switzerland},
     note = {PhD Thesis},
     url = {https://archive-ouverte.unige.ch/unige:121355},
     doi = {10.13097/archive-ouverte/unige:121355},
   }

2. Virtual Sport @ BBC – “Visualising Live Sport Events on the Web”

   Stefan Klikovits

   Sep 2013

   MSc Thesis

   Abs Bib

   There is a growing interest in the analysis of sports data. The reasons for these developments are various and range from improvement of training strategies to calculations for the sports betting industry. The BBC has television rights for a large number of sports events. For some events (e.g. certain football games) they not only receive the video and audio data, but also have access to a data stream, which is logging events on the pitch. This projects’ goal was to utilise the data and evaluate possibilities for its presentation on computers and mobile devices, and the feasibility of such an application’s implementation. In the course of this project a prototype called ”VirtualSport” was developed as a proof-of-concept. This dissertation will first show that the data quality is high enough for the creation of appealing data visualisations that can be used to augment the user’s live sports experience. An implementation will then provide an example for efficient data integration using data stream management principles. The dissertation further describes the usage of an event driven architecture to ensure server performance and HTML web sockets to implement “push”-technology for real-time updates of the clients using HTML5. To assure the simplified addition of new functionality, this thesis describes the implemented plugin architecture and proposes strategies for assuring the system’s stability and correctness. As the application will have more than ten thousand users for individual sports events (C10k problem), an important part of the prototype is the evaluation of performance and optimisation strategies. This dissertation introduces details of these problems and provides information to overcome these challenges. Further discussion on accessing other data sources in the form of server-side mashups and addition of external information using proxystyle mashup technology is provided to show how the software can include other data sources.

   @mastersthesis{klikovits:2013:msc,
     author = {Klikovits, Stefan},
     title = {Virtual Sport @ BBC -- ``Visualising Live Sport Events on the Web''},
     school = {University of Manchester, UK},
     year = {2013},
     type = {MSc Thesis},
     note = {MSc Thesis},
     month = sep,
     url = {https://studentnet.cs.manchester.ac.uk/resources/library/thesis_abstracts/MSc13/FullText/Klikovits-Stefan-fulltext.pdf},
   }

3. Guideline for the development of a GUI regression testing framework for SCADA-applications using the example of JCOP Framework used by the LHC and it’s experiments

   Stefan Klikovits

   Aug 2011

   BSc Thesis

   Bib

   @mastersthesis{klikovits:2011:bsc,
     author = {Klikovits, Stefan},
     title = {Guideline for the development of a GUI regression testing framework for SCADA-applications using the example of JCOP Framework used by the LHC and it’s experiments},
     school = {Vienna University of Technology, Austria},
     year = {2011},
     type = {BSc Thesis},
     note = {BSc Thesis},
     month = aug,
   }

Talks & Presentations

1. Pragmatic Reuse for DSML Development

   Stefan Klikovits, and Didier Buchs

   Mar 2024

   Journal First presentation at Modellierung 2024

   Abs Bib

   By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.

   @unpublished{Klikovits:Modellierung2024:JournalFirst,
     author = {Klikovits, Stefan and Buchs, Didier},
     title = {Pragmatic Reuse for DSML Development},
     month = mar,
     year = {2024},
     note = {Journal First presentation at Modellierung 2024},
     url = {https://bpt.hpi.uni-potsdam.de/modellierung2024/program/}
   }

2. Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty

   Thomas Laurent, Stefan Klikovits, Paolo Arcaini, Fuyuki Ishikawa, and Anthony Ventresque

   May 2023

   Journal First presentation at 45th International Conference on Software Engineering (ICSE 2023)

   Abs DOI: 10.1145/3550270 Bib

   Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this paper, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios; and that the cost of computing coverage can be managed with suitable heuristics.

   @unpublished{Laurent:ICSE:JournalFirst,
     author = {Laurent, Thomas and Klikovits, Stefan and Arcaini, Paolo and Ishikawa, Fuyuki and Ventresque, Anthony},
     title = {Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty},
     year = {2023},
     month = may,
     note = {Journal First presentation at 45th International Conference on Software Engineering (ICSE 2023)},
     doi = {10.1145/3550270},
   }

3. Python Frameworks for Algorithms and Data Structures

   Stefan Klikovits

   Jan 2023

   Invited Guest Lecture at IMC Fachhochschule Krems, Austria

   Bib Code

   @unpublished{Klikovits:IMCLecture:2023,
     author = {Klikovits, Stefan},
     title = {Python Frameworks for Algorithms and Data Structures},
     month = jan,
     year = {2023},
     note = {Invited Guest Lecture at IMC Fachhochschule Krems, Austria},
   }

4. Pragmatic Reuse for DSML Development

   Stefan Klikovits, and Didier Buchs

   Oct 2021

   Invited Journal First presentation at the 24th International Conference on Model Driven Engineering Languages and Systems (MODELS 21)

   Abs Bib

   By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.

   @unpublished{Klikovits:MODELS2021:JournalFirst,
     author = {Klikovits, Stefan and Buchs, Didier},
     title = {Pragmatic Reuse for DSML Development},
     month = oct,
     year = {2021},
     note = {Invited Journal First presentation at the 24th International Conference on Model Driven Engineering Languages and Systems (MODELS 21)},
   }

Other

1. Does Road Diversity Really Matter in Testing Automated Driving Systems? – A Registered Report

   Stefan Klikovits, Vincenzo Riccio, Ezequiel Castellano, Ahmet Cetinkaya, Alessio Gambi, and 1 more author

   Oct 2022

   Abs DOI: 10.48550/ARXIV.2209.05947 arXiv Bib Slides

   **Background/Context.** The use of automated driving systems (ADSs) in the real world requires rigorous testing to ensure safety. To increase trust, ADSs should be tested on a large set of diverse road scenarios. Literature suggests that if a vehicle is driven along a set of geometrically diverse roads-measured using various diversity measures (DMs)-it will react in a wide range of behaviours, thereby increasing the chances of observing failures (if any), or strengthening the confidence in its safety, if no failures are observed. To the best of our knowledge, however, this assumption has never been tested before, nor have road DMs been assessed for their properties. **Objective/Aim.** Our goal is to perform an exploratory study on 47 currently used and new, potentially promising road DMs. Specifically, our research questions look into the road DMs themselves, to analyse their properties (e.g. monotonicity, computation efficiency), and to test correlation between DMs. Furthermore, we look at the use of road DMs to investigate whether the assumption that diverse test suites of roads expose diverse driving behaviour holds. **Method.** Our empirical analysis relies on a state-of-the-art, open-source ADSs testing infrastructure and uses a data set containing over 97,000 individual road geometries and matching simulation data that were collected using two driving agents. By sampling random test suites of various sizes and measuring their roads’ geometric diversity, we study road DMs properties, the correlation between road DMs, and the correlation between road DMs and the observed behaviour.

   @misc{2209.05947,
     doi = {10.48550/ARXIV.2209.05947},
     url = {https://arxiv.org/abs/2209.05947},
     author = {Klikovits, Stefan and Riccio, Vincenzo and Castellano, Ezequiel and Cetinkaya, Ahmet and Gambi, Alessio and Arcaini, Paolo},
     keywords = {peerreviewed},
     title = {Does Road Diversity Really Matter in Testing Automated Driving Systems? -- A Registered Report},
     publisher = {arXiv},
     year = {2022},
   }

2. COST IC1404 WG1 Deliverable WG1.1: State-of-the-art on Current Formalisms used in Cyber-Physical Systems Development

   Stefan Klikovits, Rima Al-Ali, Moussa Amrani, Ankica Barisic, Fernando Barros, and 10 more authors

   Jan 2019

   DOI: 10.5281/zenodo.2533455 Bib

   @misc{klikovits_stefan_2019_2538711,
     author = {Klikovits, Stefan and Al-Ali, Rima and Amrani, Moussa and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Borde, Etienne and Buchs, Didier and Giese, Holger and Goulao, Miguel and Iacono, Mauro and Leon, Florin and Navarro, Eva and Pelliccione, Patrizio and Vanherpen, Ken},
     title = {{COST IC1404 WG1 Deliverable WG1.1:
           State-of-the-art on Current Formalisms used
           in Cyber-Physical Systems Development}},
     month = jan,
     year = {2019},
     doi = {10.5281/zenodo.2533455},
     url = {https://doi.org/10.5281/zenodo.2533455},
   }

3. COST IC1404 WG1 Deliverable WG1.2: Framework to Relate / Combine Modeling Languages and Techniques

   Rima Al-Ali, Moussa Amrani, Soumyadip Bandyopadhyay, Ankica Barisic, Fernando Barros, and 10 more authors

   Jan 2019

   DOI: 10.5281/zenodo.2527576 Bib

   @misc{al_ali_rima_2019_2527577,
     author = {Al-Ali, Rima and Amrani, Moussa and Bandyopadhyay, Soumyadip and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Erata, Ferhat and Giese, Holger and Iacono, Mauro and Klikovits, Stefan and Navarro, Eva and Pelliccione, Patrizio and Taveter, Kuldar and Tekinerdogan, Bedir and Vanherpen, Ken},
     title = {{COST IC1404 WG1 Deliverable WG1.2:
           Framework to Relate / Combine
           Modeling Languages and Techniques}},
     month = jan,
     year = {2019},
     doi = {10.5281/zenodo.2527576},
     url = {https://doi.org/10.5281/zenodo.2527576},
   }

4. CREST Formalization

   Stefan Klikovits, Alban Linard, and Didier Buchs

   Jun 2018

   Abs DOI: 10.5281/zenodo.1284561 Bib

   CREST is a novel modelling language for the definition of Continuous-time, REactive SysTems. This domain-specific language (DSL) targets small cyber-physical systems (CPS) such as home automation systems. While CREST is a graphical language and its systems can be visualised as CREST diagrams, the main form of use is as internal DSL for the Python general purpose programming language. Nevertheless, CREST systems are based on a formal structure and semantics. This report provides this formalisation and elaborates on the design choices that have been made.

   @techreport{Klikovits:CRESTFormalization,
     author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier},
     title = {{CREST} Formalization},
     institution = {Software Modeling and Verification Group, University of Geneva},
     doi = {10.5281/zenodo.1284561},
     year = {2018},
     month = jun,
   }