Publications
My research interests are quite broad, but typically revolve around the concepts of systems modelling, digital twins & cyber-physical systems engineering, search-based and evolutionary methods, and the testing of complex systems such as autonomous driving systems.
Please also check out my ORCiD, Google Scholar and DBLP pages, as well as my official university page.
Edited Volumes
- Software Engineering im Unterricht der Hochschulen, (SEUH 2024)Proceedings. Axel Schmolitzky, and Stefan Klikovits editors.Feb 2024
@proceedings{DBLP:conf/seuh/2024, editor = {Schmolitzky, Axel and Klikovits, Stefan}, title = {Software Engineering im Unterricht der Hochschulen, ({SEUH} 2024)}, series = {{LNI}}, volume = {{P-346}}, publisher = {Gesellschaft f{\"{u}}r Informatik e.V.}, year = {2024}, month = feb, url = {https://dl.gi.de/handle/20.500.12116/43785}, isbn = {978-3-88579-740-1}, }
Book Chapters
- CREST — a DSML for hybrid CPS modellingStefan Klikovits, and Didier BuchsIn Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021
Hybrid systems modelling remains a very popular topic within the modelling and simulation community. Its expressiveness allows for the definition of highly complex systems that merge discrete-state-based transitions systems with continuous value-evolutions for variables, so that cyber-physical systems can be modelled in all their intricacies. This expressive power, however, comes at a downside of complex models and undecidable verification problems even for small systems. In this chapter we present CREST, a novel modelling language for the definition of hybrid systems. CREST merges features from various formalisms and languages such as hybrid automata, dataflow programming and internal domain-specific modelling language (DSML) designs to create simple yet powerful language for the modelling of resource flows within small-scale CPS such as automated gardening applications and smart homes. The language provides an easy to learn graphical interface and is supported by a Python-based tool implementation that allows for the efficient modelling, simulation and verification of CPS models.
@incollection{Klikovits:2021:mpma4cps:ch9, title = {{CREST --- a DSML for hybrid CPS modelling}}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, publisher = {Academic Press}, chapter = {9}, pages = {221-242}, year = {2021}, isbn = {978-0-12-819105-7}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00014-3}, url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000143}, author = {Klikovits, Stefan and Buchs, Didier}, keywords = {peerreviewed} } - An ontology for multi-paradigm modellingHolger Giese, Dominique Blouin, Rima Al-Ali, Hana Mkaouar, Soumyadip Bandyopadhyay, and 4 more authorsIn Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021
This chapter presents the ontology for Multi-Paradigm Modelling (MPM) specified using the Web Ontology Language (OWL) as introduced in Chapter 2. A thorough state of the art on MPM’s core notions including multi-formalism and model management approaches, languages and tools is presented. In particular, model management approaches have been characterized according to their modularity and incremental execution properties as required to scale for the large complex CPSs we face today. Based on this state of the art, an outline of the MPM ontology is developed by introducing its main classes and properties. The validation of the ontology is presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2.
@incollection{Klikovits:2021:mpma4cps:ch4, title = {{An ontology for multi-paradigm modelling}}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, publisher = {Academic Press}, chapter = {4}, pages = {67-122}, year = {2021}, isbn = {978-0-12-819105-7}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00009-X}, url = {https://www.sciencedirect.com/science/article/pii/B978012819105700009X}, author = {Giese, Holger and Blouin, Dominique and Al-Ali, Rima and Mkaouar, Hana and Bandyopadhyay, Soumyadip and Iacono, Mauro and Amrani, Moussa and Klikovits, Stefan and Erata, Ferhat}, keywords = {peerreviewed} } - An integrated ontology for multi-paradigm modelling for cyber-physical systemsDominique Blouin, Rima Al-Ali, Holger Giese, Stefan Klikovits, Soumyadip Bandyopadhyay, and 2 more authorsIn Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, Feb 2021
This chapter presents the Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS) ontology. This ontology integrates the Shared, MPM and CPS ontologies respectively introduced in Chapters 2, 3 and 4. It includes cross-cutting notions such as viewpoints, model-based development processes and modelling paradigms that together relate the formalisms and workflows (and their paradigms) to the part of CPSs developed with these formalisms. A brief state of the art on these notions is first presented, on which the MPM4CPS ontology builds. An overview of the ontology is then developed by introducing its main classes and properties. The validation of the ontology is finally presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2. The chapter also discusses perspectives and future work on this integrated ontological framework, which can serve as a basis to develop model management solutions to relate and combine modelling languages and tools, in order to better develop cyber-physical systems with appropriate formalisms and workflows.
@incollection{Klikovits:2021:mpma4cps:ch5, title = {{An integrated ontology for multi-paradigm modelling for cyber-physical systems}}, author = {Blouin, Dominique and Al-Ali, Rima and Giese, Holger and Klikovits, Stefan and Bandyopadhyay, Soumyadip and Barišić, Ankica and Erata, Ferhat}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, chapter = {5}, pages = {123-145}, year = {2021}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00010-6}, url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000106}, isbn = {978-0-12-819105-7}, publisher = {Academic Press}, keywords = {peerreviewed} } - Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event SystemsDidier Buchs, Stefan Klikovits, and Alban LinardIn Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems, Feb 2020
The study of concurrent and parallel systems has been a challenging research domain within cyber-physical systems community. This chapter provides a pragmatic introduction to the creation and analysis of such system models using the popular Petri nets formalism. Petri nets is a formalism that convinces through its simplicity and applicability. We offer an overview of the most important Petri nets concepts, analysis techniques and model checking approaches. Finally, we show the use of so-called High-level Petri nets for the representation of complex data structures and functionality and present a novel research approach that allows the use of Petri nets inside Functional Mock-up Units and cyber-physical system models.
@incollection{Klikovits:2020:mpm4cps:foundationsbook, author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban}, title = {{Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event Systems}}, booktitle = {{Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems}}, editor = {Carreira, P. and Amaral, V. and Vangheluwe, H.}, publisher = {Springer}, doi = {10.1007/978-3-030-43946-0}, isbn = {978-3-030-43946-0}, year = {2020}, keywords = {peerreviewed} }
Journal Articles
- GeQuPI: Quantum Program Improvement with Multi-Objective Genetic ProgrammingFelix Gemeinhardt, Stefan Klikovits, and Manuel WimmerJournal of Systems and Software, Feb 2025
Processing quantum information poses novel challenges regarding the debugging of faulty quantum programs. Notably, the lack of accessible information on intermediate states during quantum processing, renders traditional debugging techniques infeasible. Moreover, even correct quantum programs might not be processable, as current quantum computers are limited in computation capacity. Thus, quantum program developers have to consider trade-offs between accuracy (i.e., probabilistically correct functionality) and computational cost of the proposed solutions. Manually finding sufficiently accurate and efficient solutions is a challenging task, even for quantum computing experts. To tackle these challenges, we propose a quantum program improvement framework for an automated generation of accurate and efficient solutions, coined Genetic Quantum Program Improver (GeQuPI). In particular, we focus on the tasks of debugging and optimization of quantum programs. Our framework uses techniques from quantum information theory and applies multi-objective genetic programming, which can be further hybridized with quantum-aware optimizers. To demonstrate the benefits of GeQuPI, it is applied to 47 quantum programs reused from literature and openly published libraries. The results show that our approach is capable of correcting faulty programs and optimize inefficient ones for the majority of the studied cases, showing average optimizations of 35% with respect to computational cost.
@article{GEMEINHARDT2025112223, title = {{GeQuPI}: Quantum Program Improvement with Multi-Objective Genetic Programming}, journal = {Journal of Systems and Software}, volume = {219}, pages = {112223}, year = {2025}, issn = {0164-1212}, doi = {https://doi.org/10.1016/j.jss.2024.112223}, url = {https://www.sciencedirect.com/science/article/pii/S016412122400267X}, author = {Gemeinhardt, Felix and Klikovits, Stefan and Wimmer, Manuel}, keywords = {Quantum computing, Evolutionary algorithms, Quantum software engineering, Quantum circuit optimization} } - Continuous Evolution of Digital Twins using the DarTwin NotationJoost Mertens, Stefan Klikovits, Francis Bordeleau, Joachim Denil, and Øystein HaugenSoftware and Systems Modeling, Nov 2024
Despite best efforts, various challenges remain in the creation and maintenance processes of digital twins (DTs). One of those primary challenges is the constant, continuous and omnipresent evolution of systems, their user’s needs and their environment, demanding the adaptation of the developed DT systems. DTs are developed for a specific purpose, which generally entails the monitoring, analysis, simulation or optimisation of a specific aspect of an actual system, referred to as the actual twin (AT). As such, when the twin system changes, that is either the AT itself changes, or the scope/purpose of a DT is modified, the DTs usually evolve in close synchronicity with the AT. As DTs are software systems, the best practices or methodologies for software evolution can be leveraged. This paper tackles the challenge of maintaining a (set of) DT(s) throughout the evolution of the user’s requirements and priorities and tries to understand how this evolution takes place. In doing so, we provide two contributions: (i) we develop DarTwin , a visual notation form that enables reasoning on a twin system, its purposes, properties and implementation, and (ii) we introduce a set of architectural transformations that describe the evolution of DT systems. The development of these transformations is driven and illustrated by the evolution and transformations of a family home’s DT, whose purpose is expanded, changed and re-prioritised throughout its ongoing lifecycle. Additionally, we evaluate the transformations on a laboratory-scale gantry crane’s DT.
@article{Mertens2024, author = {Mertens, Joost and Klikovits, Stefan and Bordeleau, Francis and Denil, Joachim and Haugen, {\O}ystein}, title = {Continuous Evolution of Digital Twins using the DarTwin Notation}, journal = {Software and Systems Modeling}, year = {2024}, month = nov, day = {11}, issn = {1619-1374}, doi = {10.1007/s10270-024-01216-7}, url = {https://doi.org/10.1007/s10270-024-01216-7}, keywords = {System evolution, System design, Composability, Digital twin, Smart home} } - A Method for Template-based Architecture Modeling and its Application to Digital TwinsDaniel Lehner, Jérôme Pfeiffer, Stefan Klikovits, Andreas Wortmann, and Manuel WimmerJournal of Object Technology, Jul 2024The 20th European Conference on Modelling Foundations and Applications (ECMFA 2024)
Digital Twins (DTs) have recently emerged to support domain experts in engineering and operating Cyber-Physical Systems (CPSs). As a result, software vendors started to create DT services offering advanced functionality to support the development and operation of DTs in the industry. However, the integration of services into a DT architecture is challenging. Services typically rely on specific software and modeling languages that are often not interoperable with other services. Hence, they have to be manually integrated which requires a significant, repetitive effort. Thus, currently, it is tedious to extend the DT’s underlying architectures with new services or exchange individual service implementations. In this paper, we propose a tool-supported method for architecture modeling and its application for digital twins. The presented method provides several steps to manage the complexity of current DT architectures. First, DT reference architectures are assembled by connecting DT templates, which provide an abstraction for a set of similar DT services. Second, dedicated DT modules are used to wrap existing services which provide concrete realizations of the DT templates. Third, a product-line-based generator supports the configuration of reference architectures into concrete architectures by selecting an appropriate set of modules for the used templates which are finally used in the derived integration solution. The transition from reference architecture modeling to the product-line-based configuration is supported by a dedicated model transformation. Our evaluation shows that the proposed DT templates enable the efficient modeling of different DT reference architectures and integration of new DT services into already existing architectures.
@article{JOT:issue_2024_03/article8, author = {Lehner, Daniel and Pfeiffer, Jérôme and Klikovits, Stefan and Wortmann, Andreas and Wimmer, Manuel}, title = {A Method for Template-based Architecture Modeling and its Application to Digital Twins}, journal = {Journal of Object Technology}, volume = {23}, number = {3}, issn = {1660-1769}, year = {2024}, month = jul, note = {The 20th European Conference on Modelling Foundations and Applications (ECMFA 2024)}, doi = {10.5381/jot.2024.23.3.a8}, url = {http://www.jot.fm/contents/issue_2024_03/article8.html}, } - Digital Transformation in Tunneling–A Project Report on TransITThomas Bednar, Robert Galler, Christian Huemer, Marco Huymajer, Stefan Klikovits, and 5 more authorsZeitschrift für Hochschulentwicklung, Jul 2024
The digital transformation has significantly affected many industries. Recently, it has gained momentum in the Architecture, Engineering, and Construction (AEC) sector in general and tunneling in particular. To support these developments, TransIT represents an inter-university and interdisciplinary research initiative with the goal of advancing digitalization in tunneling. This article discusses the challenges connected with the digital transformation of the tunneling sector. We examine this topic with different use cases demonstrating solutions to those challenges.
@article{bednar2024digital, title = {{Digital Transformation in Tunneling--A Project Report on TransIT}}, author = {Bednar, Thomas and Galler, Robert and Huemer, Christian and Huymajer, Marco and Klikovits, Stefan and Melnyk, Oleksandr and Paskaleva, Galina and Steiner, Bernhard and Wenighofer, Robert and Wimmer, Manuel}, journal = {Zeitschrift f{\"u}r Hochschulentwicklung}, volume = {19}, number = {Sonderheft Administration}, pages = {143--163}, year = {2024}, doi = {10.21240/zfhe/SH-A/09}, keywords = {peerreviewed} } - Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for ADS TestingStefan Klikovits, Ezequiel Castellano, Ahmet Cetinkaya, and Paolo ArcainiScience of Computer Programming, Jul 2023
Being capable of identifying significant safety shortcomings, search-based methods are a core tool for testing automated driving system (ADS) technologies. In this domain, Frenetic has proven to be a popular and very effective tool, searching and identifying diverse sets of roads that point out potentially faulty ADS behaviour. This paper presents Frenetic-lib, a Python library that captures Frenetic’s novel combination of road representation and genetic algorithm, and makes it generally available in a customisable way. Next to the capacity to integrate additional ADS simulators, Frenetic-lib further creates new research opportunities on search-based road testing, novel road representations and mutation operators.
@article{Klikovits:freneticlib:2023, title = {{Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for {ADS} Testing}}, journal = {Science of Computer Programming}, volume = {230}, pages = {102996}, date = {2023-07}, year = {2023}, month = jul, issn = {0167-6423}, doi = {10.1016/j.scico.2023.102996}, url = {https://www.sciencedirect.com/science/article/pii/S0167642323000783}, author = {Klikovits, Stefan and Castellano, Ezequiel and Cetinkaya, Ahmet and Arcaini, Paolo}, keywords = {peerreviewed}, } - Parameter Coverage for Testing of Autonomous Driving Systems under UncertaintyThomas Laurent, Stefan Klikovits, Paolo Arcaini, Fuyuki Ishikawa, and Anthony VentresqueACM Transactions on Software Engineering Methodology (TOSEM), Apr 2023
Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.
@article{10.1145/3550270, author = {Laurent, Thomas and Klikovits, Stefan and Arcaini, Paolo and Ishikawa, Fuyuki and Ventresque, Anthony}, title = {Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty}, year = {2023}, issue_date = {July 2023}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {32}, number = {3}, issn = {1049-331X}, url = {https://doi.org/10.1145/3550270}, doi = {10.1145/3550270}, journal = {ACM Transactions on Software Engineering Methodology (TOSEM)}, month = apr, articleno = {58}, numpages = {31}, keywords = {peerreviewed}, } - Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averagingStefan Klikovits, Cédric Ho Thanh, Ahmet Cetinkaya, and Paolo ArcainiApplied Soft Computing, Apr 2023
Multi-objective optimisation (MOO) is a popular approach for finding solutions to many types of complex problems with large search spaces and conflicting search objectives. In the past, MOO algorithms have been shown to reliably produce good optimisation results. With the rise of cyber–physical systems, however, emerges the new challenge of non-deterministic system executions, caused by e.g. imperfect sensor readings or synchronisation in multi-process/multi-agent system architectures. These systems produce varying output on each execution, causing the algorithms’ observations to be noisy. Naturally, MOO algorithms favour the fittest solutions, which may have been measured with great inaccuracy. The end results are therefore not trustworthy. In this paper, we propose kNN-averaging, a new method to address this issue by identifying the k-nearest neighbours (kNN) of a solution, and using their weighted average as an estimate for its true fitness. Our experiments demonstrate the viability of kNN-averaging on 40 synthetic benchmark problems and on a real-world case study system. In the process, we compare kNN-averaging to the noisy baseline as well as two resampling-based methods and one spectral sampling approach on a range of algorithm settings. The results show that kNN-averaging approximates the fitness of solutions more accurately than the noisy baseline, leading to more trustworthy results.
@article{Klikovits:2023:2023110631, title = {{Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averaging}}, journal = {Applied Soft Computing}, volume = {146}, pages = {110631}, date = {2023-10}, year = {2023}, issn = {1568-4946}, doi = {10.1016/j.asoc.2023.110631}, url = {https://www.sciencedirect.com/science/article/pii/S156849462300649X}, author = {Klikovits, Stefan and {Ho Thanh}, Cédric and Cetinkaya, Ahmet and Arcaini, Paolo}, keywords = {peerreviewed}, } - Pragmatic Reuse for DSML DevelopmentStefan Klikovits, and Didier BuchsSoftware and Systems Modeling (SoSyM), Apr 2021
By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.
@article{Klikovits:2020:sosym:crest, author = {Klikovits, Stefan and Buchs, Didier}, title = {{Pragmatic Reuse for DSML Development}}, journal = {{Software and Systems Modeling (SoSyM)}}, doi = {10.1007/s10270-020-00831-4}, year = {2021}, volume = {20}, pages = {837-866}, keywords = {peerreviewed} }
Conference Workshop
- Trust Your Neighbours: Handling Noise in Multi-Objective Optimisation Using kNN-Averaging (GECCO’24 Hot off the Press)Stefan Klikovits, Cédric Ho Thanh, Ahmet Cetinkaya, and Paolo ArcainiIn Proceedings of the Genetic and Evolutionary Computation Conference Companion, Melbourne, VIC, Australia, Jul 2024
The non-deterministic nature of modern systems such as cyber-physical systems (e.g. due to sensor noise) and multi-process/multi-agent systems (e.g. due to timing differences), poses a significant challenge in the field of multi-objective optimisation (MOO). Those systems may produce different objective values on every evaluation of the objective function, in which case the effectiveness of classical MOO algorithms can no longer be guaranteed. It has indeed been observed that they are prone to producing results that are either not optimal or not feasible.An obvious, yet naive, solution is to approximate the true fitness of a solution by sampling the objective function multiple times. However, this leads to significantly more evaluations of the objective function, which may not be acceptable, e.g. if the fitness function is expensive to compute.To tackle this issue, we propose kNN-averaging, an MOO algorithm that approximates the true fitness of solutions based on a k-nearest neighbours (kNN) regression scheme. We experimentally compare kNN-averaging to two resampling-based methods, a Gaussian process-based spectral sampling approach, and the default, uncorrected baseline, on 40 benchmark problems and one real-world case study.This is an extended abstract of the paper:S. Klikovits, C. Ho Thanh, A. Cetinkaya, and P. Arcaini. Trust your neighbours: Handling Noise in Multi-Objective Optimisation Using kNN-Averaging. Applied Soft Computing, Vol. 146, (2023). https://doi.org/10.1016/j.asoc.2023.110631
@inproceedings{10.1145/3638530.3664075, author = {Klikovits, Stefan and Ho Thanh, C\'{e}dric and Cetinkaya, Ahmet and Arcaini, Paolo}, title = {Trust Your Neighbours: Handling Noise in Multi-Objective Optimisation Using kNN-Averaging (GECCO'24 Hot off the Press)}, year = {2024}, month = jul, isbn = {9798400704956}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3638530.3664075}, doi = {10.1145/3638530.3664075}, booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion}, pages = {39–40}, numpages = {2}, keywords = {kNN-averaging, noisy system control, noisy fitness, search-based approaches, multi-objective optimization, cyber-physical systems}, location = {Melbourne, VIC, Australia}, series = {GECCO '24 Companion} } - Leveraging Software Product Lines for Testing Automated Driving SystemsStefan Klikovits, Alessio Gambi, Deepak Dhungana, and Rick RabiserIn 18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024), Bern, Switzerland, Feb 2024
Extensive testing of Automated Driving Systems (ADS), such as Advanced Driver Assistance Systems and Autonomous Vehicles, is commonly conducted using simulators programmed to implement various driving scenarios, a technique known as scenario-based testing. ADS scenario-based testing using simulations is challenging because it requires identifying scenarios that can effectively test ADS functionalities while ensuring that driving simulators’ features match the driving scenarios’ requirements. This short paper discusses the main challenges of systematically conducting simulation-based testing and proposes leveraging Software Product Line techniques to address them. Specifically, we argue that variability models can be used to support testers in generating test scenarios by effectively capturing and relating the variability in driving simulators, testing scenarios, and ADS implementations. We conclude by outlining an agenda for future research in this important area.
@inproceedings{Klikovits:2024:VAMOS, author = {Klikovits, Stefan and Gambi, Alessio and Dhungana, Deepak and Rabiser, Rick}, title = {{Leveraging Software Product Lines for Testing Automated Driving Systems}}, booktitle = {18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024)}, year = {2024}, month = feb, doi = {10.1145/3634713.3634720}, location = {Bern, Switzerland}, keywords = {peerreviewed} } - Model-Driven Optimization for Quantum Program Synthesis with MOMoTFelix Gemeinhardt, Martin Eisenberg, Stefan Klikovits, and Manuel WimmerIn 5th Workshop on Artificial Intelligence and Model-driven Engineering, Väster\aas, Sweden, Oct 2023
In the realm of classical software engineering, model-driven optimization has been widely used for different problems such as (re)modularization of software systems. In this paper, we investigate how techniques from model-driven optimization can be applied in the context of quantum software engineering. In quantum computing, creating executable quantum programs is a highly non-trivial task which requires significant expert knowledge in quantum information theory and linear algebra. Although different approaches for automated quantum program synthesis exist—e.g., based on reinforcement learning and genetic programming—these approaches represent tailor-made solutions requiring dedicated encodings for quantum programs. This paper applies the existing model-driven optimization approach MOMoT to the problem of quantum program synthesis. We present the resulting platform for experimenting with quantum program synthesis and present a concrete demonstration for a well-known quantum algorithm.
@inproceedings{Klikovits:2023:MDEIntelligence, author = {Gemeinhardt, Felix and Eisenberg, Martin and Klikovits, Stefan and Wimmer, Manuel}, title = {{Model-Driven Optimization for Quantum Program Synthesis with MOMoT}}, booktitle = {5th Workshop on Artificial Intelligence and Model-driven Engineering}, year = {2023}, month = oct, doi = {10.1109/MODELS-C59198.2023.00100}, location = {V\"{a}ster\aas, Sweden}, keywords = {peerreviewed} } - Towards Generating Model-Driven Speech Interfaces for Digital TwinsRamiya Jayaraman, Daniel Lehner, Stefan Klikovits, and Manuel WimmerIn 3rd International Workshop on Model-Driven Engineering of Digital Twins, Väster\aas, Sweden, Oct 2023
The recent decade saw enormous advances of the capabilities of speech synthesis and speech recognition. While specific benefits depend on the individual applications, speech interfaces typically increase accessibility, enable “hands-free” and “no-screen” interaction, and often increase interaction speed and allow for more flexible usage patterns and increased multitasking, leading to higher user satisfaction. This paper presents a method to transfer these powerful benefits to the digital twin (DT) domain by automatically generating speech interfaces for DT applications. Our approach is based on Model Driven Engineering principles, where we automatically deduce command patterns from structural model information as used in, e.g., DT platform specifications. The speech interface generation is highly configurable and extendable, thus it can be applied to different DT models. The concepts behind the generator are generic as well, thus they can be ported to other applications and platforms. We validate our approach by applying it to two DT demonstration cases and provide a detailed description of the sketch interface configuration workflow.
@inproceedings{Klikovits:2023:MODDIT, author = {Jayaraman, Ramiya and Lehner, Daniel and Klikovits, Stefan and Wimmer, Manuel}, title = {{Towards Generating Model-Driven Speech Interfaces for Digital Twins}}, booktitle = {3rd International Workshop on Model-Driven Engineering of Digital Twins}, year = {2023}, month = oct, doi = {10.1109/MODELS-C59198.2023.00080}, location = {V\"{a}ster\aas, Sweden}, keywords = {peerreviewed} } - Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator DiscoveryFelix Gemeinhardt, Stefan Klikovits, and Manuel WimmerIn Genetic and Evolutionary Computation Conference Companion (GECCO’23), Lisbon, Portugal, Jul 2023
The processing of quantum information is defined by quantum circuits. For applications on current quantum devices, these are usually parameterized, i.e., they contain operations with variable parameters. The design of such quantum circuits and aggregated higher-level quantum operators is a challenging task which requires significant knowledge in quantum information theory, provided a polynomial-sized solution can be found analytically at all. Moreover, finding an accurate solution with low computational cost represents a significant trade-off, particularly for the current generation of quantum computers. To tackle these challenges, we propose a multi-objective genetic programming approach—hybridized with a numerical parameter optimizer—to automate the synthesis of parameterized quantum operators. To demonstrate the benefits of the proposed approach, it is applied to a quantum circuit of a hybrid quantum-classical algorithm, and then compared to an analytical solution as well as a non-hybrid version. The results show that, compared to the non-hybrid version, our method produces more diverse solutions and more accurate quantum operators which even reach the quality of the analytical baseline.
@inproceedings{10.1145/3583133.3590696, author = {Gemeinhardt, Felix and Klikovits, Stefan and Wimmer, Manuel}, title = {Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator Discovery}, booktitle = {Genetic and Evolutionary Computation Conference Companion (GECCO'23)}, pages = {795–798}, numpages = {4}, date = {2023-07}, year = {2023}, month = jul, publisher = {ACM}, keywords = {peerreviewed}, doi = {10.1145/3583133.3590696}, location = {Lisbon, Portugal}, series = {GECCO '23 Companion}, } - SBFT Tool Competition 2023 - Cyber-Physical Systems TrackMatteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo RiccioIn 16th IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT 2023, Melbourne, Australia, May 14, 2023, May 2023
@inproceedings{SBFT-toolcomp23, author = {Biagiola, Matteo and Klikovits, Stefan and Peltom\"{a}ki, Jarkko and Riccio, Vincenzo}, title = {{SBFT} Tool Competition 2023 - Cyber-Physical Systems Track}, booktitle = {16th {IEEE/ACM} International Workshop on Search-Based and Fuzz Testing, {SBFT} 2023, Melbourne, Australia, May 14, 2023}, year = {2023}, month = may, pages = {45-48}, doi = {10.1109/SBFT59156.2023.00010}, } - FreneticV at the SBST 2022 Tool CompetitionEzequiel Castellano, Stefan Klikovits, Ahmet Cetinkaya, and Paolo ArcainiIn 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), May 2022
FreneticV is a search-based testing tool based on an evolutionary approach that generates roads where an automated driving agent possibly fails the lane-keeping task. It uses a curvature-based road representation and, compared to its predecessor Frenetic, considers the validity of the generated roads. In particular, it tries to avoid generating roads with overly sharp turns, detects self-intersecting roads, and can rotate and relocate roads to fit them in a given map.
@inproceedings{Castellano2022FreneticVAT, title = {{FreneticV} at the {SBST} 2022 Tool Competition}, author = {Castellano, Ezequiel and Klikovits, Stefan and Cetinkaya, Ahmet and Arcaini, Paolo}, booktitle = {2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)}, year = {2022}, month = may, pages = {47-48}, doi = {10.1145/3526072.3527532}, } - Dynamic Shielding for Reinforcement Learning in Black-Box EnvironmentsMasaki Waga, Ezequiel Castellano, Sasinee Pruekprasert, Stefan Klikovits, Toru Takisaka, and 1 more authorIn Automated Technology for Verification and Analysis (ATVA’22), May 2022
It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.
@inproceedings{10.1007/978-3-031-19992-9_2, author = {Waga, Masaki and Castellano, Ezequiel and Pruekprasert, Sasinee and Klikovits, Stefan and Takisaka, Toru and Hasuo, Ichiro}, editor = {Bouajjani, Ahmed and Hol{\'i}k, Luk{\'a}{\v{s}} and Wu, Zhilin}, title = {Dynamic Shielding for Reinforcement Learning in Black-Box Environments}, booktitle = {Automated Technology for Verification and Analysis (ATVA'22)}, year = {2022}, publisher = {Springer International Publishing}, address = {Cham}, pages = {25--41}, isbn = {978-3-031-19992-9}, doi = {10.1007/978-3-031-19992-9_2}, keywords = {peerreviewed} } - Handling Noise in Search-Based Scenario Generation for Autonomous Driving SystemsStefan Klikovits, and Paolo ArcainiIn 26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021), Nov 2021
This paper presents the first evaluation of k-nearest neighbours-Averaging (kNN-Avg) on a real-world case study. kNN-Avg is a novel technique that tackles the challenges of noisy multi-objective optimisation (MOO). Existing studies suggest the use of repetition to overcome noise. In contrast, kNN-Avg approximates these repetitions and exploits previous executions, thereby avoiding the cost of re-running. We use kNN-Avg for the scenario generation of a real-world autonomous driving system (ADS) and show that it is better than the noisy baseline. Furthermore, we compare it to the repetition-method and outline indicators as to which approach to choose in which situations.
@inproceedings{KlikovitsA2021PRDC, title = {Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems}, author = {Klikovits, Stefan and Arcaini, Paolo}, booktitle = {26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021)}, publisher = {IEEE}, year = {2021}, month = nov, doi = {10.1109/PRDC53464.2021.00011}, keywords = {peerreviewed} } - On the Need for Multi-Level ADS ScenariosStefan Klikovits, and Paolo ArcainiIn 3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS’21), Oct 2021
Currently, most existing approaches for the design of Automated Driving Systems (ADS) scenarios focus on the description at one particular abstraction level — typically the most detailed one. This practice often removes information at higher levels, such that this data has to be re-synthesized if needed. As the abstraction granularity should be adapted to the task at hand, however, engineers currently have the choice between re-calculating the needed data or operating on the wrong level of abstraction. For instance, the search in a scenario database for a driving scenario with a map of a given road-shape should abstract over the lane markings, adjacent vegetation, or weather situation. Often though, the general road shape has to be synthesized (e.g. interpolated) from the precise GPS information of road boundaries. This paper outlines our vision for multi-level ADS scenario models that facilitate scenario search, generation, and design. Our concept is based on the common modelling philosophy to interact with scenarios at the most appropriate abstraction level. We identify different abstraction levels of ADS scenarios and suggest a template abstraction hierarchy. Our vision enables seamless traversal to such a most suitable granularity level for any given scenario, search and modelling task. We envision that this approach to ADS scenario modelling will have a lasting impact on the way we store, search, design, and generate ADS scenarios, allowing for a more strategic verification of autonomous vehicles in the long run.
@inproceedings{KlikovitsA2021MPM4CPS, title = {On the Need for Multi-Level {ADS} Scenarios}, author = {Klikovits, Stefan and Arcaini, Paolo}, booktitle = {3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'21)}, series = {MODELS'21}, publisher = {ACM}, year = {2021}, month = oct, pages = {322-326}, doi = {10.1109/MODELS-C53483.2021.00054}, keywords = {peerreviewed} } - KNN-Averaging for Noisy Multi-Objective OptimisationStefan Klikovits, and Paolo ArcainiIn Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC), Faro, Portugal (Online), Sep 2021
Multi-objective optimisation is a popular approach for finding solutions to complex problems with large search spaces that reliably yields good optimisation results. However, with the rise of cyber-physical systems, emerges a new challenge of noisy fitness functions, whose objective value for a given configuration is non-deterministic, producing varying results on each execution. This leads to an optimisation process that is based on stochastically sampled information, ultimately favouring solutions with fitness values that have co-incidentally high outlier noise. In turn, the results are unfaithful due to their large discrepancies between sampled and expectable objective values. Motivated by our work on noisy automated driving systems, we present the results of our ongoing research to counteract the effect of noisy fitness functions without requiring repeated executions of each solution. Our method kNN-Avg identifies the k-nearest neighbours of a solution point and uses the weighted average value as a surrogate for its actually sampled fitness. We demonstrate the viability of kNN-Avg on common benchmark problems and show that it produces comparably good solutions whose fitness values are closer to the expected value.
@inproceedings{KlikovitsA2021knnAveraging, author = {Klikovits, Stefan and Arcaini, Paolo}, title = {{KNN-Averaging for Noisy Multi-Objective Optimisation}}, editor = {Paiva, Ana C. R. and Cavalli, Ana Rosa and Ventura Martins, Paula and P{\'e}rez-Castillo, Ricardo}, booktitle = {Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC)}, publisher = {Springer International Publishing}, pages = {503--518}, series = {Communications in Computer and Information Science (CCIS)}, volume = {1439}, isbn = {978-3-030-85347-1}, location = {Faro, Portugal (Online)}, doi = {10.1007/978-3-030-85347-1_36}, year = {2021}, month = sep, keywords = {peerreviewed} } - Architecture-Guided Test Resource Allocation via LogicClovis Eberhart, Akihisa Yamada, Stefan Klikovits, Shin-ya Katsumata, Tsutomu Kobayashi, and 2 more authorsIn Tests and Proofs (TAP 2021), Jun 2021
We introduce a new logic named Quantitative Confidence Logic (QCL) that quantifies the level of confidence one has in the conclusion of a proof. By translating a fault tree representing a system’s architecture to a proof, we show how to use QCL to give a solution to the test resource allocation problem that takes the given architecture into account. We implemented a tool called Astrahl and compared our results to other testing resource allocation strategies.
@inproceedings{Eberhart2021, author = {Eberhart, Clovis and Yamada, Akihisa and Klikovits, Stefan and Katsumata, Shin-ya and Kobayashi, Tsutomu and Hasuo, Ichiro and Ishikawa, Fuyuki}, title = {Architecture-Guided Test Resource Allocation via Logic}, booktitle = {Tests and Proofs (TAP 2021)}, series = {Lecture Notes in Computer Science (LNCS)}, volume = {12740}, doi = {10.1007/978-3-030-79379-1_2}, year = {2021}, month = jun, publisher = {Springer}, pages = {22--38}, keywords = {peerreviewed} } - Frenetic at the SBST 2021 Tool CompetitionEzequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, and 1 more authorIn 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), May 2021
Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimise the out of bound distance, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.
@inproceedings{Castellano:2021:SBST, author = {Castellano, Ezequiel and Cetinkaya, Ahmet and Thanh, Cédric Ho and Klikovits, Stefan and Zhang, Xiaoyi and Arcaini, Paolo}, title = {Frenetic at the {SBST} 2021 Tool Competition}, booktitle = {2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)}, year = {2021}, month = may, editor = {Zhang, Jie and Fredericks, Erik}, pages = {36-37}, publisher = {IEEE}, keywords = {genetic algorithms, genetic programming}, doi = {10.1109/SBST52555.2021.00016}, } - ML4CREST: Machine Learning for CPS ModelsStefan Klikovits, Aurélien Coet, and Didier BuchsIn 2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT), Oct 2018
Models of small CPS and IoT applications often use approximated values that describe physical system behaviour. Physical resources, such as electricity consumption and heating power, have to be estimated, since many off-the-shelf components lack the required descriptions. Controllers which are based on these approximations can hence use imprecise models, perform misleading simulation, and cause damaged systems and financial loss. In this paper we present ML4CREST, a machine learning approach to automatically calibrate models using sensor measurements. We show that our approach is well-suited for the calibration of the flow rates within an automated watering system, which allows precise simulation and prevents spillage.
@inproceedings{Klikovits:MDE4IOT:CREST, title = {{ML4CREST: Machine Learning for CPS Models}}, author = {Klikovits, Stefan and Coet, Aur\'{e}lien and Buchs, Didier}, year = {2018}, month = oct, pages = {515-520}, url = {http://ceur-ws.org/Vol-2245/mde4iot_paper_4.pdf}, volume = {2245}, series = {CEUR Workshop Proceedings}, booktitle = {2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT)}, keywords = {peerreviewed} } - CREST - A DSL for Reactive Cyber-Physical SystemsStefan Klikovits, Alban Linard, and Didier BuchsIn 10th System Analysis and Modeling Conference (SAM2018). Languages, Methods, and Tools for Systems Engineering, Oct 2018
This article presents CREST, a novel domain-specific language for the modelling of cyber-physical systems. CREST is designed for the simple and clear modelling, simulation and verification of small-scale systems such as home and office automation, smart gardening systems and similar. The language is designed to model the flow of resources throughout the system. It features synchronous system evolution and reactive behaviour. CREST’s formal semantics allow real-valued time advances and the modelling of timed system evolution. The continuous time concept permits the precise simulation of future system behaviour by automatically calculating next transition times. We present CREST in a practical manner, and elaborate on the Python-based DSL implementation and simulator.
@inproceedings{Klikovits:SAM18:CREST, title = {{{CREST}} - {{A DSL}} for {{Reactive Cyber}}-{{Physical Systems}}}, isbn = {978-3-030-01042-3}, booktitle = {10th System {{Analysis}} and {{Modeling}} {{Conference}} ({{SAM2018}}). {{Languages}}, {{Methods}}, and {{Tools}} for {{Systems Engineering}}}, publisher = {{Springer}}, series = {Lecture Notes in Computer Science}, volume = {11150}, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, editor = {Khendek, Ferhat and Gotzhein, Reinhard}, year = {2018}, month = oct, pages = {29-45}, doi = {10.1007/978-3-030-01042-3_3}, keywords = {peerreviewed} } - Petri Sport: A Sport for Petri NettersStefan Klikovits, Alban Linard, Dimitri Racordon, and Didier BuchsIn Petri Nets and Software Engineering. International Workshop, PNSE’18, Sep 2018
Petri nets are a family of formalisms dedicated to the representation of concurrent systems. Their strength is the compact modeling of complex behaviors using very simple rules. Despite this simplicity, many teachers observe that students often require a lot of exposure and numerous exercises to truly understand the semantics of Petri nets. In order to speed up this learning process and provide a different attack angle, we propose Petri sport, a fun game based upon the Petri net formalism. In Petri sport, players aim to gather points by moving across a Petri net-shaped playing field and "firing" transitions. A clock-based play style supports a structured game advance while at the same time it encourages players to move fast. As the playing field is shaped like a Petri net, it is possible to challenge a player’s movement speed, intellectual capabilities, as well as team coordination and communication. The difficulty level of Petri sport is based on the choice of playing field. This allows for adaptation in order to best fit the competitors’ age, experience and/or physical fitness level.
@inproceedings{Klikovits:PNSE18:PetriSport, title = {{Petri Sport: A Sport for Petri Netters}}, pages = {35--56}, author = {Klikovits, Stefan and Linard, Alban and Racordon, Dimitri and Buchs, Didier}, year = {2018}, month = sep, booktitle = {Petri Nets and Software Engineering. International Workshop, PNSE'18}, editor = {Moldt, Daniel and Kindler, Ekkart and R{\"o}lke, Heiko}, series = {CEUR Workshop Proceedings}, volume = {2138}, url = {http://ceur-ws.org/Vol-2138/paper2.pdf}, keywords = {peerreviewed} } - A Model Checker Collection for the Model Checking Contest Using Docker and Machine LearningDidier Buchs, Stefan Klikovits, Alban Linard, Romain Mencattini, and Dimitri RacordonIn Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, PETRI NETS 2018, Sep 2018
This paper introduces mcc4mcc, the Model Checker Collection for the Model Checking Contest, a tool that wraps multiple model checking solutions, and applies the most appropriate one based on the characteristics of the model it is given. It leverages machine learning algorithms to carry out this selection, based on the results gathered from the 2017 edition of the Model Checking Contest, an annual event in which multiple tools compete to verify different properties on a large variety of models. Our approach brings two important contributions. First, our tool offers the opportunity to further investigate on the relation between model characteristics and verification techniques. Second, it lays out the groundwork for a unified way to distribute model checking software using virtual containers.
@inproceedings{DBLP:conf/apn/BuchsKLMR18, author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban and Mencattini, Romain and Racordon, Dimitri}, title = {{A Model Checker Collection for the Model Checking Contest Using Docker and Machine Learning}}, booktitle = {Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, {PETRI} {NETS} 2018}, pages = {385--395}, year = {2018}, month = sep, series = {Lecture Notes in Computer Science}, volume = {10877}, publisher = {Springer}, doi = {10.1007/978-3-319-91268-4_21}, url = {https://doi.org/10.1007/978-3-319-91268-4_21}, keywords = {peerreviewed} } - CREST - A Continuous, REactive SysTems DSLStefan Klikovits, Alban Linard, and Didier BuchsIn 5th International Workshop on the Globalization of Modeling Languages (GEMOC 2017), 19 September 2017, Austin, TX, USA, Sep 2017
The advance of cyber-physical systems in everyday life requires powerful modeling capabilities. Existing formalisms often have severe limitations and require complicated notations. In this paper we introduce CREST, a domain-specific language for modeling entity behavior and resource transfers in CPS. CREST aims to support CPS architects through clarity, comprehensiveness and analyzability.
@inproceedings{DBLP:conf/models/KlikovitsLB17, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, title = {{CREST} - {A} Continuous, REactive SysTems {DSL}}, booktitle = {5th International Workshop on the {Globalization of Modeling Languages} {(GEMOC} 2017), 19 September 2017, Austin, TX, USA}, pages = {286--291}, year = {2017}, month = sep, url = {http://ceur-ws.org/Vol-2019/gemoc_2.pdf}, volume = {2019}, series = {CEUR Workshop Proceedings}, keywords = {peerreviewed} } - Modeling FramesStefan Klikovits, Joachim Denil, Alexandre Muzy, and Rick SalayIn 14th Workshop on Model-Driven Engineering, Verification and Validation (MoDeVVa 2017), 19 September 2017, Austin, TX, USA, Sep 2017
Modeling activities such as calibration, verification and validation are often executed in under-specified environments. This hinders reproducibility, reduces re-usability and generally decreases the modeling precision and quality. This paper describes a framework for the definition of model frames. Model frames capture the context an activity/model is executed in and facilitate re-use, replacement, validation and verification of models. We show the application of the frames approach onto a real-world example, introduce several kinds of frames and show their application on this case study.
@inproceedings{DBLP:conf/models/KlikovitsDMS17, author = {Klikovits, Stefan and Denil, Joachim and Muzy, Alexandre and Salay, Rick}, title = {Modeling Frames}, booktitle = {14th Workshop on {Model-Driven Engineering, Verification and Validation} {(MoDeVVa} 2017), 19 September 2017, Austin, TX, USA}, volume = {2019}, series = {CEUR Workshop Proceedings}, pages = {315--320}, year = {2017}, month = sep, keywords = {peerreviewed} } - The Experiment Model and Validity Frame in M&SJoachim Denil, Stefan Klikovits, Pieter J. Mosterman, Antonio Vallecillo, and Hans VangheluweIn Proceedings of the Symposium on Theory of Modeling & Simulation, Apr 2017
Modelling and Simulation approaches use system models to conduct simulation experiments. Experimental frames have been applied in this context to formally define a system’s context. During the creation of an experimental frame for a simple spring model it becomes clear that experimental frames in their current definition lack certain properties and omit relevant information. Our approach describes the process of capturing the context of models and systems to provide truly reproducible experiment descriptions. The information, captured as experimental setups, can then be used for different purposes and in different scenarios, in particular for checking the validity of a model, the discovery of suitable models for the design of a system, and for calibrating models.
@inproceedings{denil:2017:tmsdevs, author = {Denil, Joachim and Klikovits, Stefan and Mosterman, Pieter J. and Vallecillo, Antonio and Vangheluwe, Hans}, title = {The Experiment Model and Validity Frame in M\&S}, year = {2017}, month = apr, booktitle = {Proceedings of the Symposium on Theory of Modeling \& Simulation}, series = {TMS/DEVS '17}, pages = {109--120}, isbn = {ISBN: 978-1-5108-3823-9}, venue = {Virginia Beach, Virginia, USA}, url = {http://scs.org/wp-content/uploads/2017/06/27_Final_Manuscript.pdf}, keywords = {peerreviewed} } - Towards Language Independent (Dynamic) Symbolic ExecutionStefan Klikovits, Manuel Gonzalez-Berges, and Didier BuchsIn Proceedings of the 24th PhD Mini-Symposium, Jan 2017
Symbolic execution is well-known for its capability to produce high-coverage test suites for software source code. So far, most tools are created to support a specific language. This paper elaborates on performing language independent symbolic execution and three ways to achieve it. We describe the use of one approach to perform dynamic symbolic execution using translation of a proprietary language and show the results of the tool execution on a real-life codebase.
@inproceedings{klikovits:2017:minisy, author = {Klikovits, Stefan and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Towards Language Independent (Dynamic) Symbolic Execution}, year = {2017}, month = jan, booktitle = {Proceedings of the 24th PhD Mini-Symposium}, pages = {50--53}, isbn = {978-963-313-243-2}, venue = {Budapest, Hungary}, url = {https://doi.org/10.5281/zenodo.291899}, publisher = {{Budapest University of Technology and Economics, Department of Measurement and Information Systems}}, doi = {10.5281/zenodo.291899}, keywords = {peerreviewed} } - Automated Test Case Generation for the CTRL Programming Language Using Pex: Lessons LearnedStefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and Didier BuchsIn 8th International Workshop on Software Engineering for Resilient Systems (SERENE), Sep 2016
Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. In this paper we first present ITEC, a tool for automated test case generation in CTRL, as well as initial results of test cases executions on one of CERN’s SCADA frameworks. Our tool relies on Microsoft’s Pex for its code exploration. For the purpose of using this existing test generation tool, we have to translate the proprietary CTRL code into C#, one of Pex’s operating languages. Our main contribution lies in detailing a formal foundation for this step through source code decomposition and anonymization. We then propose a quality measure that is used to determine our confidence into the translation and the generated test cases.
@inproceedings{DBLP:conf/serene/KlikovitsLGB16, author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Automated Test Case Generation for the {CTRL} Programming Language Using Pex: Lessons Learned}, booktitle = {8th International Workshop on Software Engineering for Resilient Systems (SERENE)}, pages = {117--132}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, volume = {9823}, year = {2016}, month = sep, url = {https://doi.org/10.1007/978-3-319-45892-2_9}, doi = {10.1007/978-3-319-45892-2_9}, keywords = {peerreviewed} } - Considering Execution Environment Resilience: A White-Box ApproachStefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and Didier BuchsIn 7th International Workshop on Software Engineering for Resilient Systems (SERENE), Sep 2015
Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. Only few of them consider dependencies from outside the Code Under Test’s scope such as global variables, database values and subroutine calls as influences to the execution path. In order to fully test all possible scenarios these dependencies have to be taken into account for the test input generation. This paper introduces ITEC, a tool for automated test case generation to support execution environment resilience in large-scaled, complex systems. One of ITEC’s corner stones is a technique called semi-purification, a source code transformation technique to overcome limitations of existing tools and to set up the required system state for software testing.
@inproceedings{DBLP:conf/serene/KlikovitsLGB15, author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Considering Execution Environment Resilience: {A} White-Box Approach}, booktitle = {7th International Workshop on Software Engineering for Resilient Systems (SERENE)}, pages = {46--61}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, volume = {9274}, month = sep, year = {2015}, url = {https://doi.org/10.1007/978-3-319-23129-7_4}, doi = {10.1007/978-3-319-23129-7_4}, keywords = {peerreviewed} } - Multi-platform SCADA GUI Regression Testing at CERNPaul C. Burkimsher, Manuel Gonzalez-Berges, and Stefan KlikovitsIn Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS), Sep 2011
The JCOP Framework is a toolkit used widely at CERN for the development of industrial control systems in several domains (i.e. experiments, accelerators and technical infrastructure). The software development started 10 years ago and there is now a large base of production systems running it. For the success of the project, it was essential to formalize and automate the quality assurance process. The paper will present the overall testing strategy and will describe in detail mechanisms used for GUI testing. The choice of a commercial tool (Squish) and the architectural features making it appropriate for our multi-platform environment will be described. Practical difficulties encountered when using the tool in the CERN context are discussed as well as how these were addressed. In the light of initial experience, the test code itself has been recently reworked in OO style to facilitate future maintenance and extension. The paper concludes with a description of our initial steps towards incorporation of full-blown Continuous Integration (CI) support.
@inproceedings{burkimsher2011icalepcs, author = {Burkimsher, Paul C. and Gonzalez{-}Berges, Manuel and Klikovits, Stefan}, title = {Multi-platform {SCADA} {GUI} Regression Testing at {CERN}}, booktitle = {Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS)}, year = {2011}, venue = {Grenoble, France}, publisher = {JACoW}, url = {https://accelconf.web.cern.ch/icalepcs2011/papers/thbhmust01.pdf}, keywords = {peerreviewed} }
Theses
- A Domain-Specific Language Approach To Hybrid CPS ModellingStefan KlikovitsUniversity of Geneva, Switzerland, Jun 2019PhD Thesis
@phdthesis{klikovits_phdthesis_2019, author = {Klikovits, Stefan}, title = {A Domain-Specific Language Approach To Hybrid CPS Modelling}, year = {2019}, month = jun, school = {University of Geneva, Switzerland}, note = {PhD Thesis}, url = {https://archive-ouverte.unige.ch/unige:121355}, doi = {10.13097/archive-ouverte/unige:121355}, } - Virtual Sport @ BBC – “Visualising Live Sport Events on the Web”Stefan KlikovitsSep 2013MSc Thesis
There is a growing interest in the analysis of sports data. The reasons for these developments are various and range from improvement of training strategies to calculations for the sports betting industry. The BBC has television rights for a large number of sports events. For some events (e.g. certain football games) they not only receive the video and audio data, but also have access to a data stream, which is logging events on the pitch. This projects’ goal was to utilise the data and evaluate possibilities for its presentation on computers and mobile devices, and the feasibility of such an application’s implementation. In the course of this project a prototype called ”VirtualSport” was developed as a proof-of-concept. This dissertation will first show that the data quality is high enough for the creation of appealing data visualisations that can be used to augment the user’s live sports experience. An implementation will then provide an example for efficient data integration using data stream management principles. The dissertation further describes the usage of an event driven architecture to ensure server performance and HTML web sockets to implement “push”-technology for real-time updates of the clients using HTML5. To assure the simplified addition of new functionality, this thesis describes the implemented plugin architecture and proposes strategies for assuring the system’s stability and correctness. As the application will have more than ten thousand users for individual sports events (C10k problem), an important part of the prototype is the evaluation of performance and optimisation strategies. This dissertation introduces details of these problems and provides information to overcome these challenges. Further discussion on accessing other data sources in the form of server-side mashups and addition of external information using proxystyle mashup technology is provided to show how the software can include other data sources.
@mastersthesis{klikovits:2013:msc, author = {Klikovits, Stefan}, title = {Virtual Sport @ BBC -- ``Visualising Live Sport Events on the Web''}, school = {University of Manchester, UK}, year = {2013}, type = {MSc Thesis}, note = {MSc Thesis}, month = sep, url = {https://studentnet.cs.manchester.ac.uk/resources/library/thesis_abstracts/MSc13/FullText/Klikovits-Stefan-fulltext.pdf}, } - Guideline for the development of a GUI regression testing framework for SCADA-applications using the example of JCOP Framework used by the LHC and it’s experimentsStefan KlikovitsAug 2011BSc Thesis
@mastersthesis{klikovits:2011:bsc, author = {Klikovits, Stefan}, title = {Guideline for the development of a GUI regression testing framework for SCADA-applications using the example of JCOP Framework used by the LHC and it’s experiments}, school = {Vienna University of Technology, Austria}, year = {2011}, type = {BSc Thesis}, note = {BSc Thesis}, month = aug, }
Talks & Presentations
- Pragmatic Reuse for DSML DevelopmentStefan Klikovits, and Didier BuchsMar 2024Journal First presentation at Modellierung 2024
By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.
@unpublished{Klikovits:Modellierung2024:JournalFirst, author = {Klikovits, Stefan and Buchs, Didier}, title = {Pragmatic Reuse for DSML Development}, month = mar, year = {2024}, note = {Journal First presentation at Modellierung 2024}, url = {https://bpt.hpi.uni-potsdam.de/modellierung2024/program/} } - Parameter Coverage for Testing of Autonomous Driving Systems under UncertaintyThomas Laurent, Stefan Klikovits, Paolo Arcaini, Fuyuki Ishikawa, and Anthony VentresqueMay 2023Journal First presentation at 45th International Conference on Software Engineering (ICSE 2023)
Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this paper, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios; and that the cost of computing coverage can be managed with suitable heuristics.
@unpublished{Laurent:ICSE:JournalFirst, author = {Laurent, Thomas and Klikovits, Stefan and Arcaini, Paolo and Ishikawa, Fuyuki and Ventresque, Anthony}, title = {Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty}, year = {2023}, month = may, note = {Journal First presentation at 45th International Conference on Software Engineering (ICSE 2023)}, doi = {10.1145/3550270}, } - Python Frameworks for Algorithms and Data StructuresStefan KlikovitsJan 2023Invited Guest Lecture at IMC Fachhochschule Krems, Austria
@unpublished{Klikovits:IMCLecture:2023, author = {Klikovits, Stefan}, title = {Python Frameworks for Algorithms and Data Structures}, month = jan, year = {2023}, note = {Invited Guest Lecture at IMC Fachhochschule Krems, Austria}, } - Pragmatic Reuse for DSML DevelopmentStefan Klikovits, and Didier BuchsOct 2021Invited Journal First presentation at the 24th International Conference on Model Driven Engineering Languages and Systems (MODELS 21)
By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.
@unpublished{Klikovits:MODELS2021:JournalFirst, author = {Klikovits, Stefan and Buchs, Didier}, title = {Pragmatic Reuse for DSML Development}, month = oct, year = {2021}, note = {Invited Journal First presentation at the 24th International Conference on Model Driven Engineering Languages and Systems (MODELS 21)}, }
Other
- Does Road Diversity Really Matter in Testing Automated Driving Systems? – A Registered ReportStefan Klikovits, Vincenzo Riccio, Ezequiel Castellano, Ahmet Cetinkaya, Alessio Gambi, and 1 more authorOct 2022
**Background/Context.** The use of automated driving systems (ADSs) in the real world requires rigorous testing to ensure safety. To increase trust, ADSs should be tested on a large set of diverse road scenarios. Literature suggests that if a vehicle is driven along a set of geometrically diverse roads-measured using various diversity measures (DMs)-it will react in a wide range of behaviours, thereby increasing the chances of observing failures (if any), or strengthening the confidence in its safety, if no failures are observed. To the best of our knowledge, however, this assumption has never been tested before, nor have road DMs been assessed for their properties. **Objective/Aim.** Our goal is to perform an exploratory study on 47 currently used and new, potentially promising road DMs. Specifically, our research questions look into the road DMs themselves, to analyse their properties (e.g. monotonicity, computation efficiency), and to test correlation between DMs. Furthermore, we look at the use of road DMs to investigate whether the assumption that diverse test suites of roads expose diverse driving behaviour holds. **Method.** Our empirical analysis relies on a state-of-the-art, open-source ADSs testing infrastructure and uses a data set containing over 97,000 individual road geometries and matching simulation data that were collected using two driving agents. By sampling random test suites of various sizes and measuring their roads’ geometric diversity, we study road DMs properties, the correlation between road DMs, and the correlation between road DMs and the observed behaviour.
@misc{2209.05947, doi = {10.48550/ARXIV.2209.05947}, url = {https://arxiv.org/abs/2209.05947}, author = {Klikovits, Stefan and Riccio, Vincenzo and Castellano, Ezequiel and Cetinkaya, Ahmet and Gambi, Alessio and Arcaini, Paolo}, keywords = {peerreviewed}, title = {Does Road Diversity Really Matter in Testing Automated Driving Systems? -- A Registered Report}, publisher = {arXiv}, year = {2022}, } - COST IC1404 WG1 Deliverable WG1.1: State-of-the-art on Current Formalisms used in Cyber-Physical Systems DevelopmentStefan Klikovits, Rima Al-Ali, Moussa Amrani, Ankica Barisic, Fernando Barros, and 10 more authorsJan 2019
@misc{klikovits_stefan_2019_2538711, author = {Klikovits, Stefan and Al-Ali, Rima and Amrani, Moussa and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Borde, Etienne and Buchs, Didier and Giese, Holger and Goulao, Miguel and Iacono, Mauro and Leon, Florin and Navarro, Eva and Pelliccione, Patrizio and Vanherpen, Ken}, title = {{COST IC1404 WG1 Deliverable WG1.1: State-of-the-art on Current Formalisms used in Cyber-Physical Systems Development}}, month = jan, year = {2019}, doi = {10.5281/zenodo.2533455}, url = {https://doi.org/10.5281/zenodo.2533455}, } - COST IC1404 WG1 Deliverable WG1.2: Framework to Relate / Combine Modeling Languages and TechniquesRima Al-Ali, Moussa Amrani, Soumyadip Bandyopadhyay, Ankica Barisic, Fernando Barros, and 10 more authorsJan 2019
@misc{al_ali_rima_2019_2527577, author = {Al-Ali, Rima and Amrani, Moussa and Bandyopadhyay, Soumyadip and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Erata, Ferhat and Giese, Holger and Iacono, Mauro and Klikovits, Stefan and Navarro, Eva and Pelliccione, Patrizio and Taveter, Kuldar and Tekinerdogan, Bedir and Vanherpen, Ken}, title = {{COST IC1404 WG1 Deliverable WG1.2: Framework to Relate / Combine Modeling Languages and Techniques}}, month = jan, year = {2019}, doi = {10.5281/zenodo.2527576}, url = {https://doi.org/10.5281/zenodo.2527576}, } - CREST FormalizationStefan Klikovits, Alban Linard, and Didier BuchsJun 2018
CREST is a novel modelling language for the definition of Continuous-time, REactive SysTems. This domain-specific language (DSL) targets small cyber-physical systems (CPS) such as home automation systems. While CREST is a graphical language and its systems can be visualised as CREST diagrams, the main form of use is as internal DSL for the Python general purpose programming language. Nevertheless, CREST systems are based on a formal structure and semantics. This report provides this formalisation and elaborates on the design choices that have been made.
@techreport{Klikovits:CRESTFormalization, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, title = {{CREST} Formalization}, institution = {Software Modeling and Verification Group, University of Geneva}, doi = {10.5281/zenodo.1284561}, year = {2018}, month = jun, }