Publications | Stefan Klikovits

My research interests are quite broad, but typically revolve around the concepts of systems modelling, digital twins & cyber-physical systems engineering, search-based and evolutionary methods, and the testing of complex systems such as autonomous driving systems.

Please also check out my ORCiD, Google Scholar and DBLP pages, as well as my official university page.

Group by type

2024

Leveraging Software Product Lines for Testing Automated Driving Systems

Deepak Dhungana Stefan Klikovits, and Rick Rabiser

In 18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024), 2024

Abs DOI: 10.1145/3634713.3634720 Bib PDF

Extensive testing of Automated Driving Systems (ADS), such as Advanced Driver Assistance Systems and Autonomous Vehicles, is commonly conducted using simulators programmed to implement various driving scenarios, a technique known as scenario-based testing. ADS scenario-based testing using simulations is challenging because it requires identifying scenarios that can effectively test ADS functionalities while ensuring that driving simulators’ features match the driving scenarios’ requirements. This short paper discusses the main challenges of systematically conducting simulation-based testing and proposes leveraging Software Product Line techniques to address them. Specifically, we argue that variability models can be used to support testers in generating test scenarios by effectively capturing and relating the variability in driving simulators, testing scenarios, and ADS implementations. We conclude by outlining an agenda for future research in this important area.
@inproceedings{Klikovits:2024:VAMOS, author = {Stefan Klikovits, Alessio Gambi, Deepak Dhungana and Rabiser, Rick}, title = {{Leveraging Software Product Lines for Testing Automated Driving Systems}}, booktitle = {18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024)}, year = {2024}, month = feb, doi = {10.1145/3634713.3634720}, location = {Bern, Switzerland}, }

2023

Model-Driven Optimization for Quantum Program Synthesis with MOMoT

Felix Gemeinhardt, Martin Eisenberg, Stefan Klikovits, and 1 more author

In 5th Workshop on Artificial Intelligence and Model-driven Engineering, 2023

Abs Bib PDF

In the realm of classical software engineering, model-driven optimization has been widely used for different problems such as (re)modularization of software systems. In this paper, we investigate how techniques from model-driven optimization can be applied in the context of quantum software engineering. In quantum computing, creating executable quantum programs is a highly non-trivial task which requires significant expert knowledge in quantum information theory and linear algebra. Although different approaches for automated quantum program synthesis exist—e.g., based on reinforcement learning and genetic programming—these approaches represent tailor-made solutions requiring dedicated encodings for quantum programs. This paper applies the existing model-driven optimization approach MOMoT to the problem of quantum program synthesis. We present the resulting platform for experimenting with quantum program synthesis and present a concrete demonstration for a well-known quantum algorithm.
@inproceedings{Klikovits:2023:MDEIntelligence, author = {Gemeinhardt, Felix and Eisenberg, Martin and Klikovits, Stefan and Wimmer, Manuel}, title = {{Model-Driven Optimization for Quantum Program Synthesis with MOMoT}}, booktitle = {5th Workshop on Artificial Intelligence and Model-driven Engineering}, year = {2023}, month = oct, location = {V\"{a}ster\aas, Sweden}, }
Towards Generating Model-Driven Speech Interfaces for Digital Twins

Ramiya Jayaraman, Daniel Lehner, Stefan Klikovits, and 1 more author

In 3rd International Workshop on Model-Driven Engineering of Digital Twins, 2023

Abs Bib PDF

The recent decade saw enormous advances of the capabilities of speech synthesis and speech recognition. While specific benefits depend on the individual applications, speech interfaces typically increase accessibility, enable “hands-free” and “no-screen” interaction, and often increase interaction speed and allow for more flexible usage patterns and increased multitasking, leading to higher user satisfaction. This paper presents a method to transfer these powerful benefits to the digital twin (DT) domain by automatically generating speech interfaces for DT applications. Our approach is based on Model Driven Engineering principles, where we automatically deduce command patterns from structural model information as used in, e.g., DT platform specifications. The speech interface generation is highly configurable and extendable, thus it can be applied to different DT models. The concepts behind the generator are generic as well, thus they can be ported to other applications and platforms. We validate our approach by applying it to two DT demonstration cases and provide a detailed description of the sketch interface configuration workflow.
@inproceedings{Klikovits:2023:MODDIT, author = {Jayaraman, Ramiya and Lehner, Daniel and Klikovits, Stefan and Wimmer, Manuel}, title = {{Towards Generating Model-Driven Speech Interfaces for Digital Twins}}, booktitle = {3rd International Workshop on Model-Driven Engineering of Digital Twins}, year = {2023}, month = oct, location = {V\"{a}ster\aas, Sweden}, }
Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for ADS Testing

Stefan Klikovits, Ezequiel Castellano, Ahmet Cetinkaya, and 1 more author

Science of Computer Programming,Vol. 230 2023

Abs DOI: 10.1016/j.scico.2023.102996 Bib

Being capable of identifying significant safety shortcomings, search-based methods are a core tool for testing automated driving system (ADS) technologies. In this domain, Frenetic has proven to be a popular and very effective tool, searching and identifying diverse sets of roads that point out potentially faulty ADS behaviour. This paper presents Frenetic-lib, a Python library that captures Frenetic’s novel combination of road representation and genetic algorithm, and makes it generally available in a customisable way. Next to the capacity to integrate additional ADS simulators, Frenetic-lib further creates new research opportunities on search-based road testing, novel road representations and mutation operators.
@article{Klikovits:freneticlib:2023, title = {Frenetic-lib: An Extensible Framework for Search-Based Generation of Road Structures for ADS Testing}, journal = {Science of Computer Programming}, volume = {230}, pages = {102996}, year = {2023}, month = jul, issn = {0167-6423}, doi = {10.1016/j.scico.2023.102996}, url = {https://www.sciencedirect.com/science/article/pii/S0167642323000783}, author = {Klikovits, Stefan and Castellano, Ezequiel and Cetinkaya, Ahmet and Arcaini, Paolo}, keywords = {Frenetic, search-based testing, road representation, genetic algorithm, simulation-based testing}, }
Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator Discovery

Felix Gemeinhardt, Stefan Klikovits, and Manuel Wimmer

In Genetic and Evolutionary Computation Conference Companion (GECCO’23), 2023

Abs DOI: 10.1145/3583133.3590696 Bib

The processing of quantum information is defined by quantum circuits. For applications on current quantum devices, these are usually parameterized, i.e., they contain operations with variable parameters. The design of such quantum circuits and aggregated higher-level quantum operators is a challenging task which requires significant knowledge in quantum information theory, provided a polynomial-sized solution can be found analytically at all. Moreover, finding an accurate solution with low computational cost represents a significant trade-off, particularly for the current generation of quantum computers. To tackle these challenges, we propose a multi-objective genetic programming approach—hybridized with a numerical parameter optimizer—to automate the synthesis of parameterized quantum operators. To demonstrate the benefits of the proposed approach, it is applied to a quantum circuit of a hybrid quantum-classical algorithm, and then compared to an analytical solution as well as a non-hybrid version. The results show that, compared to the non-hybrid version, our method produces more diverse solutions and more accurate quantum operators which even reach the quality of the analytical baseline.
@inproceedings{10.1145/3583133.3590696, author = {Gemeinhardt, Felix and Klikovits, Stefan and Wimmer, Manuel}, title = {Hybrid Multi-Objective Genetic Programming for Parameterized Quantum Operator Discovery}, booktitle = {Genetic and Evolutionary Computation Conference Companion (GECCO'23)}, pages = {795–798}, numpages = {4}, year = {2023}, month = jul, publisher = {ACM}, keywords = {genetic programming, search-based quantum software engineering, quantum circuit synthesis, hybrid search}, isbn = {9798400701207}, doi = {10.1145/3583133.3590696}, location = {Lisbon, Portugal}, series = {GECCO '23 Companion}, }
Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty

Thomas Laurent, Stefan Klikovits, Paolo Arcaini, and 2 more authors

2023

Journal First presentation at 45th International Conference on Software Engineering (ICSE 2023)

Abs DOI: 10.1145/3550270

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this paper, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios; and that the cost of computing coverage can be managed with suitable heuristics.

SBFT Tool Competition 2023 - Cyber-Physical Systems Track

Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and 1 more author

In 16th IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT 2023, Melbourne, Australia, May 14, 2023, 2023

DOI: 10.1109/SBFT59156.2023.00010 Bib Website

@inproceedings{SBFT-toolcomp23,
  author = {Biagiola, Matteo and Klikovits, Stefan and Peltom\"{a}ki, Jarkko and Riccio, Vincenzo},
  title = {SBFT Tool Competition 2023 - Cyber-Physical Systems Track},
  booktitle = {16th {IEEE/ACM} International Workshop on Search-Based and Fuzz Testing, {SBFT} 2023, Melbourne, Australia, May 14, 2023},
  year = {2023},
  month = may,
  pages = {45-48},
  doi = {10.1109/SBFT59156.2023.00010},
}

Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty

Thomas Laurent, Stefan Klikovits, Paolo Arcaini, and 2 more authors

ACM Transactions on Software Engineering Methodology (TOSEM),Vol. 32(3), 2023

Abs DOI: 10.1145/3550270 Bib

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.
@article{10.1145/3550270, author = {Laurent, Thomas and Klikovits, Stefan and Arcaini, Paolo and Ishikawa, Fuyuki and Ventresque, Anthony}, title = {Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty}, year = {2023}, issue_date = {July 2023}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {32}, number = {3}, issn = {1049-331X}, url = {https://doi.org/10.1145/3550270}, doi = {10.1145/3550270}, journal = {ACM Transactions on Software Engineering Methodology (TOSEM)}, month = apr, articleno = {58}, numpages = {31}, keywords = {coverage criteria, autonomous driving, mutation analysis, Software testing}, }
Python Frameworks for Algorithms and Data Structures

Stefan Klikovits

2023

Invited Guest Lecture at IMC Fachhochschule Krems, Austria

Code
Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averaging

Stefan Klikovits, Cédric Ho Thanh, Ahmet Cetinkaya, and 1 more author

Applied Soft Computing,Vol. 146 2023

Abs DOI: 10.1016/j.asoc.2023.110631 Bib

Multi-objective optimisation (MOO) is a popular approach for finding solutions to many types of complex problems with large search spaces and conflicting search objectives. In the past, MOO algorithms have been shown to reliably produce good optimisation results. With the rise of cyber–physical systems, however, emerges the new challenge of non-deterministic system executions, caused by e.g. imperfect sensor readings or synchronisation in multi-process/multi-agent system architectures. These systems produce varying output on each execution, causing the algorithms’ observations to be noisy. Naturally, MOO algorithms favour the fittest solutions, which may have been measured with great inaccuracy. The end results are therefore not trustworthy. In this paper, we propose kNN-averaging, a new method to address this issue by identifying the k-nearest neighbours (kNN) of a solution, and using their weighted average as an estimate for its true fitness. Our experiments demonstrate the viability of kNN-averaging on 40 synthetic benchmark problems and on a real-world case study system. In the process, we compare kNN-averaging to the noisy baseline as well as two resampling-based methods and one spectral sampling approach on a range of algorithm settings. The results show that kNN-averaging approximates the fitness of solutions more accurately than the noisy baseline, leading to more trustworthy results.
@article{Klikovits:2023:2023110631, title = {{Trust your neighbours: Handling noise in multi-objective optimisation using kNN-averaging}}, journal = {Applied Soft Computing}, volume = {146}, pages = {110631}, year = {2023}, issn = {1568-4946}, doi = {10.1016/j.asoc.2023.110631}, url = {https://www.sciencedirect.com/science/article/pii/S156849462300649X}, author = {Klikovits, Stefan and {Ho Thanh}, Cédric and Cetinkaya, Ahmet and Arcaini, Paolo}, keywords = {kNN-averaging, Noisy system control, Noisy fitness, Search-based approaches, Multi-objective optimisation, Cyber–physical systems}, }

2022

FreneticV at the SBST 2022 Tool Competition

Ezequiel Castellano, Stefan Klikovits, Ahmet Cetinkaya, and 1 more author

In 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), 2022

Abs DOI: 10.1145/3526072.3527532 Bib Video

FreneticV is a search-based testing tool based on an evolutionary approach that generates roads where an automated driving agent possibly fails the lane-keeping task. It uses a curvature-based road representation and, compared to its predecessor Frenetic, considers the validity of the generated roads. In particular, it tries to avoid generating roads with overly sharp turns, detects self-intersecting roads, and can rotate and relocate roads to fit them in a given map.
@inproceedings{Castellano2022FreneticVAT, title = {FreneticV at the SBST 2022 Tool Competition}, author = {Castellano, Ezequiel and Klikovits, Stefan and Cetinkaya, Ahmet and Arcaini, Paolo}, booktitle = {2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)}, year = {2022}, month = may, pages = {47-48}, doi = {10.1145/3526072.3527532}, }
Dynamic Shielding for Reinforcement Learning in Black-Box Environments

Masaki Waga, Ezequiel Castellano, Sasinee Pruekprasert, and 3 more authors

In Automated Technology for Verification and Analysis (ATVA’22), 2022

Abs DOI: 10.1007/978-3-031-19992-9_2 Bib

It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.
@inproceedings{10.1007/978-3-031-19992-9_2, author = {Waga, Masaki and Castellano, Ezequiel and Pruekprasert, Sasinee and Klikovits, Stefan and Takisaka, Toru and Hasuo, Ichiro}, editor = {Bouajjani, Ahmed and Hol{\'i}k, Luk{\'a}{\v{s}} and Wu, Zhilin}, title = {Dynamic Shielding for Reinforcement Learning in Black-Box Environments}, booktitle = {Automated Technology for Verification and Analysis (ATVA'22)}, year = {2022}, publisher = {Springer International Publishing}, address = {Cham}, pages = {25--41}, isbn = {978-3-031-19992-9}, doi = {10.1007/978-3-031-19992-9_2}, }
Does Road Diversity Really Matter in Testing Automated Driving Systems? – A Registered Report

Stefan Klikovits, Vincenzo Riccio, Ezequiel Castellano, and 3 more authors

2022

Abs DOI: 10.48550/ARXIV.2209.05947 arXiv Bib Slides

**Background/Context.** The use of automated driving systems (ADSs) in the real world requires rigorous testing to ensure safety. To increase trust, ADSs should be tested on a large set of diverse road scenarios. Literature suggests that if a vehicle is driven along a set of geometrically diverse roads-measured using various diversity measures (DMs)-it will react in a wide range of behaviours, thereby increasing the chances of observing failures (if any), or strengthening the confidence in its safety, if no failures are observed. To the best of our knowledge, however, this assumption has never been tested before, nor have road DMs been assessed for their properties. **Objective/Aim.** Our goal is to perform an exploratory study on 47 currently used and new, potentially promising road DMs. Specifically, our research questions look into the road DMs themselves, to analyse their properties (e.g. monotonicity, computation efficiency), and to test correlation between DMs. Furthermore, we look at the use of road DMs to investigate whether the assumption that diverse test suites of roads expose diverse driving behaviour holds. **Method.** Our empirical analysis relies on a state-of-the-art, open-source ADSs testing infrastructure and uses a data set containing over 97,000 individual road geometries and matching simulation data that were collected using two driving agents. By sampling random test suites of various sizes and measuring their roads’ geometric diversity, we study road DMs properties, the correlation between road DMs, and the correlation between road DMs and the observed behaviour.
@misc{2209.05947, doi = {10.48550/ARXIV.2209.05947}, url = {https://arxiv.org/abs/2209.05947}, author = {Klikovits, Stefan and Riccio, Vincenzo and Castellano, Ezequiel and Cetinkaya, Ahmet and Gambi, Alessio and Arcaini, Paolo}, keywords = {Software Engineering (cs.SE), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Does Road Diversity Really Matter in Testing Automated Driving Systems? -- A Registered Report}, publisher = {arXiv}, year = {2022}, }

2021

Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems

Stefan Klikovits, and Paolo Arcaini

In 26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021), 2021

Abs DOI: 10.1109/PRDC53464.2021.00011 Bib

This paper presents the first evaluation of k-nearest neighbours-Averaging (kNN-Avg) on a real-world case study. kNN-Avg is a novel technique that tackles the challenges of noisy multi-objective optimisation (MOO). Existing studies suggest the use of repetition to overcome noise. In contrast, kNN-Avg approximates these repetitions and exploits previous executions, thereby avoiding the cost of re-running. We use kNN-Avg for the scenario generation of a real-world autonomous driving system (ADS) and show that it is better than the noisy baseline. Furthermore, we compare it to the repetition-method and outline indicators as to which approach to choose in which situations.
@inproceedings{KlikovitsA2021PRDC, title = {Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems}, author = {Klikovits, Stefan and Arcaini, Paolo}, booktitle = {26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021)}, publisher = {IEEE}, year = {2021}, month = nov, doi = {10.1109/PRDC53464.2021.00011}, }
Pragmatic Reuse for DSML Development

Stefan Klikovits, and Didier Buchs

2021

Invited Journal First talk at the 24th International Conference on Model Driven Engineering Languages and Systems (MODELS 21)

Abs

By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.
On the Need for Multi-Level ADS Scenarios

Stefan Klikovits, and Paolo Arcaini

In 3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS’21), 2021

Abs DOI: 10.1109/MODELS-C53483.2021.00054 Bib

Currently, most existing approaches for the design of Automated Driving Systems (ADS) scenarios focus on the description at one particular abstraction level — typically the most detailed one. This practice often removes information at higher levels, such that this data has to be re-synthesized if needed. As the abstraction granularity should be adapted to the task at hand, however, engineers currently have the choice between re-calculating the needed data or operating on the wrong level of abstraction. For instance, the search in a scenario database for a driving scenario with a map of a given road-shape should abstract over the lane markings, adjacent vegetation, or weather situation. Often though, the general road shape has to be synthesized (e.g. interpolated) from the precise GPS information of road boundaries. This paper outlines our vision for multi-level ADS scenario models that facilitate scenario search, generation, and design. Our concept is based on the common modelling philosophy to interact with scenarios at the most appropriate abstraction level. We identify different abstraction levels of ADS scenarios and suggest a template abstraction hierarchy. Our vision enables seamless traversal to such a most suitable granularity level for any given scenario, search and modelling task. We envision that this approach to ADS scenario modelling will have a lasting impact on the way we store, search, design, and generate ADS scenarios, allowing for a more strategic verification of autonomous vehicles in the long run.
@inproceedings{KlikovitsA2021MPM4CPS, title = {On the Need for Multi-Level ADS Scenarios}, author = {Klikovits, Stefan and Arcaini, Paolo}, booktitle = {3rd International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'21)}, series = {MODELS'21}, publisher = {ACM}, year = {2021}, month = oct, pages = {322-326}, doi = {10.1109/MODELS-C53483.2021.00054}, }
KNN-Averaging for Noisy Multi-Objective Optimisation

Stefan Klikovits, and Paolo Arcaini

In Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC), 2021

Abs DOI: 10.1007/978-3-030-85347-1_36 Bib

Multi-objective optimisation is a popular approach for finding solutions to complex problems with large search spaces that reliably yields good optimisation results. However, with the rise of cyber-physical systems, emerges a new challenge of noisy fitness functions, whose objective value for a given configuration is non-deterministic, producing varying results on each execution. This leads to an optimisation process that is based on stochastically sampled information, ultimately favouring solutions with fitness values that have co-incidentally high outlier noise. In turn, the results are unfaithful due to their large discrepancies between sampled and expectable objective values. Motivated by our work on noisy automated driving systems, we present the results of our ongoing research to counteract the effect of noisy fitness functions without requiring repeated executions of each solution. Our method kNN-Avg identifies the k-nearest neighbours of a solution point and uses the weighted average value as a surrogate for its actually sampled fitness. We demonstrate the viability of kNN-Avg on common benchmark problems and show that it produces comparably good solutions whose fitness values are closer to the expected value.
@inproceedings{KlikovitsA2021knnAveraging, author = {Klikovits, Stefan and Arcaini, Paolo}, title = {{KNN-Averaging for Noisy Multi-Objective Optimisation}}, editor = {Paiva, Ana C. R. and Cavalli, Ana Rosa and Ventura Martins, Paula and P{\'e}rez-Castillo, Ricardo}, booktitle = {Proc. 14th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC)}, publisher = {Springer International Publishing}, pages = {503--518}, series = {Communications in Computer and Information Science (CCIS)}, volume = {1439}, isbn = {978-3-030-85347-1}, location = {Faro, Portugal (Online)}, doi = {10.1007/978-3-030-85347-1_36}, year = {2021}, month = sep, }
Architecture-Guided Test Resource Allocation via Logic

Clovis Eberhart, Akihisa Yamada, Stefan Klikovits, and 4 more authors

In Tests and Proofs (TAP 2021), 2021

Abs DOI: 10.1007/978-3-030-79379-1_2 arXiv Bib

We introduce a new logic named Quantitative Confidence Logic (QCL) that quantifies the level of confidence one has in the conclusion of a proof. By translating a fault tree representing a system’s architecture to a proof, we show how to use QCL to give a solution to the test resource allocation problem that takes the given architecture into account. We implemented a tool called Astrahl and compared our results to other testing resource allocation strategies.
@inproceedings{Eberhart2021, author = {Eberhart, Clovis and Yamada, Akihisa and Klikovits, Stefan and Katsumata, Shin-ya and Kobayashi, Tsutomu and Hasuo, Ichiro and Ishikawa, Fuyuki}, title = {Architecture-Guided Test Resource Allocation via Logic}, booktitle = {Tests and Proofs (TAP 2021)}, series = {Lecture Notes in Computer Science (LNCS)}, volume = {12740}, doi = {10.1007/978-3-030-79379-1_2}, year = {2021}, month = jun, publisher = {Springer}, pages = {22--38}, }
Frenetic at the SBST 2021 Tool Competition

Ezequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, and 3 more authors

In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021

Abs DOI: 10.1109/SBST52555.2021.00016 Bib Video

Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimise the out of bound distance, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.
@inproceedings{Castellano:2021:SBST, author = {Castellano, Ezequiel and Cetinkaya, Ahmet and Thanh, Cédric Ho and Klikovits, Stefan and Zhang, Xiaoyi and Arcaini, Paolo}, title = {Frenetic at the SBST 2021 Tool Competition}, booktitle = {2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)}, year = {2021}, month = may, editor = {Zhang, Jie and Fredericks, Erik}, pages = {36-37}, publisher = {IEEE}, keywords = {genetic algorithms, genetic programming}, doi = {10.1109/SBST52555.2021.00016}, }
Chapter 9 – CREST — a DSML for hybrid CPS modelling

Stefan Klikovits, and Didier Buchs

In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, 2021

Abs DOI: 10.1016/B978-0-12-819105-7.00014-3 Bib

Hybrid systems modelling remains a very popular topic within the modelling and simulation community. Its expressiveness allows for the definition of highly complex systems that merge discrete-state-based transitions systems with continuous value-evolutions for variables, so that cyber-physical systems can be modelled in all their intricacies. This expressive power, however, comes at a downside of complex models and undecidable verification problems even for small systems. In this chapter we present CREST, a novel modelling language for the definition of hybrid systems. CREST merges features from various formalisms and languages such as hybrid automata, dataflow programming and internal domain-specific modelling language (DSML) designs to create simple yet powerful language for the modelling of resource flows within small-scale CPS such as automated gardening applications and smart homes. The language provides an easy to learn graphical interface and is supported by a Python-based tool implementation that allows for the efficient modelling, simulation and verification of CPS models.
@incollection{Klikovits:2021:mpma4cps:ch9, title = {{Chapter 9 -- CREST --- a DSML for hybrid CPS modelling}}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, publisher = {Academic Press}, pages = {221-242}, year = {2021}, isbn = {978-0-12-819105-7}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00014-3}, url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000143}, author = {Klikovits, Stefan and Buchs, Didier}, }
Chapter 4 - An ontology for multi-paradigm modelling

Holger Giese, Dominique Blouin, Rima Al-Ali, and 6 more authors

In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, 2021

Abs DOI: 10.1016/B978-0-12-819105-7.00009-X Bib

This chapter presents the ontology for Multi-Paradigm Modelling (MPM) specified using the Web Ontology Language (OWL) as introduced in Chapter 2. A thorough state of the art on MPM’s core notions including multi-formalism and model management approaches, languages and tools is presented. In particular, model management approaches have been characterized according to their modularity and incremental execution properties as required to scale for the large complex CPSs we face today. Based on this state of the art, an outline of the MPM ontology is developed by introducing its main classes and properties. The validation of the ontology is presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2.
@incollection{Klikovits:2021:mpma4cps:ch4, title = {{Chapter 4 - An ontology for multi-paradigm modelling}}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, publisher = {Academic Press}, pages = {67-122}, year = {2021}, isbn = {978-0-12-819105-7}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00009-X}, url = {https://www.sciencedirect.com/science/article/pii/B978012819105700009X}, author = {Giese, Holger and Blouin, Dominique and Al-Ali, Rima and Mkaouar, Hana and Bandyopadhyay, Soumyadip and Iacono, Mauro and Amrani, Moussa and Klikovits, Stefan and Erata, Ferhat}, }
Chapter 5 - An integrated ontology for multi-paradigm modelling for cyber-physical systems

Dominique Blouin, Rima Al-Ali, Holger Giese, and 4 more authors

In Multi-Paradigm Modelling Approaches for Cyber-Physical Systems, 2021

Abs DOI: 10.1016/B978-0-12-819105-7.00010-6 Bib

This chapter presents the Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS) ontology. This ontology integrates the Shared, MPM and CPS ontologies respectively introduced in Chapters 2, 3 and 4. It includes cross-cutting notions such as viewpoints, model-based development processes and modelling paradigms that together relate the formalisms and workflows (and their paradigms) to the part of CPSs developed with these formalisms. A brief state of the art on these notions is first presented, on which the MPM4CPS ontology builds. An overview of the ontology is then developed by introducing its main classes and properties. The validation of the ontology is finally presented by showing how it can adequately model the two case studies briefly introduced in Chapter 2. The chapter also discusses perspectives and future work on this integrated ontological framework, which can serve as a basis to develop model management solutions to relate and combine modelling languages and tools, in order to better develop cyber-physical systems with appropriate formalisms and workflows.
@incollection{Klikovits:2021:mpma4cps:ch5, title = {{Chapter 5 - An integrated ontology for multi-paradigm modelling for cyber-physical systems}}, author = {Blouin, Dominique and Al-Ali, Rima and Giese, Holger and Klikovits, Stefan and Bandyopadhyay, Soumyadip and Barišić, Ankica and Erata, Ferhat}, booktitle = {Multi-Paradigm Modelling Approaches for Cyber-Physical Systems}, editor = {Tekinerdogan, Bedir and Blouin, Dominique and Vangheluwe, Hans and Goulão, Miguel and Carreira, Paulo and Amaral, Vasco}, pages = {123-145}, year = {2021}, doi = {https://doi.org/10.1016/B978-0-12-819105-7.00010-6}, url = {https://www.sciencedirect.com/science/article/pii/B9780128191057000106}, isbn = {978-0-12-819105-7}, publisher = {Academic Press}, }
Pragmatic Reuse for DSML Development

Stefan Klikovits, and Didier Buchs

Software and Systems Modeling (SoSyM),Vol. 20 2021

Abs DOI: 10.1007/s10270-020-00831-4 Bib

By bridging the semantic gap, domain-specific language (DSLs) serve an important role in the conquest to allow domain experts to model their systems themselves. In this publication we present a case study of the development of the Continuous REactive SysTems language (CREST), a DSL for hybrid systems modeling. The language focuses on the representation of continuous resource flows such as water, electricity, light or heat. Our methodology follows a very pragmatic approach, combining the syntactic and semantic principles of well-known modeling means such as hybrid automata, data-flow languages and architecture description languages into a coherent language. The borrowed aspects have been carefully combined and formalised in a well-defined operational semantics. The DSL provides two concrete syntaxes: CREST diagrams, a graphical language that is easily understandable and serves as a model basis, and crestdsl, an internal DSL implementation that supports rapid prototyping—both are geared towards usability and clarity. We present the DSL’s semantics, which thoroughly connect the various language concerns into an executable formalism that enables sound simulation and formal verification in crestdsl, and discuss the lessons learned throughout the project.
@article{Klikovits:2020:sosym:crest, author = {Klikovits, Stefan and Buchs, Didier}, title = {{Pragmatic Reuse for DSML Development}}, journal = {{Software and Systems Modeling (SoSyM)}}, doi = {10.1007/s10270-020-00831-4}, year = {2021}, volume = {20}, pages = {837-866}, }

2020

Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event Systems

Didier Buchs, Stefan Klikovits, and Alban Linard

In Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems, 2020

Abs DOI: 10.1007/978-3-030-43946-0 Bib

The study of concurrent and parallel systems has been a challenging research domain within cyber-physical systems community. This chapter provides a pragmatic introduction to the creation and analysis of such system models using the popular Petri nets formalism. Petri nets is a formalism that convinces through its simplicity and applicability. We offer an overview of the most important Petri nets concepts, analysis techniques and model checking approaches. Finally, we show the use of so-called High-level Petri nets for the representation of complex data structures and functionality and present a novel research approach that allows the use of Petri nets inside Functional Mock-up Units and cyber-physical system models.
@incollection{Klikovits:2020:mpm4cps:foundationsbook, author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban}, title = {{Petri Nets: A Formal Language to Specify and Verify Concurrent Non-Deterministic Event Systems}}, booktitle = {{Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems}}, editor = {Carreira, P. and Amaral, V. and Vangheluwe, H.}, publisher = {Springer}, doi = {10.1007/978-3-030-43946-0}, isbn = {978-3-030-43946-0}, year = {2020}, }

2019

A Domain-Specific Language Approach To Hybrid CPS Modelling

Stefan Klikovits

University of Geneva, Switzerland, 2019

PhD Thesis

DOI: 10.13097/archive-ouverte/unige:121355 Bib

@phdthesis{klikovits_phdthesis_2019,
  author = {Klikovits, Stefan},
  title = {A Domain-Specific Language Approach To Hybrid CPS Modelling},
  year = {2019},
  month = jun,
  school = {University of Geneva, Switzerland},
  note = {PhD Thesis},
  url = {https://archive-ouverte.unige.ch/unige:121355},
  doi = {10.13097/archive-ouverte/unige:121355},
}

COST IC1404 WG1 Deliverable WG1.1: State-of-the-art on Current Formalisms used in Cyber-Physical Systems Development

Stefan Klikovits, Rima Al-Ali, Moussa Amrani, and 12 more authors

2019

DOI: 10.5281/zenodo.2533455 Bib

@misc{klikovits_stefan_2019_2538711,
  author = {Klikovits, Stefan and Al-Ali, Rima and Amrani, Moussa and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Borde, Etienne and Buchs, Didier and Giese, Holger and Goulao, Miguel and Iacono, Mauro and Leon, Florin and Navarro, Eva and Pelliccione, Patrizio and Vanherpen, Ken},
  title = {{COST IC1404 WG1 Deliverable WG1.1:
        State-of-the-art on Current Formalisms used
        in Cyber-Physical Systems Development}},
  month = jan,
  year = {2019},
  doi = {10.5281/zenodo.2533455},
  url = {https://doi.org/10.5281/zenodo.2533455},
}

COST IC1404 WG1 Deliverable WG1.2: Framework to Relate / Combine Modeling Languages and Techniques

Rima Al-Ali, Moussa Amrani, Soumyadip Bandyopadhyay, and 12 more authors

2019

DOI: 10.5281/zenodo.2527576 Bib

@misc{al_ali_rima_2019_2527577,
  author = {Al-Ali, Rima and Amrani, Moussa and Bandyopadhyay, Soumyadip and Barisic, Ankica and Barros, Fernando and Blouin, Dominique and Erata, Ferhat and Giese, Holger and Iacono, Mauro and Klikovits, Stefan and Navarro, Eva and Pelliccione, Patrizio and Taveter, Kuldar and Tekinerdogan, Bedir and Vanherpen, Ken},
  title = {{COST IC1404 WG1 Deliverable WG1.2:
        Framework to Relate / Combine
        Modeling Languages and Techniques}},
  month = jan,
  year = {2019},
  doi = {10.5281/zenodo.2527576},
  url = {https://doi.org/10.5281/zenodo.2527576},
}

2018

ML4CREST: Machine Learning for CPS Models

Stefan Klikovits, Aurélien Coet, and Didier Buchs

In 2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT), 2018

Abs Bib PDF Slides

Models of small CPS and IoT applications often use approximated values that describe physical system behaviour. Physical resources, such as electricity consumption and heating power, have to be estimated, since many off-the-shelf components lack the required descriptions. Controllers which are based on these approximations can hence use imprecise models, perform misleading simulation, and cause damaged systems and financial loss. In this paper we present ML4CREST, a machine learning approach to automatically calibrate models using sensor measurements. We show that our approach is well-suited for the calibration of the flow rates within an automated watering system, which allows precise simulation and prevents spillage.
@inproceedings{Klikovits:MDE4IOT:CREST, title = {{ML4CREST: Machine Learning for CPS Models}}, author = {Klikovits, Stefan and Coet, Aur\'{e}lien and Buchs, Didier}, year = {2018}, month = oct, pages = {515-520}, url = {http://ceur-ws.org/Vol-2245/mde4iot_paper_4.pdf}, volume = {2245}, series = {CEUR Workshop Proceedings}, booktitle = {2nd International Workshop on Model-Driven Engineering for the Internet-of-Things (MDE4IoT)}, }
CREST - A DSL for Reactive Cyber-Physical Systems

Stefan Klikovits, Alban Linard, and Didier Buchs

In 10th System Analysis and Modeling Conference (SAM2018). Languages, Methods, and Tools for Systems Engineering, 2018

Abs DOI: 10.1007/978-3-030-01042-3_3 Bib Slides

This article presents CREST, a novel domain-specific language for the modelling of cyber-physical systems. CREST is designed for the simple and clear modelling, simulation and verification of small-scale systems such as home and office automation, smart gardening systems and similar. The language is designed to model the flow of resources throughout the system. It features synchronous system evolution and reactive behaviour. CREST’s formal semantics allow real-valued time advances and the modelling of timed system evolution. The continuous time concept permits the precise simulation of future system behaviour by automatically calculating next transition times. We present CREST in a practical manner, and elaborate on the Python-based DSL implementation and simulator.
@inproceedings{Klikovits:SAM18:CREST, title = {{{CREST}} - {{A DSL}} for {{Reactive Cyber}}-{{Physical Systems}}}, isbn = {978-3-030-01042-3}, booktitle = {10th System {{Analysis}} and {{Modeling}} {{Conference}} ({{SAM2018}}). {{Languages}}, {{Methods}}, and {{Tools}} for {{Systems Engineering}}}, publisher = {{Springer}}, series = {Lecture Notes in Computer Science}, volume = {11150}, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, editor = {Khendek, Ferhat and Gotzhein, Reinhard}, year = {2018}, month = oct, pages = {29-45}, doi = {10.1007/978-3-030-01042-3_3}, }
Petri Sport: A Sport for Petri Netters

Stefan Klikovits, Alban Linard, Dimitri Racordon, and 1 more author

In Petri Nets and Software Engineering. International Workshop, PNSE’18, 2018

Abs Bib PDF Slides

Petri nets are a family of formalisms dedicated to the representation of concurrent systems. Their strength is the compact modeling of complex behaviors using very simple rules. Despite this simplicity, many teachers observe that students often require a lot of exposure and numerous exercises to truly understand the semantics of Petri nets. In order to speed up this learning process and provide a different attack angle, we propose Petri sport, a fun game based upon the Petri net formalism. In Petri sport, players aim to gather points by moving across a Petri net-shaped playing field and "firing" transitions. A clock-based play style supports a structured game advance while at the same time it encourages players to move fast. As the playing field is shaped like a Petri net, it is possible to challenge a player’s movement speed, intellectual capabilities, as well as team coordination and communication. The difficulty level of Petri sport is based on the choice of playing field. This allows for adaptation in order to best fit the competitors’ age, experience and/or physical fitness level.
@inproceedings{Klikovits:PNSE18:PetriSport, title = {{Petri Sport: A Sport for Petri Netters}}, pages = {35--56}, author = {Klikovits, Stefan and Linard, Alban and Racordon, Dimitri and Buchs, Didier}, year = {2018}, month = sep, booktitle = {Petri Nets and Software Engineering. International Workshop, PNSE'18}, editor = {Moldt, Daniel and Kindler, Ekkart and R{\"o}lke, Heiko}, series = {CEUR Workshop Proceedings}, volume = {2138}, url = {http://ceur-ws.org/Vol-2138/paper2.pdf}, }
A Model Checker Collection for the Model Checking Contest Using Docker and Machine Learning

Didier Buchs, Stefan Klikovits, Alban Linard, and 2 more authors

In Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, PETRI NETS 2018, 2018

Abs DOI: 10.1007/978-3-319-91268-4_21 Bib Slides

This paper introduces mcc4mcc, the Model Checker Collection for the Model Checking Contest, a tool that wraps multiple model checking solutions, and applies the most appropriate one based on the characteristics of the model it is given. It leverages machine learning algorithms to carry out this selection, based on the results gathered from the 2017 edition of the Model Checking Contest, an annual event in which multiple tools compete to verify different properties on a large variety of models. Our approach brings two important contributions. First, our tool offers the opportunity to further investigate on the relation between model characteristics and verification techniques. Second, it lays out the groundwork for a unified way to distribute model checking software using virtual containers.
@inproceedings{DBLP:conf/apn/BuchsKLMR18, author = {Buchs, Didier and Klikovits, Stefan and Linard, Alban and Mencattini, Romain and Racordon, Dimitri}, title = {{A Model Checker Collection for the Model Checking Contest Using Docker and Machine Learning}}, booktitle = {Proceedings of Application and Theory of Petri Nets and Concurrency - 39th International Conference, {PETRI} {NETS} 2018}, pages = {385--395}, year = {2018}, month = sep, series = {Lecture Notes in Computer Science}, volume = {10877}, publisher = {Springer}, doi = {10.1007/978-3-319-91268-4_21}, url = {https://doi.org/10.1007/978-3-319-91268-4_21}, }
CREST Formalization

Stefan Klikovits, Alban Linard, and Didier Buchs

2018

Abs DOI: 10.5281/zenodo.1284561 Bib

CREST is a novel modelling language for the definition of Continuous-time, REactive SysTems. This domain-specific language (DSL) targets small cyber-physical systems (CPS) such as home automation systems. While CREST is a graphical language and its systems can be visualised as CREST diagrams, the main form of use is as internal DSL for the Python general purpose programming language. Nevertheless, CREST systems are based on a formal structure and semantics. This report provides this formalisation and elaborates on the design choices that have been made.
@techreport{Klikovits:CRESTFormalization, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, title = {{CREST} Formalization}, institution = {Software Modeling and Verification Group, University of Geneva}, doi = {10.5281/zenodo.1284561}, year = {2018}, month = jun, }

2017

CREST - A Continuous, REactive SysTems DSL

Stefan Klikovits, Alban Linard, and Didier Buchs

In 5th International Workshop on the Globalization of Modeling Languages (GEMOC 2017), 19 September 2017, Austin, TX, USA, 2017

Abs Bib Slides

The advance of cyber-physical systems in everyday life requires powerful modeling capabilities. Existing formalisms often have severe limitations and require complicated notations. In this paper we introduce CREST, a domain-specific language for modeling entity behavior and resource transfers in CPS. CREST aims to support CPS architects through clarity, comprehensiveness and analyzability.
@inproceedings{DBLP:conf/models/KlikovitsLB17, author = {Klikovits, Stefan and Linard, Alban and Buchs, Didier}, title = {{CREST} - {A} Continuous, REactive SysTems {DSL}}, booktitle = {5th International Workshop on the {Globalization of Modeling Languages} {(GEMOC} 2017), 19 September 2017, Austin, TX, USA}, pages = {286--291}, year = {2017}, month = sep, url = {http://ceur-ws.org/Vol-2019/gemoc_2.pdf}, volume = {2019}, series = {CEUR Workshop Proceedings}, }
Modeling Frames

Stefan Klikovits, Joachim Denil, Alexandre Muzy, and 1 more author

In 14th Workshop on Model-Driven Engineering, Verification and Validation (MoDeVVa 2017), 19 September 2017, Austin, TX, USA, 2017

Abs Bib Slides

Modeling activities such as calibration, verification and validation are often executed in under-specified environments. This hinders reproducibility, reduces re-usability and generally decreases the modeling precision and quality. This paper describes a framework for the definition of model frames. Model frames capture the context an activity/model is executed in and facilitate re-use, replacement, validation and verification of models. We show the application of the frames approach onto a real-world example, introduce several kinds of frames and show their application on this case study.
@inproceedings{DBLP:conf/models/KlikovitsDMS17, author = {Klikovits, Stefan and Denil, Joachim and Muzy, Alexandre and Salay, Rick}, title = {Modeling Frames}, booktitle = {14th Workshop on {Model-Driven Engineering, Verification and Validation} {(MoDeVVa} 2017), 19 September 2017, Austin, TX, USA}, volume = {2019}, series = {CEUR Workshop Proceedings}, pages = {315--320}, year = {2017}, month = sep, }
The Experiment Model and Validity Frame in M&S

Joachim Denil, Stefan Klikovits, Pieter J. Mosterman, and 2 more authors

In Proceedings of the Symposium on Theory of Modeling & Simulation, 2017

Abs Bib

Modelling and Simulation approaches use system models to conduct simulation experiments. Experimental frames have been applied in this context to formally define a system’s context. During the creation of an experimental frame for a simple spring model it becomes clear that experimental frames in their current definition lack certain properties and omit relevant information. Our approach describes the process of capturing the context of models and systems to provide truly reproducible experiment descriptions. The information, captured as experimental setups, can then be used for different purposes and in different scenarios, in particular for checking the validity of a model, the discovery of suitable models for the design of a system, and for calibrating models.
@inproceedings{denil:2017:tmsdevs, author = {Denil, Joachim and Klikovits, Stefan and Mosterman, Pieter J. and Vallecillo, Antonio and Vangheluwe, Hans}, title = {The Experiment Model and Validity Frame in M\&S}, year = {2017}, month = apr, booktitle = {Proceedings of the Symposium on Theory of Modeling \& Simulation}, series = {TMS/DEVS '17}, pages = {109--120}, isbn = {ISBN: 978-1-5108-3823-9}, venue = {Virginia Beach, Virginia, USA}, url = {http://scs.org/wp-content/uploads/2017/06/27_Final_Manuscript.pdf}, }
Towards Language Independent (Dynamic) Symbolic Execution

Stefan Klikovits, Manuel Gonzalez-Berges, and Didier Buchs

In Proceedings of the 24th PhD Mini-Symposium, 2017

Abs DOI: 10.5281/zenodo.291899 Bib Slides

Symbolic execution is well-known for its capability to produce high-coverage test suites for software source code. So far, most tools are created to support a specific language. This paper elaborates on performing language independent symbolic execution and three ways to achieve it. We describe the use of one approach to perform dynamic symbolic execution using translation of a proprietary language and show the results of the tool execution on a real-life codebase.
@inproceedings{klikovits:2017:minisy, author = {Klikovits, Stefan and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Towards Language Independent (Dynamic) Symbolic Execution}, year = {2017}, month = jan, booktitle = {Proceedings of the 24th PhD Mini-Symposium}, pages = {50--53}, isbn = {978-963-313-243-2}, venue = {Budapest, Hungary}, url = {https://doi.org/10.5281/zenodo.291899}, publisher = {{Budapest University of Technology and Economics, Department of Measurement and Information Systems}}, doi = {10.5281/zenodo.291899}, }

2016

Automated Test Case Generation for the CTRL Programming Language Using Pex: Lessons Learned

Stefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and 1 more author

In 8th International Workshop on Software Engineering for Resilient Systems (SERENE), 2016

Abs DOI: 10.1007/978-3-319-45892-2_9 Bib Slides

Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. In this paper we first present ITEC, a tool for automated test case generation in CTRL, as well as initial results of test cases executions on one of CERN’s SCADA frameworks. Our tool relies on Microsoft’s Pex for its code exploration. For the purpose of using this existing test generation tool, we have to translate the proprietary CTRL code into C#, one of Pex’s operating languages. Our main contribution lies in detailing a formal foundation for this step through source code decomposition and anonymization. We then propose a quality measure that is used to determine our confidence into the translation and the generated test cases.
@inproceedings{DBLP:conf/serene/KlikovitsLGB16, author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Automated Test Case Generation for the {CTRL} Programming Language Using Pex: Lessons Learned}, booktitle = {8th International Workshop on Software Engineering for Resilient Systems (SERENE)}, pages = {117--132}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, volume = {9823}, year = {2016}, month = sep, url = {https://doi.org/10.1007/978-3-319-45892-2_9}, doi = {10.1007/978-3-319-45892-2_9}, }

2015

Considering Execution Environment Resilience: A White-Box Approach

Stefan Klikovits, David P. Y. Lawrence, Manuel Gonzalez-Berges, and 1 more author

In 7th International Workshop on Software Engineering for Resilient Systems (SERENE), 2015

Abs DOI: 10.1007/978-3-319-23129-7_4 Bib Slides

Over the last decade code-based test case generation techniques such as combinatorial testing or dynamic symbolic execution have seen growing research popularity. Most algorithms and tool implementations are based on finding assignments for input parameter values in order to maximise the execution branch coverage. Only few of them consider dependencies from outside the Code Under Test’s scope such as global variables, database values and subroutine calls as influences to the execution path. In order to fully test all possible scenarios these dependencies have to be taken into account for the test input generation. This paper introduces ITEC, a tool for automated test case generation to support execution environment resilience in large-scaled, complex systems. One of ITEC’s corner stones is a technique called semi-purification, a source code transformation technique to overcome limitations of existing tools and to set up the required system state for software testing.
@inproceedings{DBLP:conf/serene/KlikovitsLGB15, author = {Klikovits, Stefan and Lawrence, David P. Y. and Gonzalez{-}Berges, Manuel and Buchs, Didier}, title = {Considering Execution Environment Resilience: {A} White-Box Approach}, booktitle = {7th International Workshop on Software Engineering for Resilient Systems (SERENE)}, pages = {46--61}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, volume = {9274}, month = sep, year = {2015}, url = {https://doi.org/10.1007/978-3-319-23129-7_4}, doi = {10.1007/978-3-319-23129-7_4}, }

2011

Multi-platform SCADA GUI Regression Testing at CERN

Paul C. Burkimsher, Manuel Gonzalez-Berges, and Stefan Klikovits

In Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS), 2011

Abs Bib PDF Slides

The JCOP Framework is a toolkit used widely at CERN for the development of industrial control systems in several domains (i.e. experiments, accelerators and technical infrastructure). The software development started 10 years ago and there is now a large base of production systems running it. For the success of the project, it was essential to formalize and automate the quality assurance process. The paper will present the overall testing strategy and will describe in detail mechanisms used for GUI testing. The choice of a commercial tool (Squish) and the architectural features making it appropriate for our multi-platform environment will be described. Practical difficulties encountered when using the tool in the CERN context are discussed as well as how these were addressed. In the light of initial experience, the test code itself has been recently reworked in OO style to facilitate future maintenance and extension. The paper concludes with a description of our initial steps towards incorporation of full-blown Continuous Integration (CI) support.
@inproceedings{burkimsher2011icalepcs, author = {Burkimsher, Paul C. and Gonzalez{-}Berges, Manuel and Klikovits, Stefan}, title = {Multi-platform SCADA GUI Regression Testing at CERN}, booktitle = {Proceedings of the 13th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS)}, year = {2011}, venue = {Grenoble, France}, publisher = {JACoW}, url = {https://accelconf.web.cern.ch/icalepcs2011/papers/thbhmust01.pdf}, }