publications

2024

1. FSE Visions 2024

   

   The lion, the ecologist and the plankton: a classification of species in multi-bot ecosystems

   Dimitrios Platis, Linda Erlenhov, and Francisco Gomes Oliveira Neto

   In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024) - Ideas, Visions, and Reflections Track , 2024

   To appear.

   Bib

   @inproceedings{platis:24,
     title = {{The lion, the ecologist and the plankton: a classification of species in multi-bot ecosystems}},
     booktitle = {{Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024) - Ideas, Visions, and Reflections Track}},
     author = {Platis, Dimitrios and Erlenhov, Linda and de Oliveira Neto, Francisco Gomes},
     year = {2024},
     note = {To appear.},
   }

2. AIware @ FSE 2024

   

   From Human-to-Human to Human-to-Bot Conversations in Software Engineering

   Ranim Khojah, Francisco Gomes Oliveira Neto, and Philipp Leitner

   In Proceedings of the The First International Conference on AIware (AIware) , 2024

   To appear.

   Bib PDF

   @inproceedings{khojah:24:2,
     title = {{From Human-to-Human to Human-to-Bot Conversations in Software
       Engineering}},
     booktitle = {{Proceedings of the The First International Conference on AIware (AIware)}},
     author = {Khojah, Ranim and de Oliveira Neto, Francisco Gomes and Leitner, Philipp},
     year = {2024},
     note = {To appear.},
   }

3. FSE 2024

   

   Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

   Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes Oliveira Neto

   In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024) , 2024

   To appear.

   Bib PDF Supp

   @inproceedings{khojah:24,
     title = {{Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice}},
     booktitle = {{Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024)}},
     author = {Khojah, Ranim and Mohamad, Mazen and Leitner, Philipp and de Oliveira Neto, Francisco Gomes},
     year = {2024},
     note = {To appear.},
   }

4. JSS

   

   An empirical investigation on the competences and roles of practitioners in Microservices-based Architectures

   Hamdy Michael Ayas, Regina Hebig, and Philipp Leitner

   Journal of Systems and Software, 2024

   Abs Bib Supp

   Microservices-based Architectures (MSAs) are gaining popularity since, among others, they enable rapid and independent delivery of software at scale, facilitating the delivery of business value. Additionally, there are attempts towards understanding practitioners’ roles and technical knowledge. MSAs call for affinity in several technologies as well as business domains. This diversity makes it challenging to scope and describe the roles of practitioners. In addition, practitioners often do not receive training and contents of MSA training remain largely undefined, even though there are challenges in finding or developing relevant technical expertise. In this research, we determine the different technical roles that are required in MSAs, along with their detailed competences. We use public online forums (e.g., StackOverflow), where developers share technical knowledge. We analyze 13,517 public profiles of software engineers, deriving their technical competences. Our taxonomy of technical competences in MSAs, contains 11 competences clusters, organized in 3 collections of competences — Web Technologies, DevOps, and Data Technologies. In addition, we derive the roles of microservice practitioners and the characteristics of their roles. Our findings organize the technical competences of MSAs practitioners and determine the training topics and combination of topics that can prepare engineers for MSAs.

   @article{ayas:24:1,
     title = {An empirical investigation on the competences and roles of practitioners in Microservices-based Architectures},
     journal = {Journal of Systems and Software},
     volume = {213},
     pages = {112055},
     year = {2024},
     issn = {0164-1212},
     doi = {https://doi.org/10.1016/j.jss.2024.112055},
     author = {{Michael Ayas}, Hamdy and Hebig, Regina and Leitner, Philipp},
     keywords = {Microservices, Technical competences, Practitioners profiles, Developer roles, Stackoverflow},
   }

5. CHASE @ ICSE 2024

   

   The Roles, Responsibilities, and Skills of Engineers in the Era of Microservices-Based Architectures

   Hamdy Michael Ayas, Regina Hebig, and Philipp Leitner

   In Proceedings of the 16th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE 2023) , 2024

   To appear.

   Bib

   @inproceedings{ayas:24,
     title = {{The Roles, Responsibilities, and Skills of Engineers in the Era of Microservices-Based Architectures}},
     booktitle = {{Proceedings of the 16th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE 2023)}},
     author = {Ayas, Hamdy Michael and Hebig, Regina and Leitner, Philipp},
     year = {2024},
     note = {To appear.},
   }

6. ICPC @ ICSE 2024

   

   The Impact of Compiler Warnings on Code Quality in C++ Projects

   Albin Johansson, Carl Holmberg, Francisco Gomes De Oliveira Neto, and Philipp Leitner

   In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension , Lisbon, Portugal, 2024

   Abs Bib Thesis

   Modern compilers often offer a variety of warning flags, which developers can enable to get feedback on code that, while syntactically correct, may be problematic. In the case of C++, one example of such "correct but problematic" code is code that leads to undefined behavior (UB). The usage of compiler warnings has long been suspected as a way to decrease bugs and increase code quality. However, empirical evidence that supports this hypothesis is rare. In this study, we present evidence from a study of 127 open source C++ projects. We categorize their usage of compiler warnings into five groups based on which warning flags are being used, and analyse the relationship between compiler warnings and five quality metrics (bugs, critical issues, vulnerabilities, code smells, and technical debt) using Bayesian analysis. We conclude that, in general, compiler warnings indeed correlate with, and potentially cause, higher code quality, with the clearest impact being on the number of critical issues in a project. Using stricter warning flags expectantly correlates with higher code quality in our study objects. However, there are substantial differences between projects, which we attribute to the project’s individual development culture. That is, while warnings matter, other factors such as quality culture, are likely to be even more important to source code quality.

   @inproceedings{johansson:24,
     author = {Johansson, Albin and Holmberg, Carl and Gomes De Oliveira Neto, Francisco and Leitner, Philipp},
     title = {The Impact of Compiler Warnings on Code Quality in C++ Projects},
     year = {2024},
     isbn = {9798400705861},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3643916.3644410},
     booktitle = {Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension},
     pages = {270–279},
     numpages = {10},
     keywords = {compiler warnings, code quality, bayesian data analysis},
     location = {Lisbon, Portugal},
     series = {ICPC '24},
   }

7. SIGMETRICS 2024

   

   Machine Learning Systems are Bloated and Vulnerable

   Huaifeng Zhang, Mohannad Alhanahnah, Fahmi Abdulqadir Ahmed, Dyako Fatih, Philipp Leitner, and Ahmed Ali-Eldin

   Proceedings of SIGMETRICS, Feb 2024

   Abs Bib PDF

   Today’s software is bloated with both code and features that are not used by most users. This bloat is prevalent across the entire software stack, from operating systems and applications to containers. Containers are lightweight virtualization technologies used to package code and dependencies, providing portable, reproducible and isolated environments. For their ease of use, data scientists often utilize machine learning containers to simplify their workflow. However, this convenience comes at a cost: containers are often bloated with unnecessary code and dependencies, resulting in very large sizes. In this paper, we analyze and quantify bloat in machine learning containers. We develop MMLB, a framework for analyzing bloat in software systems, focusing on machine learning containers. MMLB measures the amount of bloat at both the container and package levels, quantifying the sources of bloat. In addition, MMLB integrates with vulnerability analysis tools and performs package dependency analysis to evaluate the impact of bloat on container vulnerabilities. Through experimentation with 15 machine learning containers from TensorFlow, PyTorch, and Nvidia, we show that bloat accounts for up to 80% of machine learning container sizes, increasing container provisioning times by up to 370% and exacerbating vulnerabilities by up to 99%.

   @article{zhang:24,
     author = {Zhang, Huaifeng and Alhanahnah, Mohannad and Ahmed, Fahmi Abdulqadir and Fatih, Dyako and Leitner, Philipp and Ali-Eldin, Ahmed},
     title = {Machine Learning Systems are Bloated and Vulnerable},
     year = {2024},
     issue_date = {March 2024},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {8},
     number = {1},
     doi = {10.1145/3639032},
     journal = {Proceedings of SIGMETRICS},
     month = feb,
     articleno = {6},
     numpages = {30},
     keywords = {machine learning systems, software debloating},
   }

2023

1. Big Data 2023

   

   Batch Mode Deep Active Learning for Regression on Graph Data

   P. Samoaa, L. Aronsson, P. Leitner, and M. Chehreghani

   In 2023 IEEE International Conference on Big Data (BigData) , Dec 2023

   Abs Bib

   Acquiring labelled data for machine learning tasks, for example, for software performance prediction, remains a resource-intensive task. This study extends our previous work by introducing a batch-mode deep active learning approach tailored for regression in graph-structured data. Our framework leverages the source code conversion into Flow Augmented-AST graphs (FA-AST), subsequently utilizing both supervised and unsupervised graph embeddings. In contrast to single-instance querying, the batch-mode paradigm adaptively selects clusters of unlabeled data for labelling. We deploy an array of base kernels, kernel transformations, and selection methods, informed by both Bayesian and non-Bayesian strategies, to enhance the sample efficiency of neural network regression. Our experimental evaluation, conducted on multiple real-world software performance datasets, demonstrates the efficacy of the batch mode deep active learning approach in achieving robust performance with a reduced labelling budget. The methodology scales effectively to larger datasets and requires minimal alterations to existing neural network architectures.

   @inproceedings{samoaa:23,
     author = {Samoaa, P. and Aronsson, L. and Leitner, P. and Chehreghani, M.},
     booktitle = {2023 IEEE International Conference on Big Data (BigData)},
     title = {Batch Mode Deep Active Learning for Regression on Graph Data},
     year = {2023},
     volume = {},
     issn = {},
     pages = {5904-5913},
     keywords = {source coding;neural networks;software performance;machine learning;gaussian processes;computer architecture;big data},
     doi = {10.1109/BigData59044.2023.10386685},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = dec,
   }

2. SOSE 2023

   

   The Perceived Impact and Sequence of Activities When Transitioning to Microservices

   H. Ayas, P. Leitner, and R. Hebig

   In 2023 IEEE International Conference on Service-Oriented System Engineering (SOSE) , Jul 2023

   (short paper)

   Abs Bib

   Microservices migrations are often present in organizations that want to modernize and integrate their software systems. There is little empirical evidence showing how migration projects take place. Investigating migration activities and the perceived impact of migrations from practitioners is important to understand how migration projects materialize. In this study, we ask 54 practitioners about their views on specific aspects of microservices migrations. Specifically, we derive the sequence of migration activities as well as the perceived impact of microservices on the development process. Database refactoring, back-end refactoring, and setting up DevOps tend to take place before front-end refactoring, setting up communication patterns, and splitting teams. In addition, aligning teams with profitable value propositions, improving the testing process, and having fewer dependencies between teams are among the prominent impact areas of microservices. Our results call for further empirical research in understanding transitions toward MSAs.

   @inproceedings{ayas:23:2,
     author = {Ayas, H. and Leitner, P. and Hebig, R.},
     booktitle = {2023 IEEE International Conference on Service-Oriented System Engineering (SOSE)},
     title = {The Perceived Impact and Sequence of Activities When Transitioning to Microservices},
     year = {2023},
     volume = {},
     issn = {},
     pages = {156-160},
     keywords = {surveys;service-oriented systems engineering;databases;microservice architectures;organizations;software systems;testing},
     doi = {10.1109/SOSE58276.2023.00026},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = jul,
     note = {(short paper)},
   }

3. BotSE @ ICSE 2023

   

   Towards Continuous Performance Assessment of Java Applications With PerfBot

   Florian Markusse, Alexander Serebrenik, and Philipp Leitner

   In 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE) , Jul 2023

   Bib

   @inproceedings{markusse:23,
     author = {Markusse, Florian and Serebrenik, Alexander and Leitner, Philipp},
     booktitle = {2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE)},
     title = {Towards Continuous Performance Assessment of Java Applications With PerfBot},
     year = {2023},
     volume = {},
     number = {},
     pages = {6-8},
     keywords = {Java;Conferences;Software performance;Benchmark testing;Chatbots;Task analysis;Software engineering},
     doi = {10.1109/BotSE59190.2023.00009},
   }

4. EMSE

   

   An empirical study of the systemic and technical migration towards microservices

   Hamdy Michael Ayas, Philipp Leitner, and Regina Hebig

   Empirical Software Engineering, May 2023

   Bib

   @article{ayas:23,
     author = {Michael Ayas, Hamdy and Leitner, Philipp and Hebig, Regina},
     title = {An empirical study of the systemic and technical migration towards microservices},
     year = {2023},
     issue_date = {Jul 2023},
     publisher = {Kluwer Academic Publishers},
     address = {USA},
     volume = {28},
     number = {4},
     issn = {1382-3256},
     doi = {10.1007/s10664-023-10308-9},
     journal = {Empirical Software Engineering},
     month = may,
     numpages = {50},
     keywords = {Microservices, Migrations, Grounded theory, Empirical research},
   }

5. TCC

   

   Using Microbenchmark Suites to Detect Application Performance Changes

   M. Grambow, D. Kovalev, C. Laaber, P. Leitner, and D. Bermbach

   IEEE Transactions on Cloud Computing, Jul 2023

   Abs Bib PDF

   Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of the two time-series database systems InfluxDB and VictoriaMetrics and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is possible to detect application performance changes using an optimized microbenchmark suite if frequent false-positive alarms can be tolerated.

   @article{grambow:22,
     author = {Grambow, M. and Kovalev, D. and Laaber, C. and Leitner, P. and Bermbach, D.},
     journal = {IEEE Transactions on Cloud Computing},
     title = {Using Microbenchmark Suites to Detect Application Performance Changes},
     year = {2023},
     volume = {11},
     number = {03},
     issn = {2168-7161},
     pages = {2575-2590},
     keywords = {benchmark testing;codes;cloud computing;reliability;software;pipelines;mimics},
     doi = {10.1109/TCC.2022.3217947},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = jul,
   }

6. TSE

   

   Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit Tests

   Mostafa Jangali, Yiming Tang, Niclas Alexandersson, Philipp Leitner, Jinqiu Yang, and Weiyi Shang

   IEEE Transactions on Software Engineering, Apr 2023

   Abs Bib PDF

   Performance is a crucial non-functional requirement of many software systems. Despite the widespread use of performance testing, developers still struggle to construct and evaluate the quality of performance tests. To address these two major challenges, we implement a framework, dubbed ju2jmh, to automatically generate performance microbenchmarks from JUnit tests and use mutation testing to study the quality of generated microbenchmarks. Specifically, we compare our ju2jmh generated benchmarks to manually written JMH benchmarks and to automatically generated JMH benchmarks using the AutoJMH framework, as well as directly measuring system performance with JUnit tests. For this purpose, we have conducted a study on three subjects (Rxjava, Eclipse-collections, and Zipkin) with sim∼454K source lines of code (SLOC), 2,417 JMH benchmarks (including manually written and generated AutoJMH benchmarks) and 35,084 JUnit tests. Our results show that the ju2jmh generated JMH benchmarks consistently outperform using the execution time and throughput of JUnit tests as a proxy of performance and JMH benchmarks automatically generated using the AutoJMH framework while being comparable to JMH benchmarks manually written by developers in terms of tests’ stability and ability to detect performance bugs. Nevertheless, ju2jmh benchmarks are able to cover more of the software applications than manually written JMH benchmarks during the microbenchmark execution. Furthermore, ju2jmh benchmarks are generated automatically, while manually written JMH benchmarks require many hours of hard work and attention; therefore our study can reduce developers’ effort to construct microbenchmarks. In addition, we identify three factors (too low test workload, unstable tests and limited mutant coverage) that affect a benchmark’s ability to detect performance bugs. To the best of our knowledge, this is the first study aimed at assisting developers in fully automated microbenchmark creation and assessing microbenchmark quality for performance testing.

   @article{jangali:22,
     author = {Jangali, Mostafa and Tang, Yiming and Alexandersson, Niclas and Leitner, Philipp and Yang, Jinqiu and Shang, Weiyi},
     title = {Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit Tests},
     year = {2023},
     issue_date = {April 2023},
     publisher = {IEEE Press},
     volume = {49},
     number = {4},
     issn = {0098-5589},
     doi = {10.1109/TSE.2022.3188005},
     journal = {IEEE Transactions on Software Engineering},
     month = apr,
     pages = {1704–1725},
     numpages = {22},
   }

2022

1. UCC 2022

   

   CrossFit: Fine-grained Benchmarking of Serverless Application Performance across Cloud Providers

   Joel Scheuner, Rui Deng, Jan-Philipp Steghöfer, and Philipp Leitner

   In 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing (UCC) , Apr 2022

   Bib PDF

   @inproceedings{scheuner:22:2,
     author = {Scheuner, Joel and Deng, Rui and Steghöfer, Jan-Philipp and Leitner, Philipp},
     booktitle = {2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing (UCC)},
     title = {CrossFit: Fine-grained Benchmarking of Serverless Application Performance across Cloud Providers},
     year = {2022},
     volume = {},
     number = {},
     pages = {51-60},
     keywords = {Performance evaluation;Cloud computing;Runtime;Instruments;Serverless computing;Tail;Standardization;serverless;FaaS;distributed tracing;observability;performance;benchmarking},
     doi = {10.1109/UCC56403.2022.00016},
   }

2. PROFES 2022

   

   TEP-GNN: Accurate Execution Time Prediction of Functional Tests Using Graph Neural Networks

   Hazem Peter Samoaa, Antonio Longa, Mazen Mohamad, Morteza Haghir Chehreghani, and Philipp Leitner

   In Product-Focused Software Process Improvement , Apr 2022

   Abs Bib PDF

   Predicting the performance of production code prior to actual execution is known to be highly challenging. In this paper, we propose a predictive model, dubbed  TEP-GNN, which demonstrates that high-accuracy performance prediction is possible for the special case of predicting unit test execution times.  TEP-GNN uses FA-ASTs, or flow-augmented ASTs, as a graph-based code representation approach, and predicts test execution times using a powerful graph neural network (GNN) deep learning model. We evaluate  TEP-GNN using four real-life Java open source programs, based on 922 test files mined from the projects’ public repositories. We find that our approach achieves a high Pearson correlation of 0.789, considerable outperforming a baseline deep learning model. Our work demonstrates that FA-ASTs and GNNs are a feasible approach for predicting absolute performance values, and serves as an important intermediary step towards being able to predict the performance of arbitrary code prior to execution.

   @inproceedings{samoaa:22:1,
     author = {Samoaa, Hazem Peter and Longa, Antonio and Mohamad, Mazen and Chehreghani, Morteza Haghir and Leitner, Philipp},
     editor = {Taibi, Davide and Kuhrmann, Marco and Mikkonen, Tommi and Klünder, Jil and Abrahamsson, Pekka},
     title = {TEP-GNN: Accurate Execution Time Prediction of Functional Tests Using Graph Neural Networks},
     booktitle = {Product-Focused Software Process Improvement},
     year = {2022},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {464--479},
     isbn = {978-3-031-21388-5},
   }

3. IC2E 2022

   

   TriggerBench: A Performance Benchmark for Serverless Function Triggers

   Joel Scheuner, Marcus Bertilsson, Oskar Grönqvist, Henrik Tao, Henrik Lagergren, Jan-Philipp Steghöfer, and Philipp Leitner

   In 2022 IEEE International Conference on Cloud Engineering (IC2E) , Apr 2022

   Bib PDF

   @inproceedings{scheuner:22,
     author = {Scheuner, Joel and Bertilsson, Marcus and Grönqvist, Oskar and Tao, Henrik and Lagergren, Henrik and Steghöfer, Jan-Philipp and Leitner, Philipp},
     booktitle = {2022 IEEE International Conference on Cloud Engineering (IC2E)},
     title = {TriggerBench: A Performance Benchmark for Serverless Function Triggers},
     year = {2022},
     volume = {},
     number = {},
     pages = {96-103},
     keywords = {Runtime;Scalability;Buildings;Serverless computing;Tail;Benchmark testing;Delays;serverless;FaaS;triggers;distributed tracing;observability;performance;benchmarking},
     doi = {10.1109/IC2E55432.2022.00018},
   }

4. SEAA 2022

   

   An Empirical Analysis of Microservices Systems Using Consumer-Driven Contract Testing

   Hamdy Michael Ayas, Hartmut Fischer, Philipp Leitner, and Francisco Gomes De Oliveira Neto

   In 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , Apr 2022

   Bib

   @inproceedings{ayas:22,
     author = {Ayas, Hamdy Michael and Fischer, Hartmut and Leitner, Philipp and De Oliveira Neto, Francisco Gomes},
     booktitle = {2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)},
     title = {An Empirical Analysis of Microservices Systems Using Consumer-Driven Contract Testing},
     year = {2022},
     volume = {},
     number = {},
     pages = {92-99},
     keywords = {Shape;Microservice architectures;Computer architecture;Software;Best practices;Contracts;Testing;Microservice architecture;consumer-driven contract test;repository mining},
     doi = {10.1109/SEAA56994.2022.00022},
   }

5. IEEE Software

   

   Using Benchmarking Bots for Continuous Performance Assessment

   Florian Markusse, Philipp Leitner, and Alexander Serebrenik

   IEEE Software, Apr 2022

   Bib PDF

   @article{markusse:22,
     author = {Markusse, Florian and Leitner, Philipp and Serebrenik, Alexander},
     journal = {IEEE Software},
     title = {Using Benchmarking Bots for Continuous Performance Assessment},
     year = {2022},
     volume = {39},
     number = {5},
     pages = {50-55},
     keywords = {Benchmark testing;Bot (Internet);Software;Codes;Software development management;Manuals;Open source software},
     doi = {10.1109/MS.2022.3184430},
   }

6. IET Software

   

   A systematic mapping study of source code representation for deep learning in software engineering

   Hazem Peter Samoaa, Firas Bayram, Pasquale Salza, and Philipp Leitner

   IET Software, Apr 2022

   Abs Bib

   The usage of deep learning (DL) approaches for software engineering has attracted much attention, particularly in source code modelling and analysis. However, in order to use DL, source code needs to be formatted to fit the expected input form of DL models. This problem is known as source code representation. Source code can be represented via different approaches, most importantly, the tree-based, token-based, and graph-based approaches. We use a systematic mapping study to investigate i detail the representation approaches adopted in 103 studies that use DL in the context of software engineering. Thus, studies are collected from 2014 to 2021 from 14 different journals and 27 conferences. We show that each way of representing source code can provide a different, yet orthogonal view of the same source code. Thus, different software engineering tasks might require different (combinations of) code representation approaches, depending on the nature and complexity of the task. Particularly, we show that it is crucial to define whether the DL approach requires lexical, syntactical, or semantic code information. Our analysis shows that a wide range of different representations and combinations of representations (hybrid representations) are used to solve a wide range of common software engineering problems. However, we also observe that current research does not generally attempt to transfer existing representations or models to other studies even though there are other contexts in which these representations and models may also be useful. We believe that there is potential for more reuse and the application of transfer learning when applying DL to software engineering tasks.

   @article{samoaa:22,
     author = {Samoaa, Hazem Peter and Bayram, Firas and Salza, Pasquale and Leitner, Philipp},
     title = {A systematic mapping study of source code representation for deep learning in software engineering},
     journal = {IET Software},
     volume = {16},
     number = {4},
     pages = {351-385},
     doi = {https://doi.org/10.1049/sfw2.12064},
     eprint = {https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/sfw2.12064},
     year = {2022},
   }

7. TSE

   

   The State of Serverless Applications: Collection, Characterization, and Community Consensus

   Simon Eismann, Joel Scheuner, Erwin van Eyk, Maximilian Schwinger, Johannes Grohmann, Nikolas Herbst, Cristina L. Abad, and Alexandru Iosup

   IEEE Transactions on Software Engineering, Oct 2022

   Abs Bib Supp

   Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.

   @article{10.1109/TSE.2021.3113940,
     author = {Eismann, Simon and Scheuner, Joel and Eyk, Erwin van and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina L. and Iosup, Alexandru},
     title = {The State of Serverless Applications: Collection, Characterization, and Community Consensus},
     year = {2022},
     issue_date = {Oct. 2022},
     publisher = {IEEE Press},
     volume = {48},
     number = {10},
     issn = {0098-5589},
     doi = {10.1109/TSE.2021.3113940},
     journal = {IEEE Transactions on Software Engineering},
     month = oct,
     pages = {4152–4166},
     numpages = {15},
   }

2021

1. PeerJ CS

   

   Dependency Management Bots in Open-Source Systems - Prevalence and Adoption

   Linda Erlenhov, Francisco Gomes Oliveira Neto, and Philipp Leitner

   PeerJ Computer Science, Oct 2021

   Bib Supp

   @article{erlenhov:22,
     title = {{Dependency Management Bots in Open-Source Systems - Prevalence and Adoption}},
     journal = {PeerJ Computer Science},
     year = {2021},
     author = {Erlenhov, Linda and de Oliveira Neto, Francisco Gomes and Leitner, Philipp},
     volume = {8},
     pages = {e849},
     doi = {https://doi.org/10.7717/peerj-cs.849},
   }

2. ESEM 2021

   

   Facing the Giant: a Grounded Theory Study of Decision-Making in Microservices Migrations

   Hamdy Michael Ayas, Philipp Leitner, and Regina Hebig

   In Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , Bari, Italy, Oct 2021

   Abs Bib

   Background: Microservices migrations are challenging and expensive projects with many decisions that need to be made in a multitude of dimensions. Existing research tends to focus on technical issues and decisions (e.g., how to split services). Equally important organizational or business issues and their relations with technical aspects often remain out of scope or on a high level of abstraction.Aims: In this study, we aim to holistically chart the decision-making that happens on all dimensions of a migration project towards microservices (including, but not limited to, the technical dimension).Method: We investigate 16 different migration cases in a grounded theory interview study, with 19 participants that recently migrated towards microservices. This study strongly focuses on the human aspects of a migration, through stakeholders and their decisions.Results: We identify 3 decision-making processes consisting of 22 decision-points and their alternative options. The decision-points are related to creating stakeholder engagement and assessing feasibility, technical implementation, and organizational restructuring.Conclusions: Our study provides an initial theory of decisionmaking in migrations to microservices. It also outfits practitioners with a roadmap of which decisions they should be prepared to make and at which point in the migration.

   @inproceedings{ayas:21:2,
     author = {Ayas, Hamdy Michael and Leitner, Philipp and Hebig, Regina},
     title = {Facing the Giant: a Grounded Theory Study of Decision-Making in Microservices Migrations},
     year = {2021},
     isbn = {9781450386654},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3475716.3475792},
     booktitle = {Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)},
     articleno = {16},
     numpages = {11},
     keywords = {migration, microservices, interview study, grounded theory, decision-making},
     location = {Bari, Italy},
     series = {ESEM '21},
   }

3. EMSE

   

   Applying test case prioritization to software microbenchmarks

   Christoph Laaber, Harald C. Gall, and Philipp Leitner

   Empirical Software Engineering, Oct 2021

   Bib Supp

   @article{laaber:21,
     author = {Laaber, Christoph and Gall, Harald C. and Leitner, Philipp},
     title = {Applying test case prioritization to software microbenchmarks},
     journal = {Empirical Software Engineering},
     volume = {26},
     number = {6},
     pages = {133},
     year = {2021},
     doi = {10.1007/s10664-021-10037-x},
   }

4. PeerJ CS

   

   Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

   Martin Grambow, Christoph Laaber, Philipp Leitner, and David Bermbach

   PeerJ Computer Science, Oct 2021

   Bib

   @article{grambow:21,
     author = {Grambow, Martin and Laaber, Christoph and Leitner, Philipp and Bermbach, David},
     title = {Using application benchmark call graphs to quantify and improve the
                    practical relevance of microbenchmark suites},
     journal = {PeerJ Computer Science},
     volume = {7},
     pages = {e548},
     year = {2021},
     doi = {10.7717/peerj-cs.548},
   }

5. SOFSEM 2021

   

   A Pipeline for Measuring Brand Loyalty Through Social Media Mining

   Hazem Samoaa, and Barbara Catania

   In SOFSEM 2021: Theory and Practice of Computer Science , Oct 2021

   Abs Bib

   Enhancing customer relationships through social media is an area of high relevance for companies. To this aim, Social Business Intelligence (SBI) plays a crucial role by supporting companies in combining corporate data with user-generated content, usually available as textual clips on social media. Unfortunately, SBI research is often constrained by the lack of publicly-available, real-world data for experimental activities. In this paper, we describe our experience in extracting social data and processing them through an enrichment pipeline for brand analysis. As a first step, we collect texts from social media and we annotate them based on predefined metrics for brand analysis, using features such as sentiment and geolocation. Annotations rely on various learning and natural language processing approaches, including deep learning and geographical ontologies. Structured data obtained from the annotation process are then stored in a distributed data warehouse for further analysis. Preliminary results, obtained from the analysis of three well known ICT brands, using data gathered from Twitter, news portals, and Amazon product reviews, show that different evaluation metrics can lead to different outcomes, indicating that no single metric is dominant for all brand analysis use cases.

   @inproceedings{samoaa:21,
     author = {Samoaa, Hazem and Catania, Barbara},
     title = {A Pipeline for Measuring Brand Loyalty Through Social Media Mining},
     booktitle = {SOFSEM 2021: Theory and Practice of Computer Science},
     year = {2021},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {489--504},
     isbn = {978-3-030-67731-2},
   }

6. ICPE 2021

   

   An Exploratory Study of the Impact of Parameterization on JMH Measurement Results in Open-Source Projects

   Hazem Samoaa, and Philipp Leitner

   In ICPE ’21: ACM/SPEC International Conference on Performance Engineering, Virtual Event, France, April 19-21, 2021 , Oct 2021

   Bib

   @inproceedings{samoaa:21:2,
     author = {Samoaa, Hazem and Leitner, Philipp},
     editor = {Bourcier, Johann and Jiang, Zhen Ming (Jack) and Bezemer, Cor{-}Paul and Cortellessa, Vittorio and Pompeo, Daniele Di and Varbanescu, Ana Lucia},
     title = {An Exploratory Study of the Impact of Parameterization on {JMH} Measurement
                    Results in Open-Source Projects},
     booktitle = {{ICPE} '21: {ACM/SPEC} International Conference on Performance Engineering,
                    Virtual Event, France, April 19-21, 2021},
     pages = {213--224},
     publisher = {{ACM}},
     year = {2021},
     doi = {10.1145/3427921.3450243},
   }

7. IEEE Software

   

   Serverless Applications: Why, When, and How?

   Simon Eismann, Joel Scheuner, Erwin Eyk, Maximilian Schwinger, Johannes Grohmann, Nikolas Herbst, Cristina L. Abad, and Alexandru Iosup

   IEEE Software, Oct 2021

   Bib

   @article{eismann:21,
     author = {Eismann, Simon and Scheuner, Joel and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina L. and Iosup, Alexandru},
     journal = {IEEE Software},
     title = {Serverless Applications: Why, When, and How?},
     year = {2021},
     volume = {38},
     number = {1},
     pages = {32-39},
     doi = {10.1109/MS.2020.3023302},
   }

8. TSE

   

   What’s Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks

   Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak

   IEEE Transactions on Software Engineering, Oct 2021

   Bib

   @article{costa:19,
     author = {Costa, Diego and Bezemer, Cor-Paul and Leitner, Philipp and Andrzejak, Artur},
     journal = {IEEE Transactions on Software Engineering},
     title = {What's Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks},
     year = {2021},
     volume = {47},
     number = {7},
     pages = {1452-1467},
     doi = {10.1109/TSE.2019.2925345},
   }

9. TSE

   

   Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

   Giovanni Grano, Christoph Laaber, Annibale Panichella, and Sebastiano Panichella

   IEEE Transactions on Software Engineering, Oct 2021

   Bib

   @article{grano:19,
     author = {Grano, Giovanni and Laaber, Christoph and Panichella, Annibale and Panichella, Sebastiano},
     title = {Testing with Fewer Resources: An Adaptive Approach to Performance-Aware
                    Test Case Generation},
     journal = {{IEEE} Transactions on Software Engineering},
     volume = {47},
     number = {11},
     pages = {2332--2347},
     year = {2021},
     doi = {10.1109/TSE.2019.2946773},
   }

2020

1. Topology-Aware Continuous Experimentation in Microservice-Based Applications

   Gerald Schermann, Fábio Oliveira, Erik Wittern, and Philipp Leitner

   In Service-Oriented Computing - 18th International Conference, ICSOC 2020, Dubai, United Arab Emirates, December 14-17, 2020, Proceedings , Oct 2020

   Bib

   @inproceedings{schermann:20,
     author = {Schermann, Gerald and Oliveira, F{\'{a}}bio and Wittern, Erik and Leitner, Philipp},
     editor = {Kafeza, Eleanna and Benatallah, Boualem and Martinelli, Fabio and Hacid, Hakim and Bouguettaya, Athman and Motahari, Hamid},
     title = {Topology-Aware Continuous Experimentation in Microservice-Based Applications},
     booktitle = {Service-Oriented Computing - 18th International Conference, {ICSOC}
                    2020, Dubai, United Arab Emirates, December 14-17, 2020, Proceedings},
     series = {Lecture Notes in Computer Science},
     volume = {12571},
     pages = {19--35},
     publisher = {Springer},
     year = {2020},
     doi = {10.1007/978-3-030-65310-1\_2}
   }

2. Function-as-a-Service performance evaluation: A multivocal literature review

   Joel Scheuner, and Philipp Leitner

   Journal of Systems and Software, Oct 2020

   Bib

   @article{scheuner:20,
     author = {Scheuner, Joel and Leitner, Philipp},
     title = {Function-as-a-Service performance evaluation: {A} multivocal literature
                    review},
     journal = {Journal of Systems and Software},
     volume = {170},
     pages = {110708},
     year = {2020},
     doi = {10.1016/j.jss.2020.110708}
   }

3. FSE 2020

   

   Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality

   Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner

   In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , Virtual Event, USA, Oct 2020

   Bib

   @inproceedings{laaber:20,
     author = {Laaber, Christoph and W\"{u}rsten, Stefan and Gall, Harald C. and Leitner, Philipp},
     title = {Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality},
     year = {2020},
     isbn = {9781450370431},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3368089.3409683},
     booktitle = {Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
     pages = {989–1001},
     numpages = {13},
     keywords = {JMH, configuration, software benchmarking, performance testing},
     location = {Virtual Event, USA},
     series = {ESEC/FSE 2020},
   }

4. An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective

   Linda Erlenhov, Francisco Gomes de Oliveira Neto, and Philipp Leitner

   In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , Virtual Event, USA, Oct 2020

   Bib

   @inproceedings{erlenhov:20,
     author = {Erlenhov, Linda and Neto, Francisco Gomes de Oliveira and Leitner, Philipp},
     title = {An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective},
     year = {2020},
     isbn = {9781450370431},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3368089.3409680},
     booktitle = {Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
     pages = {445–455},
     numpages = {11},
     keywords = {Software engineering, Software bot, Empirical study},
     location = {Virtual Event, USA},
     series = {ESEC/FSE 2020}
   }

5. Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark

   Erwin Eyk, Joel Scheuner, Simon Eismann, Cristina L. Abad, and Alexandru Iosup

   In Companion of the ACM/SPEC International Conference on Performance Engineering , Apr 2020

   Bib

   @inproceedings{eyk:20,
     author = {van Eyk, Erwin and Scheuner, Joel and Eismann, Simon and Abad, Cristina L. and Iosup, Alexandru},
     booktitle = {Companion of the ACM/SPEC International Conference on
     Performance Engineering},
     doi = {10.1145/3375555.3384381},
     month = apr,
     pages = {26--31},
     publisher = {ACM/SPEC},
     series = {ICPE '20},
     title = {Beyond Microbenchmarks: The SPEC-RG Vision for a
     Comprehensive Serverless Benchmark},
     year = {2020}
   }

2019

1. Studying the impact of CI on pull request delivery time in open source projects — a conceptual replication

   Yunfang Guo, and Philipp Leitner

   PeerJ Computer Science, Apr 2019

   Bib

   @article{guo:19,
     title = {{Studying the impact of CI on pull request delivery time in open source projects — a conceptual replication}},
     journal = {PeerJ Computer Science},
     year = {2019},
     author = {Guo, Yunfang and Leitner, Philipp},
     doi = {10.7717/peerj-cs.245}
   }

2. Transpiling Applications into Optimized Serverless Orchestrations

   Joel Scheuner, and Philipp Leitner

   In 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) , Jun 2019

   2nd Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf)

   Bib

   @inproceedings{scheuner:19:3,
     author = {Scheuner, Joel and Leitner, Philipp},
     booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)},
     doi = {10.1109/FAS-W.2019.00031},
     month = jun,
     pages = {72--73},
     publisher = {IEEE},
     title = {Transpiling Applications into Optimized Serverless Orchestrations},
     year = {2019},
     note = {2nd Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf)}
   }

3. Tutorial – Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench

   Joel Scheuner, and Philipp Leitner

   In 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) , Jun 2019

   Tutorial co-located with ICAC/SASO

   Bib

   @inproceedings{scheuner:19:2,
     author = {Scheuner, Joel and Leitner, Philipp},
     booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)},
     doi = {10.1109/FAS-W.2019.00070},
     month = jun,
     pages = {257--258},
     publisher = {IEEE},
     title = {Tutorial -- Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench},
     year = {2019},
     note = {Tutorial co-located with ICAC/SASO}
   }

4. Current and Future Bots in Software Development

   Linda Erlenhov, Francisco Gomes, Riccardo Scandariato, and Philipp Leitner

   In Proceedings of the 1st International Workshop on Bots in Software Engineering (BotSE) at ICSE 2019 , Jun 2019

   Bib

   @inproceedings{erlenhov:19,
     title = {{Current and Future Bots in Software Development}},
     booktitle = {{Proceedings of the 1st International Workshop on Bots in Software
                    Engineering (BotSE) at ICSE 2019}},
     author = {Erlenhov, Linda and Gomes, Francisco and Scandariato, Riccardo and Leitner, Philipp},
     pages = {7--11},
     publisher = {{IEEE} / {ACM}},
     year = {2019},
     doi = {10.1109/BotSE.2019.00009}
   }

5. Cloud Futurology

   Blesson Varghese, Philipp Leitner, Suprio Ray, Kyle Chard, Adam Barker, Yehia Elkhatib, Herry Herry, Cheol-Ho Hong, Jeremy Singer, Fung Po Tso, Eiko Yoneki, and  Mohamed-Faten Zhani

   IEEE Computer, Jun 2019

   Bib

   @article{varghese:19,
     title = {{Cloud Futurology}},
     journal = {{IEEE} Computer},
     author = {Varghese, Blesson and Leitner, Philipp and Ray, Suprio and Chard, Kyle and Barker, Adam and Elkhatib, Yehia and Herry, Herry and Hong, Cheol-Ho and Singer, Jeremy and Tso, Fung Po and Yoneki, Eiko and and Mohamed-Faten Zhani},
     volume = {52},
     number = {9},
     pages = {68--77},
     year = {2019},
     doi = {10.1109/MC.2019.2895307}
   }

6. Tutorial: Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench

   Joel Scheuner, and Philipp Leitner

   In Companion Proceedings of the 10th ACM/SPEC International Conference on Performance Engineering (ICPE) , Mumbai, India, Jun 2019

   Bib

   @inproceedings{scheuner:19,
     title = {{Tutorial: Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench}},
     booktitle = {Companion Proceedings of the 10th ACM/SPEC International Conference on Performance Engineering (ICPE)},
     year = {2019},
     author = {Scheuner, Joel and Leitner, Philipp},
     doi = {10.1145/3302541.3310294},
     keywords = {benchmarking, cloud computing, performance},
     isbn = {978-1-4503-6286-3},
     location = {Mumbai, India},
     pages = {53--56}
   }

7. 

   Software microbenchmarking in the cloud. How bad is it really?

   Christoph Laaber, Joel Scheuner, and Philipp Leitner

   Empirical Software Engineering, Apr 2019

   Abs Bib

   Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to >\thinspace100%). However, executing test and control experiments on the same instances (in randomized order) allows us to detect slowdowns of 10% or less with high confidence, using state-of-the-art statistical tests (i.e., Wilcoxon rank-sum and overlapping bootstrapped confidence intervals). Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns in cloud environments.

   @article{laaber:19a,
     author = {Laaber, Christoph and Scheuner, Joel and Leitner, Philipp},
     title = {Software microbenchmarking in the cloud. How bad is it really?},
     journal = {Empirical Software Engineering},
     year = {2019},
     month = apr,
     day = {17},
     issn = {1573-7616},
     doi = {10.1007/s10664-019-09681-1},
     publisher = {Springer},
     keywords = {performance testing, microbenchmarking, cloud, performance-regression detection},
     numpages = {40},
     abbrev = {EMSE},
   }

8. Continuous Software Performance Assessment: Detecting Performance Problems of Software Libraries on Every Build

   Christoph Laaber

   In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , Beijing, China, Apr 2019

   Bib

   @inproceedings{laaber:19b,
     author = {Laaber, Christoph},
     title = {Continuous Software Performance Assessment: Detecting Performance Problems of Software Libraries on Every Build},
     booktitle = {Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis},
     series = {ISSTA 2019},
     year = {2019},
     isbn = {978-1-4503-6224-5},
     location = {Beijing, China},
     pages = {410--414},
     numpages = {5},
     doi = {10.1145/3293882.3338982},
     acmid = {3338982},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {cloud, continuous integration, performance testing}
   }

9. A Mixed-Method Empirical Study of Function-as-a-Service Software Development in Industrial Practice

   Philipp Leitner, Erik Wittern, Josef Spillner, and Waldemar Hummer

   Journal of Systems and Software, Apr 2019

   Bib

   @article{leitner:18,
     title = {{A Mixed-Method Empirical Study of Function-as-a-Service Software Development in Industrial Practice}},
     journal = {Journal of Systems and Software},
     volume = {149},
     pages = {340 - 359},
     year = {2019},
     issn = {0164-1212},
     doi = {https://doi.org/10.1016/j.jss.2018.12.013},
     author = {Leitner, Philipp and Wittern, Erik and Spillner, Josef and Hummer, Waldemar},
     keywords = {Cloud computing, Serverless, Function-as-a-Service, Empirical research}
   }

10. ICSE 2019

    

    Interactive Production Performance Feedback in the IDE

    Jürgen Cito, Philipp Leitner, Martin Rinard, and Harald Gall

    In Proceedings of the 41st International Conference on Software Engineering (ICSE) , Apr 2019

    Bib PDF

    @inproceedings{cito:19,
      author = {Cito, Jürgen and Leitner, Philipp and Rinard, Martin and Gall, Harald},
      title = {Interactive Production Performance Feedback in the IDE},
      booktitle = {{Proceedings of the 41st International Conference on Software Engineering (ICSE)}},
      pages = {971--981},
      publisher = {{IEEE} / {ACM}},
      year = {2019},
      doi = {10.1109/ICSE.2019.00102},
    }

11. Cachematic -– Automatic Invalidation in Application-Level Caching Systems

    Viktor Holmqvist, Jonathan Nilsfors, and Philipp Leitner

    In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (ICPE) , Apr 2019

    Bib

    @inproceedings{holmqvist:19,
      author = {Holmqvist, Viktor and Nilsfors, Jonathan and Leitner, Philipp},
      title = {Cachematic -– Automatic Invalidation in Application-Level Caching Systems},
      booktitle = {Proceedings of the 2019 {ACM/SPEC} International Conference on Performance
                      Engineering {(ICPE)}},
      pages = {167--178},
      publisher = {{ACM}},
      year = {2019},
    }

2018

1. Visualizing Test Diversity to Support Test Optimisation

   F. G. De Oliveira Neto, R. Feldt, L. Erlenhov, and J. B. D. S. Nunes

   In 2018 25th Asia-Pacific Software Engineering Conference (APSEC) , Dec 2018

   Bib

   @inproceedings{neto:18,
     author = {{De Oliveira Neto}, F. G. and {Feldt}, R. and {Erlenhov}, L. and {Nunes}, J. B. D. S.},
     booktitle = {2018 25th Asia-Pacific Software Engineering Conference (APSEC)},
     title = {Visualizing Test Diversity to Support Test Optimisation},
     year = {2018},
     volume = {},
     number = {},
     pages = {149-158},
     keywords = {data visualisation;fault diagnosis;program testing;test optimisation;test suites;cost-effective testing;test cases;diversity information;diversity-based approaches;test requirements;test scripts;test diversity data;test similarity maps;pair-wise diversity calculations;test repositories;maintenance;optimisation activities;Optimization;Testing;Indexes;Data visualization;Fault detection;Maintenance engineering;Subspace constraints;Software Testing;Diversity;Search based Software Testing;Empirical Study},
     doi = {10.1109/APSEC.2018.00029},
     issn = {},
     month = dec,
   }

2. Structured Information on State and Evolution of Dockerfiles on Github

   Gerald Schermann, Sali Zumberi, and Jürgen Cito

   In Proceedings of the 15th International Conference on Mining Software Repositories , Gothenburg, Sweden, Dec 2018

   Bib

   @inproceedings{schermann:18:4,
     author = {Schermann, Gerald and Zumberi, Sali and Cito, J\"{u}rgen},
     title = {Structured Information on State and Evolution of Dockerfiles on Github},
     booktitle = {Proceedings of the 15th International Conference on Mining Software Repositories},
     series = {MSR '18},
     year = {2018},
     isbn = {978-1-4503-5716-6},
     location = {Gothenburg, Sweden},
     pages = {26--29},
     numpages = {4},
     doi = {10.1145/3196398.3196456},
     acmid = {3196456},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {containers, docker, github, mining software repositories}
   }

3. Search-Based Scheduling of Experiments in Continuous Deployment

   G. Schermann, and P. Leitner

   In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) , Sep 2018

   Bib

   @inproceedings{schermann:18:3,
     author = {Schermann, G. and Leitner, P.},
     booktitle = {2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)},
     title = {Search-Based Scheduling of Experiments in Continuous Deployment},
     year = {2018},
     pages = {485-495},
     keywords = {genetic algorithms;parallel processing;scheduling;search problems;continuous deployment;production environments;user assignments;optimization problem;parallel execution;genetic algorithm;local search;genetic implementation;search-based scheduling;Schedules;Companies;Testing;Google;Genetic algorithms;Simulated annealing;continuous experimentation;search based software engineering;scheduling;release engineering},
     doi = {10.1109/ICSME.2018.00059},
     issn = {2576-3148},
     month = sep
   }

4. Estimating Cloud Application Performance Based on Micro-Benchmark Profiling

   Joel Scheuner, and Philipp Leitner

   In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD) , Jul 2018

   Bib

   @inproceedings{scheuner:18:2,
     author = {Scheuner, Joel and Leitner, Philipp},
     booktitle = {2018 IEEE 11th International Conference on Cloud Computing (CLOUD)},
     title = {Estimating Cloud Application Performance Based on Micro-Benchmark Profiling},
     year = {2018},
     pages = {90-97},
     keywords = {benchmark testing;cloud computing;Amazon EC2;Web serving application;real-world applications;synthetic microbenchmarks;service selection;cloud computing market;microbenchmark profiling;cloud application performance;scientific computing application;cloud service performance;cloud provider;cloud benchmarking methodology;Benchmark testing;Cloud computing;Measurement;Random access memory;Computational modeling;Time factors;Predictive models;cloud computing;benchmarking;performance;micro benchmark;application benchmark;Web application;performance prediction},
     doi = {10.1109/CLOUD.2018.00019},
     issn = {2159-6190},
     month = jul
   }

5. An Evaluation of Open-source Software Microbenchmark Suites for Continuous Performance Assessment

   Christoph Laaber, and Philipp Leitner

   In Proceedings of the 15th International Conference on Mining Software Repositories , Gothenburg, Sweden, Jul 2018

   Bib

   @inproceedings{laaber:18,
     author = {Laaber, Christoph and Leitner, Philipp},
     title = {An Evaluation of Open-source Software Microbenchmark Suites for Continuous Performance Assessment},
     booktitle = {Proceedings of the 15th International Conference on Mining Software Repositories},
     series = {MSR '18},
     year = {2018},
     isbn = {978-1-4503-5716-6},
     location = {Gothenburg, Sweden},
     pages = {119--130},
     numpages = {12},
     doi = {10.1145/3196398.3196407},
     acmid = {3196407},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {Go, Java, continuous integration, empirical study, microbenchmarking, software performance testing}
   }

6. We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation

   Gerald Schermann, Jürgen Cito, Philipp Leitner, Uwe Zdun, and Harald C. Gall

   Journal of Information and Software Technology, Jul 2018

   Bib

   @article{schermann:18:2,
     author = {Schermann, Gerald and Cito, Jürgen and Leitner, Philipp and Zdun, Uwe and Gall, Harald C.},
     title = {{We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation}},
     journal = {{Journal of Information and Software Technology}},
     year = {2018},
     volume = { 99 },
     pages = {41--57},
     doi = {https://doi.org/10.1016/j.infsof.2018.02.010},
   }

7. A Cloud Benchmark Suite Combining Micro and Applications Benchmarks

   Joel Scheuner, and Philipp Leitner

   In Companion of the 2018 ACM/SPEC International Conference on Performance Engineering , Berlin, Germany, Jul 2018

   Bib

   @inproceedings{scheuner:18,
     author = {Scheuner, Joel and Leitner, Philipp},
     title = {A Cloud Benchmark Suite Combining Micro and Applications Benchmarks},
     booktitle = {Companion of the 2018 ACM/SPEC International Conference on Performance Engineering},
     series = {ICPE '18},
     year = {2018},
     isbn = {978-1-4503-5629-9},
     location = {Berlin, Germany},
     pages = {161--166},
     numpages = {6},
     doi = {10.1145/3185768.3186286},
     acmid = {3186286},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {application benchmark, benchmarking, cloud computing, micro benchmark, performance, web application}
   }

8. PerformanceHat: Augmenting Source Code with Runtime Performance Traces in the IDE

   Jürgen Cito, Philipp Leitner, Christian Bosshard, Markus Knecht, Genc Mazlami, and Harald C. Gall

   In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings , Gothenburg, Sweden, Jul 2018

   Bib

   @inproceedings{cito:18,
     author = {Cito, J\"{u}rgen and Leitner, Philipp and Bosshard, Christian and Knecht, Markus and Mazlami, Genc and Gall, Harald C.},
     title = {PerformanceHat: Augmenting Source Code with Runtime Performance Traces in the IDE},
     booktitle = {Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings},
     series = {ICSE '18},
     year = {2018},
     isbn = {978-1-4503-5663-3},
     location = {Gothenburg, Sweden},
     pages = {41--44},
     numpages = {4},
     doi = {10.1145/3183440.3183481},
     acmid = {3183481},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {development workflow, program analysis, software performance engineering}
   }

9. Continuous Experimentation: Challenges, Implementation Techniques, and Current Research

   G. Schermann, J. Cito, and P. Leitner

   IEEE Software, Mar 2018

   Bib

   @article{schermann:18,
     author = {Schermann, G. and Cito, J. and Leitner, P.},
     journal = {IEEE Software},
     title = {Continuous Experimentation: Challenges, Implementation Techniques, and Current Research},
     year = {2018},
     volume = {35},
     number = {2},
     pages = {26-31},
     keywords = {formal specification;Internet;program testing;systems analysis;requirements engineering;Web-based systems;Software engineering;Routing;Software testing;Data science;Computer architecture;Software product lines;Product life cycle management;release engineering;continuous experimentation;feature toggles;traffic routing;regression-driven experiments;business-driven experiments;software development;software engineering},
     doi = {10.1109/MS.2018.111094748},
     issn = {0740-7459},
     month = mar
   }

2017

1. Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

   Andre Hoorn, Pooyan Jamshidi, Philipp Leitner, and Ingo Weber

   CoRR, Mar 2017

   Bib

   @article{vanhoorn:18,
     author = {van Hoorn, Andre and Jamshidi, Pooyan and Leitner, Philipp and Weber, Ingo},
     title = {{Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World}},
     journal = {CoRR},
     year = {2017},
     volume = { nn },
     pages = {nn},
   }

2. Optimized IoT service placement in the fog

   Olena Skarlat, Matteo Nardelli, Stefan Schulte, Michael Borkowski, and Philipp Leitner

   Service Oriented Computing and Applications, Dec 2017

   Bib

   @article{skarlat:17,
     author = {Skarlat, Olena and Nardelli, Matteo and Schulte, Stefan and Borkowski, Michael and Leitner, Philipp},
     title = {Optimized IoT service placement in the fog},
     journal = {Service Oriented Computing and Applications},
     year = {2017},
     month = dec,
     day = {01},
     volume = {11},
     number = {4},
     pages = {427--443},
     issn = {1863-2394},
     doi = {10.1007/s11761-017-0219-8},
   }

3. A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

   Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella

   In Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME) , Dec 2017

   Bib

   @inproceedings{vassallo:17,
     author = {Vassallo, Carmine and Schermann, Gerald and Zampetti, Fiorella and Romano, Daniele and Leitner, Philipp and Zaidman, Andy and Penta, Massimiliano Di and Panichella, Sebastiano},
     title = {A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective},
     booktitle = {Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME)},
     year = {2017},
     pages = {183-193},
     doi = {10.1109/ICSME.2017.67},
   }

4. Extraction of Microservices from Monolithic Software Architectures

   Genc Mazlami, Jürgen Cito, and Philipp Leitner

   In Proceedings of the 24th IEEE International Conference on Web Services (ICWS) - Applications Track , Dec 2017

   Bib

   @inproceedings{mazlami:17,
     author = {Mazlami, Genc and Cito, Jürgen and Leitner, Philipp},
     title = {Extraction of Microservices from Monolithic Software Architectures},
     booktitle = {Proceedings of the 24th IEEE International Conference on Web Services (ICWS) - Applications Track},
     year = {2017},
     pages = {524--531},
     doi = {10.1109/ICWS.2017.61}
   }

5. An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software

   Thomas Rausch, Waldemar Hummer, Philipp Leitner, and Stefan Schulte

   In Proceedings of the 14th International Conference on Mining Software Repositories , Buenos Aires, Argentinia, Dec 2017

   Bib

   @inproceedings{rausch:17,
     author = {Rausch, Thomas and Hummer, Waldemar and Leitner, Philipp and Schulte, Stefan},
     title = {An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software},
     booktitle = {Proceedings of the 14th International Conference on Mining Software Repositories},
     series = {MSR'17},
     year = {2017},
     pages = {345--355},
     location = {Buenos Aires, Argentinia},
     publisher = {ACM},
     address = {New York, NY, USA},
   }

6. An Empirical Analysis of the Docker Container Ecosystem on GitHub

   Jürgen Cito, Gerald Schermann, Erik Wittern, Philipp Leitner, Sali Zumberi, and Harald Gall

   In Proceedings of the 14th International Conference on Mining Software Repositories , Buenos Aires, Argentinia, Dec 2017

   Bib

   @inproceedings{cito:17,
     author = {Cito, Jürgen and Schermann, Gerald and Wittern, Erik and Leitner, Philipp and Zumberi, Sali and Gall, Harald},
     title = {An Empirical Analysis of the Docker Container Ecosystem on GitHub},
     booktitle = {Proceedings of the 14th International Conference on Mining Software Repositories},
     series = {MSR'17},
     year = {2017},
     pages = {323--333},
     location = {Buenos Aires, Argentinia},
     publisher = {ACM},
     address = {New York, NY, USA},
   }

7. (h | g)opper: Performance History Mining and Analysis

   Christoph Laaber, and Philipp Leitner

   In Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE) - Poster and Demonstration Track , L’Aquila, Italy, Dec 2017

   Bib

   @inproceedings{laaber:17,
     author = {Laaber, Christoph and Leitner, Philipp},
     title = {{(h | g)opper: Performance History Mining and Analysis}},
     booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE) - Poster and Demonstration Track},
     year = {2017},
     doi = {10.1145/3030207.3053662},
     acmid = {3053662},
     publisher = {ACM},
     address = {New York, NY, USA},
     isbn = {978-1-4503-4404-3},
     location = {L'Aquila, Italy},
     pages = {167--168}
   }

8. An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications

   Christian Davatz, Christian Inzinger, Joel Scheuner, and Philipp Leitner

   In Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing , Madrid, Spain, Dec 2017

   Bib

   @inproceedings{davatz:17,
     author = {Davatz, Christian and Inzinger, Christian and Scheuner, Joel and Leitner, Philipp},
     title = {An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications},
     booktitle = {Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing},
     series = {CCGrid '17},
     year = {2017},
     isbn = {978-1-5090-6610-0},
     location = {Madrid, Spain},
     pages = {534--543},
     numpages = {10},
     doi = {10.1109/CCGRID.2017.12},
     acmid = {3101184},
     publisher = {IEEE Press},
     address = {Piscataway, NJ, USA}
   }

9. Context-Based Analytics – Establishing Explicit Links between Runtime Traces and Source Code

   Jürgen Cito, Faìbio Oliveira, Philipp Leitner, Priya Nagpurkar, and Harald Gall

   In Proceedings of the 38th International Conference on Software Engineering (ICSE’17) - Software Engineering in Practice (SEIP) Track , Dec 2017

   Bib

   @inproceedings{cito:20,
     author = {Cito, Jürgen and Oliveira, Faìbio and Leitner, Philipp and Nagpurkar, Priya and Gall, Harald},
     title = { Context-Based Analytics -- Establishing Explicit Links between Runtime Traces and Source Code},
     booktitle = {Proceedings of the 38th International Conference on Software Engineering (ICSE'17) - Software Engineering in Practice (SEIP) Track},
     year = {2017},
     pages = {193--202},
   }

10. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects

    Philipp Leitner, and Cor-Paul Bezemer

    In Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE) , Dec 2017

    Bib

    @inproceedings{leitner:17,
      author = {Leitner, Philipp and Bezemer, Cor-Paul},
      title = {An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects},
      booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE)},
      year = {2017},
      pages = {373--384},
      doi = {10.1145/2901739.2901774},
    }

2016

1. Probr – A Generic and Passive WiFi Tracking System

   Joel Scheuner, Genc Mazlami, Dominik Schöni, Sebastian Stephan, Alessandro De Carli, Thomas Bocek, and Burkhard Stiller

   In Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN) , Dec 2016

   Bib

   @inproceedings{scheuner:16probr,
     author = {Scheuner, Joel and Mazlami, Genc and Sch{\"o}ni, Dominik and Stephan, Sebastian and Carli, Alessandro De and Bocek, Thomas and Stiller, Burkhard},
     booktitle = {Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN)},
     title = {{Probr -- A Generic and Passive WiFi Tracking System}},
     year = {2016},
     doi = {10.1109/LCN.2016.30},
   }

2. Probr Demonstration – Visualizing Passive WiFi Data

   Joel Scheuner, Genc Mazlami, Dominik Schöni, Sebastian Stephan, Alessandro De Carli, Thomas Bocek, and Burkhard Stiller

   In Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN) , Dec 2016

   Bib

   @inproceedings{scheuner:16probrdemo,
     author = {Scheuner, Joel and Mazlami, Genc and Sch{\"o}ni, Dominik and Stephan, Sebastian and Carli, Alessandro De and Bocek, Thomas and Stiller, Burkhard},
     booktitle = {Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN)},
     title = {{Probr Demonstration -- Visualizing Passive WiFi Data}},
     year = {2016},
   }

3. Resource Provisioning for IoT Services in the Fog

   Olena Skarlat, Stefan Schulte, Michael Borkowski, and Philipp Leitner

   In 9th IEEE International Conference on Service-Oriented Computing and Applications (SOCA) , Dec 2016

   Bib

   @inproceedings{skarlat:16,
     author = {Skarlat, Olena and Schulte, Stefan and Borkowski, Michael and Leitner, Philipp},
     title = {Resource Provisioning for IoT Services in the Fog},
     booktitle = {9th {IEEE} International Conference on Service-Oriented Computing
                   and Applications ({SOCA})},
     pages = {32--39},
     year = {2016},
     doi = {10.1109/SOCA.2016.10}
   }

4. Modelling and Managing Deployment Costs of Microservice-Based Cloud Applications

   Philipp Leitner, Jürgen Cito, and Emanuel Stöckli

   In Proceedings of the 9th IEEE/ACM International Conference on Utility and Cloud Computing (UCC) , Dec 2016

   Bib

   @inproceedings{leitner:16:2,
     author = {Leitner, Philipp and Cito, Jürgen and Stöckli, Emanuel},
     title = {Modelling and Managing Deployment Costs of Microservice-Based Cloud Applications},
     booktitle = {Proceedings of the 9th IEEE/ACM International Conference on Utility and Cloud Computing (UCC)},
     year = {2016},
     pages = {nn},
   }

5. Bifrost - Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies

   Gerald Schermann, Dominik Schöni, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 2016 ACM/IFIP/USENIX Middleware Conference, Best Student Paper Award , Trento, Italy, Dec 2016

   Bib

   @inproceedings{schermann:16:2,
     author = {Schermann, Gerald and Sch\"oni, Dominik and Leitner, Philipp and Gall, Harald C.},
     title = {Bifrost - Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies},
     booktitle = {Proceedings of the 2016 ACM/IFIP/USENIX Middleware Conference, <strong>Best Student Paper Award</strong>},
     year = {2016},
     location = {Trento, Italy},
     pages = {nn},
   }

6. TemPerf: Temporal Correlation Between Performance Metrics and Source Code

   Jürgen Cito, Genc Mazlami, and Philipp Leitner

   In Proceedings of the 2nd International Workshop on Quality-Aware DevOps , Saarbrücken, Germany, Dec 2016

   Bib

   @inproceedings{cito:16,
     author = {Cito, Jürgen and Mazlami, Genc and Leitner, Philipp},
     title = {TemPerf: Temporal Correlation Between Performance Metrics and Source Code},
     booktitle = {Proceedings of the 2nd International Workshop on Quality-Aware DevOps},
     series = {QUDOS 2016},
     year = {2016},
     isbn = {978-1-4503-4411-1},
     location = {Saarbr\&\#252;cken, Germany},
     pages = {46--47},
     numpages = {2},
     doi = {10.1145/2945408.2945420},
     acmid = {2945420},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {DevOps, Developer Tools, Performance}
   }

7. Towards Quality Gates in Continuous Delivery and Deployment

   Gerald Schermann, Jürgen Cito, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 24TH IEEE International Conference on Program Comprehension (ICPC), Best Short Paper Award , Austin, Texas, Dec 2016

   Bib

   @inproceedings{schermann:16,
     author = {Schermann, Gerald and Cito, Jürgen and Leitner, Philipp and Gall, Harald C.},
     title = {Towards Quality Gates in Continuous Delivery and Deployment},
     booktitle = {Proceedings of the 24TH IEEE International Conference on Program Comprehension (ICPC), <strong>Best Short Paper Award</strong>},
     year = {2016},
     location = {Austin, Texas},
     pages = {nn}
   }

8. TOIT

   

   Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds

   Philipp Leitner, and Jürgen Cito

   ACM Transactions on Internet Technology, Apr 2016

   Bib PDF

   @article{leitner:16,
     author = {Leitner, Philipp and Cito, Jürgen},
     title = {Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds},
     journal = {ACM Transactions on Internet Technology},
     issue_date = {June 2016},
     volume = {16},
     number = {3},
     month = apr,
     year = {2016},
     issn = {1533-5399},
     pages = {15:1--15:23},
     articleno = {15},
     numpages = {23},
     doi = {10.1145/2885497},
     acmid = {2885497},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {Infrastructure-as-a-service, benchmarking, public cloud},
   }

2015

1. All the Services Large and Micro: Revisiting Industrial Practice in Services Computing

   Gerald Schermann, Jürgen Cito, and Philipp Leitner

   In Proceedings of the 11th International Workshop on Engineering Service Oriented Applications (WESOA’15) , Goa, India, Apr 2015

   Bib

   @inproceedings{scherman:15:1,
     author = {Schermann, Gerald and Cito, Jürgen and Leitner, Philipp},
     title = {All the Services Large and Micro: Revisiting Industrial Practice in Services Computing},
     booktitle = {Proceedings of the 11th International Workshop on Engineering Service Oriented Applications (WESOA'15)},
     year = {2015},
     pages = {nn},
     location = {Goa, India}
   }

2. A Framework for a Cost-Efficient Cloud Ecosystem

   Eliot Salant, Philipp Leitner, Karl Wallbom, and James Athes

   In eChallenges e-2015 Conference , Vilnius, Lithuania, Apr 2015

   Bib

   @inproceedings{salant:15,
     author = {Salant, Eliot and Leitner, Philipp and Wallbom, Karl and Athes, James},
     title = {{A Framework for a Cost-Efficient Cloud Ecosystem}},
     booktitle = {{eChallenges e-2015 Conference}},
     year = {2015},
     pages = {nn},
     location = {Vilnius, Lithuania}
   }

3. Bursting With Possibilities – an Empirical Study of Credit-Based Bursting Cloud Instance Types

   Philipp Leitner, and Joel Scheuner

   In Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC) , Limassol, Cyprus, Apr 2015

   Bib

   @inproceedings{leitner:15,
     author = {Leitner, Philipp and Scheuner, Joel},
     title = {Bursting With Possibilities – an Empirical Study of Credit-Based Bursting Cloud Instance Types},
     booktitle = {Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC)},
     year = {2015},
     location = {Limassol, Cyprus},
     pages = {nn}
   }

4. Intent, Tests, and Release Dependencies: Pragmatic Recipes for Source Code Integration

   Martin Brandtner, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 15th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) , Bremen, Germany, Apr 2015

   Bib

   @inproceedings{brandtner:15:2,
     author = {Brandtner, Martin and Leitner, Philipp and Gall, Harald C.},
     title = {Intent, Tests, and Release Dependencies: Pragmatic Recipes for Source Code Integration},
     booktitle = {Proceedings of the 15th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM)},
     year = {2015},
     location = {Bremen, Germany},
     pages = {nn}
   }

5. CloudWave - Leveraging DevOps for Cloud Management and Application Development

   Dario Bruneo, Aryan Dadashi, Philipp Leitner, Boris Moltchanov, Francesco Javier Nieto De-Santos, Avi Miron, Eliot Salant, Amir Molzam Sharifloo, Karl Wallbom, and Chris Woods

   In Proceedings of the Fourth European Conference on Service-Oriented and Cloud Computing (ESOCC’15) - EU Projects Track , Taormina, Italy, Apr 2015

   Bib

   @inproceedings{bruneo:15,
     author = {Bruneo, Dario and Dadashi, Aryan and Leitner, Philipp and Moltchanov, Boris and De-Santos, Francesco Javier Nieto and Miron, Avi and Salant, Eliot and Sharifloo, Amir Molzam and Wallbom, Karl and Woods, Chris},
     title = {CloudWave - Leveraging DevOps for Cloud Management and Application Development},
     booktitle = {Proceedings of the Fourth European Conference on Service-Oriented and Cloud Computing (ESOCC'15) - EU Projects Track},
     year = {2015},
     location = {Taormina, Italy},
     pages = {nn}
   }

6. JCloudScale: Closing the Gap Between IaaS and PaaS

   Rostyslav Zabolotnyi, Philipp Leitner, Waldemar Hummer, and Schahram Dustdar

   ACM Transactions on Internet Technology, Jul 2015

   Bib

   @article{zabolotnyi:15,
     author = {Zabolotnyi, Rostyslav and Leitner, Philipp and Hummer, Waldemar and Dustdar, Schahram},
     title = {JCloudScale: Closing the Gap Between IaaS and PaaS},
     journal = {ACM Transactions on Internet Technology},
     issue_date = {September 2015},
     volume = {15},
     number = {3},
     month = jul,
     year = {2015},
     issn = {1533-5399},
     pages = {10:1--10:20},
     articleno = {10},
     numpages = {20},
     doi = {10.1145/2792980},
     acmid = {2792980},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {Cloud computing, JCloudScale, middleware, programming}
   }

7. The Making of Cloud Applications – An Empirical Study on Software Development for the Cloud

   Jürgen Cito, Philipp Leitner, Thomas Fritz, and Harald C. Gall

   In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE) , Bergamo, Italy, Jul 2015

   Bib

   @inproceedings{cito:15:3,
     author = {Cito, Jürgen and Leitner, Philipp and Fritz, Thomas and Gall, Harald C.},
     title = {The Making of Cloud Applications – An Empirical Study on Software Development for the Cloud},
     booktitle = {Proceedings of the 10th Joint Meeting of the European Software Engineering
                    Conference and the {ACM} {SIGSOFT} International Symposium on Foundations
                    of Software Engineering (ESEC/FSE)},
     year = {2015},
     location = {Bergamo, Italy},
     publisher = {ACM},
     pages = {393--403},
     address = {New York, NY, USA}
   }

8. Onward! @ SPLASH 2015

   

   Runtime Metric Meets Developer - Building Better Cloud Applications Using Feedback

   Jürgen Cito, Philipp Leitner, Harald C. Gall, Aryan Dadashi, Anne Keller, and Andreas Roth

   In Proceedings of the 2015 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2015) , Pittsburgh, Pennsylvania, United States, Jul 2015

   Bib PDF

   @inproceedings{cito:15:2,
     author = {Cito, Jürgen and Leitner, Philipp and Gall, Harald C. and Dadashi, Aryan and Keller, Anne and Roth, Andreas},
     title = {Runtime Metric Meets Developer - Building Better Cloud
                  Applications Using Feedback},
     booktitle = {Proceedings of the 2015 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming \& Software (Onward! 2015)},
     year = {2015},
     pages = {14--27},
     location = {Pittsburgh, Pennsylvania, United States},
     publisher = {ACM},
     address = {New York, NY, USA},
   }

9. Identifying Web Performance Degradations Through Synthetic and Real-user Monitoring

   Jürgen Cito, Devan Gotowka, Philipp Leitner, Ryan Pelette, Dritan Suljoti, and Schahram Dustdar

   Journal of Web Engineering, Nov 2015

   Bib

   @article{cito:15,
     author = {Cito, Jürgen and Gotowka, Devan and Leitner, Philipp and Pelette, Ryan and Suljoti, Dritan and Dustdar, Schahram},
     title = {Identifying Web Performance Degradations Through Synthetic and Real-user Monitoring},
     journal = {Journal of Web Engineering},
     issue_date = {November 2015},
     volume = {14},
     number = {5-6},
     month = nov,
     year = {2015},
     issn = {1540-9589},
     pages = {414--442},
     numpages = {29},
     acmid = {2871278},
     publisher = {Rinton Press, Incorporated},
     address = {Paramus, NJ},
     keywords = {application performance monitoring, changepoint analysis, performance engineering, web services}
   }

10. SPEEDL - A Declarative Event-Based Language for Cloud Scaling Definition

    Rostyslav Zabolotnyi, Philipp Leitner, Stefan Schulte, and Schahram Dustdar

    In Proceedings of the Future of Software Engineering For and In Cloud, Visionary Track of IEEE Services , Nov 2015

    Bib

    @inproceedings{zabolotnyi:16,
      author = {Zabolotnyi, Rostyslav and Leitner, Philipp and Schulte, Stefan and Dustdar, Schahram},
      title = {SPEEDL - A Declarative Event-Based Language for Cloud Scaling Definition},
      booktitle = {Proceedings of the Future of Software Engineering For and In Cloud, Visionary Track of IEEE Services},
      year = {2015},
      pages = {nn}
    }

11. Profile-based View Composition in Development Dashboards

    Martin Brandtner, Philipp Leitner, and Harald Gall

    In Proceedings of the 17th Workshop Software-Reengineering & Evolution (WSRE) , Nov 2015

    Bib

    @inproceedings{brandtner:15,
      author = {Brandtner, Martin and Leitner, Philipp and Gall, Harald},
      booktitle = {Proceedings of the 17th Workshop Software-Reengineering & Evolution (WSRE)},
      title = {Profile-based View Composition in Development Dashboards},
      year = {2015},
      pages = {nn}
    }

12. Discovering Loners and Phantoms in Commit and Issue Data

    Gerald Schermann, Martin Brandtner, Sebastiano Panichella, Philipp Leitner, and Harald C. Gall

    In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension , Florence, Italy, Nov 2015

    Bib

    @inproceedings{schermann:15,
      author = {Schermann, Gerald and Brandtner, Martin and Panichella, Sebastiano and Leitner, Philipp and Gall, Harald C.},
      title = {Discovering Loners and Phantoms in Commit and Issue Data},
      booktitle = {Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension},
      series = {ICPC '15},
      year = {2015},
      location = {Florence, Italy},
      pages = {4--14},
      numpages = {11},
      acmid = {2820287},
      publisher = {IEEE Press},
      address = {Piscataway, NJ, USA}
    }

13. Cloud WorkBench: Benchmarking IaaS Providers Based on Infrastructure-as-Code

    Joel Scheuner, Jürgen Cito, Philipp Leitner, and Harald C. Gall

    In Proceedings of the 24th International Conference on World Wide Web , Florence, Italy, Nov 2015

    Bib

    @inproceedings{scheuner:15,
      author = {Scheuner, Joel and Cito, Jürgen and Leitner, Philipp and Gall, Harald C.},
      title = {Cloud WorkBench: Benchmarking IaaS Providers Based on Infrastructure-as-Code},
      booktitle = {Proceedings of the 24th International Conference on World Wide Web},
      series = {WWW '15 Companion},
      year = {2015},
      isbn = {978-1-4503-3473-0},
      location = {Florence, Italy},
      pages = {239--242},
      numpages = {4},
      doi = {10.1145/2740908.2742833},
      acmid = {2742833},
      publisher = {ACM},
      address = {New York, NY, USA},
      keywords = {benchmarking, cloud computing, devops, iaas, iac}
    }

14. SQA-Profiles: Rule-Based Activity Profiles for Continuous Integration Environments

    Martin Brandtner, Sebastian C. Müller, Philipp Leitner, and Harald Gall

    In Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER’15) , Nov 2015

    Bib

    @inproceedings{brandtner:15:3,
      author = {Brandtner, Martin and Müller, Sebastian C. and Leitner, Philipp and Gall, Harald},
      booktitle = {Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER'15)},
      title = {SQA-Profiles: Rule-Based Activity Profiles for Continuous Integration Environments},
      year = {2015},
      pages = {nn}
    }

15. Comparing and Combining Predictive Business Process Monitoring Techniques

    Andreas Metzger, Philipp Leitner, Dragan Ivanovic, Eric Schmieders, Rod Franklin, Manuel Carro, Schahram Dustdar, and Klaus Pohl

    IEEE Transactions on Systems, Man, and Cybernetics: Systems, Feb 2015

    Bib

    @article{metzger:14,
      author = {Metzger, Andreas and Leitner, Philipp and Ivanovic, Dragan and Schmieders, Eric and Franklin, Rod and Carro, Manuel and Dustdar, Schahram and Pohl, Klaus},
      journal = {IEEE Transactions on Systems, Man, and Cybernetics: Systems},
      title = {Comparing and Combining Predictive Business Process Monitoring Techniques},
      year = {2015},
      volume = {45},
      number = {2},
      pages = {276-290},
      keywords = {business data processing;constraint satisfaction problems;learning (artificial intelligence);QoS aggregation;accuracy indicators;constraint satisfaction;machine learning;predictive business process monitoring techniques;process execution;quality-of-service aggregation;Accuracy;Business;Data models;Monitoring;Predictive models;Quality of service;Real-time systems;Business data processing;failure analysis;forecasting;neural network applications;transportation},
      doi = {10.1109/TSMC.2014.2347265},
      issn = {2168-2216},
      month = feb,
      top = {true}
    }

2014

1. CloudCom 2014

   

   Cloud WorkBench - Infrastructure-as-Code Based Cloud Benchmarking

   Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall

   In Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’14) , Feb 2014

   Bib PDF

   @inproceedings{scheuner:14,
     author = {Scheuner, Joel and Leitner, Philipp and Cito, Jürgen and Gall, Harald},
     booktitle = {Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom'14)},
     title = {Cloud WorkBench - Infrastructure-as-Code Based Cloud Benchmarking},
     year = {2014},
     pages = {nn},
     doi = {10.1109/CloudCom.2014.98},
   }

2. WPress: Benchmarking Infrastructure-as-a-Service Cloud Computing Systems for On-line Transaction Processing Applications

   Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung

   In Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC) , Feb 2014

   Bib

   @inproceedings{borhani:14,
     author = {Borhani, Amir Hossein and Leitner, Philipp and Lee, Bu-Sung and Li, Xiaorong and Hung, Terence},
     booktitle = {Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC)},
     title = {WPress: Benchmarking Infrastructure-as-a-Service Cloud Computing Systems for On-line Transaction Processing Applications},
     year = {2014},
     pages = {nn}
   }

3. Profiling-Based Task Scheduling for Factory-Worker Applications in Infrastructure-as-a-Service Clouds

   Rostyslav Zabolotnyi, Philipp Leitner, and Schahram Dustdar

   In Proceedings of the 40th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , Feb 2014

   Bib

   @inproceedings{zabolotnyi:14:2,
     author = {Zabolotnyi, Rostyslav and Leitner, Philipp and Dustdar, Schahram},
     booktitle = {Proceedings of the 40th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)},
     title = {Profiling-Based Task Scheduling for Factory-Worker Applications in Infrastructure-as-a-Service Clouds},
     year = {2014},
     pages = {nn}
   }

4. CloudWave: where Adaptive Cloud Management Meets DevOps

   Dario Bruneo, Thomas Fritz, Sharon Keidar-Barner, Philipp Leitner, Francesco Longo, Clarissa Marquezan, Andreas Metzger, Klaus Pohl, Antonio Puliafito, Danny Raz, Andreas Roth, Eliot Salant, Itai Segall, Massimo Villari, Yaron Wolfsthal, and Chris Woods

   In Proceedings of the Fourth International Workshop on Management of Cloud Systems (MoCS 2014) , Feb 2014

   Bib

   @inproceedings{bruneo:14,
     title = {CloudWave: where Adaptive Cloud Management Meets DevOps},
     author = {Bruneo, Dario and Fritz, Thomas and Keidar-Barner, Sharon and Leitner, Philipp and Longo, Francesco and Marquezan, Clarissa and Metzger, Andreas and Pohl, Klaus and Puliafito, Antonio and Raz, Danny and Roth, Andreas and Salant, Eliot and Segall, Itai and Villari, Massimo and Wolfsthal, Yaron and Woods, Chris},
     booktitle = {Proceedings of the Fourth International Workshop on Management of Cloud Systems (MoCS 2014)},
     year = {2014},
     pages = {nn}
   }

5. Identifying Root Causes of Web Performance Degradation Using Changepoint Analysis

   Jürgen Cito, Dritan Suljoti, Philipp Leitner, and Schahram Dustdar

   In Web Engineering: 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings , Feb 2014

   Bib

   @inproceedings{cito:14,
     author = {Cito, Jürgen and Suljoti, Dritan and Leitner, Philipp and Dustdar, Schahram},
     editor = {Casteleyn, Sven and Rossi, Gustavo and Winckler, Marco},
     title = {Identifying Root Causes of Web Performance Degradation Using Changepoint Analysis},
     booktitle = {Web Engineering: 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings},
     year = {2014},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {181--199},
     isbn = {978-3-319-08245-5},
     doi = {10.1007/978-3-319-08245-5_11},
   }

6. A Note on Software Tools and Techniques for Monitoring and Prediction of Cloud Services

   Rajiv Ranjan, Rajkumar Buyya, Philipp Leitner, Armin Haller, and Stefan Tai

   Software: Practice and Experience, Jul 2014

   Bib

   @article{ranjan:14,
     author = {Ranjan, Rajiv and Buyya, Rajkumar and Leitner, Philipp and Haller, Armin and Tai, Stefan},
     title = {A Note on Software Tools and Techniques for Monitoring and Prediction of Cloud Services},
     journal = {Software: Practice and Experience},
     issue_date = {July 2014},
     volume = {44},
     number = {7},
     month = jul,
     year = {2014},
     issn = {0038-0644},
     pages = {771--775},
     numpages = {5},
     doi = {10.1002/spe.2266},
     acmid = {2904546},
     publisher = {John Wiley \& Sons, Inc.},
     address = {New York, NY, USA}
   }