publications

2025

1. TSE

   

   The Impact of Prompt Programming on Function-Level Code Generation

   Ranim Khojah, Francisco Gomes Oliveira Neto, Mazen Mohamad, and Philipp Leitner

   IEEE Transactions on Software Engineering (TSE), 2025

   To appear

   DOI
2. EASE 2025

   

   Emotional Strain and Frustration in LLM Interactions in Software Engineering

   Cristina Martinez Montes and Ranim Khojah

   In Proceedings of the ACM International Conference on Evaluation and Assessment in Software Engineering (EASE 2025), 2025

2024

1. EAAI

   

   A Unified Active Learning Framework for Annotating Graph Data foor Regression Tasks

   Peter Samoaa, Linus Aronsson, Antonio Longa, Philipp Leitner, and Morteza Haghir Chehreghani

   Journal of Engineering Applications of Artificial Intelligence, 2024

   PDF
2. FSE Visions 2024

   

   The lion, the ecologist and the plankton: a classification of species in multi-bot ecosystems

   Dimitrios Platis, Linda Erlenhov, and Francisco Gomes Oliveira Neto

   In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024) - Ideas, Visions, and Reflections Track, Porto de Galinhas, Brazil, 2024

   DOI
3. AIware @ FSE 2024

   

   From Human-to-Human to Human-to-Bot Conversations in Software Engineering

   Ranim Khojah, Francisco Gomes Oliveira Neto, and Philipp Leitner

   In Proceedings of the The First International Conference on AIware (AIware), Porto de Galinhas, Brazil, 2024

   DOI PDF
4. FSE 2024

   

   Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

   Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes Oliveira Neto

   In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024), 2024

   DOI PDF Supp
5. JSS

   

   An empirical investigation on the competences and roles of practitioners in Microservices-based Architectures

   Hamdy Michael Ayas, Regina Hebig, and Philipp Leitner

   Journal of Systems and Software, 2024

   Abs DOI Supp

   Microservices-based Architectures (MSAs) are gaining popularity since, among others, they enable rapid and independent delivery of software at scale, facilitating the delivery of business value. Additionally, there are attempts towards understanding practitioners’ roles and technical knowledge. MSAs call for affinity in several technologies as well as business domains. This diversity makes it challenging to scope and describe the roles of practitioners. In addition, practitioners often do not receive training and contents of MSA training remain largely undefined, even though there are challenges in finding or developing relevant technical expertise. In this research, we determine the different technical roles that are required in MSAs, along with their detailed competences. We use public online forums (e.g., StackOverflow), where developers share technical knowledge. We analyze 13,517 public profiles of software engineers, deriving their technical competences. Our taxonomy of technical competences in MSAs, contains 11 competences clusters, organized in 3 collections of competences — Web Technologies, DevOps, and Data Technologies. In addition, we derive the roles of microservice practitioners and the characteristics of their roles. Our findings organize the technical competences of MSAs practitioners and determine the training topics and combination of topics that can prepare engineers for MSAs.

6. CHASE @ ICSE 2024

   

   The Roles, Responsibilities, and Skills of Engineers in the Era of Microservices-Based Architectures

   Hamdy Michael Ayas, Regina Hebig, and Philipp Leitner

   In Proceedings of the 16th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE 2023), Lisbon, Portugal, 2024

   DOI
7. ICPC @ ICSE 2024

   

   The Impact of Compiler Warnings on Code Quality in C++ Projects

   Albin Johansson, Carl Holmberg, Francisco Gomes De Oliveira Neto, and Philipp Leitner

   In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, Lisbon, Portugal, 2024

   Abs DOI Thesis

   Modern compilers often offer a variety of warning flags, which developers can enable to get feedback on code that, while syntactically correct, may be problematic. In the case of C++, one example of such "correct but problematic" code is code that leads to undefined behavior (UB). The usage of compiler warnings has long been suspected as a way to decrease bugs and increase code quality. However, empirical evidence that supports this hypothesis is rare. In this study, we present evidence from a study of 127 open source C++ projects. We categorize their usage of compiler warnings into five groups based on which warning flags are being used, and analyse the relationship between compiler warnings and five quality metrics (bugs, critical issues, vulnerabilities, code smells, and technical debt) using Bayesian analysis. We conclude that, in general, compiler warnings indeed correlate with, and potentially cause, higher code quality, with the clearest impact being on the number of critical issues in a project. Using stricter warning flags expectantly correlates with higher code quality in our study objects. However, there are substantial differences between projects, which we attribute to the project’s individual development culture. That is, while warnings matter, other factors such as quality culture, are likely to be even more important to source code quality.

8. SIGMETRICS 2024

   

   Machine Learning Systems are Bloated and Vulnerable

   Huaifeng Zhang, Mohannad Alhanahnah, Fahmi Abdulqadir Ahmed, Dyako Fatih, Philipp Leitner, and Ahmed Ali-Eldin

   Proceedings of SIGMETRICS, Feb 2024

   Abs DOI PDF

   Today’s software is bloated with both code and features that are not used by most users. This bloat is prevalent across the entire software stack, from operating systems and applications to containers. Containers are lightweight virtualization technologies used to package code and dependencies, providing portable, reproducible and isolated environments. For their ease of use, data scientists often utilize machine learning containers to simplify their workflow. However, this convenience comes at a cost: containers are often bloated with unnecessary code and dependencies, resulting in very large sizes. In this paper, we analyze and quantify bloat in machine learning containers. We develop MMLB, a framework for analyzing bloat in software systems, focusing on machine learning containers. MMLB measures the amount of bloat at both the container and package levels, quantifying the sources of bloat. In addition, MMLB integrates with vulnerability analysis tools and performs package dependency analysis to evaluate the impact of bloat on container vulnerabilities. Through experimentation with 15 machine learning containers from TensorFlow, PyTorch, and Nvidia, we show that bloat accounts for up to 80% of machine learning container sizes, increasing container provisioning times by up to 370% and exacerbating vulnerabilities by up to 99%.

2023

1. Big Data 2023

   

   Batch Mode Deep Active Learning for Regression on Graph Data

   P. Samoaa, L. Aronsson, P. Leitner, and M. Chehreghani

   In 2023 IEEE International Conference on Big Data (BigData), Dec 2023

   Abs DOI PDF

   Acquiring labelled data for machine learning tasks, for example, for software performance prediction, remains a resource-intensive task. This study extends our previous work by introducing a batch-mode deep active learning approach tailored for regression in graph-structured data. Our framework leverages the source code conversion into Flow Augmented-AST graphs (FA-AST), subsequently utilizing both supervised and unsupervised graph embeddings. In contrast to single-instance querying, the batch-mode paradigm adaptively selects clusters of unlabeled data for labelling. We deploy an array of base kernels, kernel transformations, and selection methods, informed by both Bayesian and non-Bayesian strategies, to enhance the sample efficiency of neural network regression. Our experimental evaluation, conducted on multiple real-world software performance datasets, demonstrates the efficacy of the batch mode deep active learning approach in achieving robust performance with a reduced labelling budget. The methodology scales effectively to larger datasets and requires minimal alterations to existing neural network architectures.

2. SOSE 2023

   

   The Perceived Impact and Sequence of Activities When Transitioning to Microservices

   H. Ayas, P. Leitner, and R. Hebig

   In 2023 IEEE International Conference on Service-Oriented System Engineering (SOSE), Jul 2023

   (short paper)

   Abs DOI

   Microservices migrations are often present in organizations that want to modernize and integrate their software systems. There is little empirical evidence showing how migration projects take place. Investigating migration activities and the perceived impact of migrations from practitioners is important to understand how migration projects materialize. In this study, we ask 54 practitioners about their views on specific aspects of microservices migrations. Specifically, we derive the sequence of migration activities as well as the perceived impact of microservices on the development process. Database refactoring, back-end refactoring, and setting up DevOps tend to take place before front-end refactoring, setting up communication patterns, and splitting teams. In addition, aligning teams with profitable value propositions, improving the testing process, and having fewer dependencies between teams are among the prominent impact areas of microservices. Our results call for further empirical research in understanding transitions toward MSAs.

3. BotSE @ ICSE 2023

   

   Towards Continuous Performance Assessment of Java Applications With PerfBot

   Florian Markusse, Alexander Serebrenik, and Philipp Leitner

   In 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE), Jul 2023

   DOI
4. EMSE

   

   An empirical study of the systemic and technical migration towards microservices

   Hamdy Michael Ayas, Philipp Leitner, and Regina Hebig

   Empirical Software Engineering, May 2023

   DOI
5. TCC

   

   Using Microbenchmark Suites to Detect Application Performance Changes

   M. Grambow, D. Kovalev, C. Laaber, P. Leitner, and D. Bermbach

   IEEE Transactions on Cloud Computing, Jul 2023

   Abs DOI PDF

   Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of the two time-series database systems InfluxDB and VictoriaMetrics and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is possible to detect application performance changes using an optimized microbenchmark suite if frequent false-positive alarms can be tolerated.

6. TSE

   

   Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit Tests

   Mostafa Jangali, Yiming Tang, Niclas Alexandersson, Philipp Leitner, Jinqiu Yang, and Weiyi Shang

   IEEE Transactions on Software Engineering, Apr 2023

   Abs DOI PDF

   Performance is a crucial non-functional requirement of many software systems. Despite the widespread use of performance testing, developers still struggle to construct and evaluate the quality of performance tests. To address these two major challenges, we implement a framework, dubbed ju2jmh, to automatically generate performance microbenchmarks from JUnit tests and use mutation testing to study the quality of generated microbenchmarks. Specifically, we compare our ju2jmh generated benchmarks to manually written JMH benchmarks and to automatically generated JMH benchmarks using the AutoJMH framework, as well as directly measuring system performance with JUnit tests. For this purpose, we have conducted a study on three subjects (Rxjava, Eclipse-collections, and Zipkin) with sim∼454K source lines of code (SLOC), 2,417 JMH benchmarks (including manually written and generated AutoJMH benchmarks) and 35,084 JUnit tests. Our results show that the ju2jmh generated JMH benchmarks consistently outperform using the execution time and throughput of JUnit tests as a proxy of performance and JMH benchmarks automatically generated using the AutoJMH framework while being comparable to JMH benchmarks manually written by developers in terms of tests’ stability and ability to detect performance bugs. Nevertheless, ju2jmh benchmarks are able to cover more of the software applications than manually written JMH benchmarks during the microbenchmark execution. Furthermore, ju2jmh benchmarks are generated automatically, while manually written JMH benchmarks require many hours of hard work and attention; therefore our study can reduce developers’ effort to construct microbenchmarks. In addition, we identify three factors (too low test workload, unstable tests and limited mutant coverage) that affect a benchmark’s ability to detect performance bugs. To the best of our knowledge, this is the first study aimed at assisting developers in fully automated microbenchmark creation and assessing microbenchmark quality for performance testing.

2022

1. UCC 2022

   

   CrossFit: Fine-grained Benchmarking of Serverless Application Performance across Cloud Providers

   Joel Scheuner, Rui Deng, Jan-Philipp Steghöfer, and Philipp Leitner

   In 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing (UCC), Apr 2022

   DOI PDF
2. PROFES 2022

   

   TEP-GNN: Accurate Execution Time Prediction of Functional Tests Using Graph Neural Networks

   Hazem Peter Samoaa, Antonio Longa, Mazen Mohamad, Morteza Haghir Chehreghani, and Philipp Leitner

   In Product-Focused Software Process Improvement, Apr 2022

   Abs PDF

   Predicting the performance of production code prior to actual execution is known to be highly challenging. In this paper, we propose a predictive model, dubbed  TEP-GNN, which demonstrates that high-accuracy performance prediction is possible for the special case of predicting unit test execution times.  TEP-GNN uses FA-ASTs, or flow-augmented ASTs, as a graph-based code representation approach, and predicts test execution times using a powerful graph neural network (GNN) deep learning model. We evaluate  TEP-GNN using four real-life Java open source programs, based on 922 test files mined from the projects’ public repositories. We find that our approach achieves a high Pearson correlation of 0.789, considerable outperforming a baseline deep learning model. Our work demonstrates that FA-ASTs and GNNs are a feasible approach for predicting absolute performance values, and serves as an important intermediary step towards being able to predict the performance of arbitrary code prior to execution.

3. IC2E 2022

   

   TriggerBench: A Performance Benchmark for Serverless Function Triggers

   Joel Scheuner, Marcus Bertilsson, Oskar Grönqvist, Henrik Tao, Henrik Lagergren, Jan-Philipp Steghöfer, and Philipp Leitner

   In 2022 IEEE International Conference on Cloud Engineering (IC2E), Apr 2022

   DOI PDF
4. SEAA 2022

   

   An Empirical Analysis of Microservices Systems Using Consumer-Driven Contract Testing

   Hamdy Michael Ayas, Hartmut Fischer, Philipp Leitner, and Francisco Gomes De Oliveira Neto

   In 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Apr 2022

   DOI
5. IEEE Software

   

   Using Benchmarking Bots for Continuous Performance Assessment

   Florian Markusse, Philipp Leitner, and Alexander Serebrenik

   IEEE Software, Apr 2022

   DOI PDF
6. IET Software

   

   A systematic mapping study of source code representation for deep learning in software engineering

   Hazem Peter Samoaa, Firas Bayram, Pasquale Salza, and Philipp Leitner

   IET Software, Apr 2022

   Abs DOI

   The usage of deep learning (DL) approaches for software engineering has attracted much attention, particularly in source code modelling and analysis. However, in order to use DL, source code needs to be formatted to fit the expected input form of DL models. This problem is known as source code representation. Source code can be represented via different approaches, most importantly, the tree-based, token-based, and graph-based approaches. We use a systematic mapping study to investigate i detail the representation approaches adopted in 103 studies that use DL in the context of software engineering. Thus, studies are collected from 2014 to 2021 from 14 different journals and 27 conferences. We show that each way of representing source code can provide a different, yet orthogonal view of the same source code. Thus, different software engineering tasks might require different (combinations of) code representation approaches, depending on the nature and complexity of the task. Particularly, we show that it is crucial to define whether the DL approach requires lexical, syntactical, or semantic code information. Our analysis shows that a wide range of different representations and combinations of representations (hybrid representations) are used to solve a wide range of common software engineering problems. However, we also observe that current research does not generally attempt to transfer existing representations or models to other studies even though there are other contexts in which these representations and models may also be useful. We believe that there is potential for more reuse and the application of transfer learning when applying DL to software engineering tasks.

7. TSE

   

   The State of Serverless Applications: Collection, Characterization, and Community Consensus

   Simon Eismann, Joel Scheuner, Erwin van Eyk, Maximilian Schwinger, Johannes Grohmann, Nikolas Herbst, Cristina L. Abad, and Alexandru Iosup

   IEEE Transactions on Software Engineering, Oct 2022

   Abs DOI Supp

   Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.

2021

1. PeerJ CS

   

   Dependency Management Bots in Open-Source Systems - Prevalence and Adoption

   Linda Erlenhov, Francisco Gomes Oliveira Neto, and Philipp Leitner

   PeerJ Computer Science, Oct 2021

   DOI Supp
2. ESEM 2021

   

   Facing the Giant: a Grounded Theory Study of Decision-Making in Microservices Migrations

   Hamdy Michael Ayas, Philipp Leitner, and Regina Hebig

   In Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Bari, Italy, Oct 2021

   Abs DOI

   Background: Microservices migrations are challenging and expensive projects with many decisions that need to be made in a multitude of dimensions. Existing research tends to focus on technical issues and decisions (e.g., how to split services). Equally important organizational or business issues and their relations with technical aspects often remain out of scope or on a high level of abstraction.Aims: In this study, we aim to holistically chart the decision-making that happens on all dimensions of a migration project towards microservices (including, but not limited to, the technical dimension).Method: We investigate 16 different migration cases in a grounded theory interview study, with 19 participants that recently migrated towards microservices. This study strongly focuses on the human aspects of a migration, through stakeholders and their decisions.Results: We identify 3 decision-making processes consisting of 22 decision-points and their alternative options. The decision-points are related to creating stakeholder engagement and assessing feasibility, technical implementation, and organizational restructuring.Conclusions: Our study provides an initial theory of decisionmaking in migrations to microservices. It also outfits practitioners with a roadmap of which decisions they should be prepared to make and at which point in the migration.

3. EMSE

   

   Applying test case prioritization to software microbenchmarks

   Christoph Laaber, Harald C. Gall, and Philipp Leitner

   Empirical Software Engineering, Oct 2021

   DOI Supp
4. PeerJ CS

   

   Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

   Martin Grambow, Christoph Laaber, Philipp Leitner, and David Bermbach

   PeerJ Computer Science, Oct 2021

   DOI
5. SOFSEM 2021

   

   A Pipeline for Measuring Brand Loyalty Through Social Media Mining

   Hazem Samoaa and Barbara Catania

   In SOFSEM 2021: Theory and Practice of Computer Science, Oct 2021

   Abs

   Enhancing customer relationships through social media is an area of high relevance for companies. To this aim, Social Business Intelligence (SBI) plays a crucial role by supporting companies in combining corporate data with user-generated content, usually available as textual clips on social media. Unfortunately, SBI research is often constrained by the lack of publicly-available, real-world data for experimental activities. In this paper, we describe our experience in extracting social data and processing them through an enrichment pipeline for brand analysis. As a first step, we collect texts from social media and we annotate them based on predefined metrics for brand analysis, using features such as sentiment and geolocation. Annotations rely on various learning and natural language processing approaches, including deep learning and geographical ontologies. Structured data obtained from the annotation process are then stored in a distributed data warehouse for further analysis. Preliminary results, obtained from the analysis of three well known ICT brands, using data gathered from Twitter, news portals, and Amazon product reviews, show that different evaluation metrics can lead to different outcomes, indicating that no single metric is dominant for all brand analysis use cases.

6. ICPE 2021

   

   An Exploratory Study of the Impact of Parameterization on JMH Measurement Results in Open-Source Projects

   Hazem Samoaa and Philipp Leitner

   In ICPE ’21: ACM/SPEC International Conference on Performance Engineering, Virtual Event, France, April 19-21, 2021, Oct 2021

   DOI
7. IEEE Software

   

   Serverless Applications: Why, When, and How?

   Simon Eismann, Joel Scheuner, Erwin Eyk, Maximilian Schwinger, Johannes Grohmann, Nikolas Herbst, Cristina L. Abad, and Alexandru Iosup

   IEEE Software, Oct 2021

   DOI
8. TSE

   

   What’s Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks

   Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak

   IEEE Transactions on Software Engineering, Oct 2021

   DOI
9. TSE

   

   Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

   Giovanni Grano, Christoph Laaber, Annibale Panichella, and Sebastiano Panichella

   IEEE Transactions on Software Engineering, Oct 2021

   DOI

2020

1. Topology-Aware Continuous Experimentation in Microservice-Based Applications

   Gerald Schermann, Fábio Oliveira, Erik Wittern, and Philipp Leitner

   In Service-Oriented Computing - 18th International Conference, ICSOC 2020, Dubai, United Arab Emirates, December 14-17, 2020, Proceedings, Oct 2020

   DOI
2. Function-as-a-Service performance evaluation: A multivocal literature review

   Joel Scheuner and Philipp Leitner

   Journal of Systems and Software, Oct 2020

   DOI
3. FSE 2020

   

   Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality

   Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner

   In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, Oct 2020

   DOI
4. FSE 2020

   

   An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective

   Linda Erlenhov, Francisco Gomes de Oliveira Neto, and Philipp Leitner

   In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, Oct 2020

   DOI
5. Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark

   Erwin Eyk, Joel Scheuner, Simon Eismann, Cristina L. Abad, and Alexandru Iosup

   In Companion of the ACM/SPEC International Conference on Performance Engineering, Apr 2020

   DOI

2019

1. Studying the impact of CI on pull request delivery time in open source projects — a conceptual replication

   Yunfang Guo and Philipp Leitner

   PeerJ Computer Science, Apr 2019

   DOI
2. Transpiling Applications into Optimized Serverless Orchestrations

   Joel Scheuner and Philipp Leitner

   In 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W), Jun 2019

   2nd Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf)

   DOI
3. Tutorial – Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench

   Joel Scheuner and Philipp Leitner

   In 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W), Jun 2019

   Tutorial co-located with ICAC/SASO

   DOI
4. Current and Future Bots in Software Development

   Linda Erlenhov, Francisco Gomes, Riccardo Scandariato, and Philipp Leitner

   In Proceedings of the 1st International Workshop on Bots in Software Engineering (BotSE) at ICSE 2019, Jun 2019

   DOI
5. Cloud Futurology

   Blesson Varghese, Philipp Leitner, Suprio Ray, Kyle Chard, Adam Barker, Yehia Elkhatib, Herry Herry, Cheol-Ho Hong, Jeremy Singer, Fung Po Tso, Eiko Yoneki, and Mohamed-Faten Zhani

   IEEE Computer, Jun 2019

   DOI
6. Tutorial: Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench

   Joel Scheuner and Philipp Leitner

   In Companion Proceedings of the 10th ACM/SPEC International Conference on Performance Engineering (ICPE), Mumbai, India, Jun 2019

   DOI

7. 

   Software microbenchmarking in the cloud. How bad is it really?

   Christoph Laaber, Joel Scheuner, and Philipp Leitner

   Empirical Software Engineering, Apr 2019

   Abs DOI

   Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to >\thinspace100%). However, executing test and control experiments on the same instances (in randomized order) allows us to detect slowdowns of 10% or less with high confidence, using state-of-the-art statistical tests (i.e., Wilcoxon rank-sum and overlapping bootstrapped confidence intervals). Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns in cloud environments.

8. Continuous Software Performance Assessment: Detecting Performance Problems of Software Libraries on Every Build

   Christoph Laaber

   In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Beijing, China, Apr 2019

   DOI
9. A Mixed-Method Empirical Study of Function-as-a-Service Software Development in Industrial Practice

   Philipp Leitner, Erik Wittern, Josef Spillner, and Waldemar Hummer

   Journal of Systems and Software, Apr 2019

   DOI
10. ICSE 2019

    

    Interactive Production Performance Feedback in the IDE

    Jürgen Cito, Philipp Leitner, Martin Rinard, and Harald Gall

    In Proceedings of the 41st International Conference on Software Engineering (ICSE), Apr 2019

    DOI PDF
11. Cachematic -– Automatic Invalidation in Application-Level Caching Systems

    Viktor Holmqvist, Jonathan Nilsfors, and Philipp Leitner

    In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (ICPE), Apr 2019

2018

1. Visualizing Test Diversity to Support Test Optimisation

   F. G. De Oliveira Neto, R. Feldt, L. Erlenhov, and J. B. D. S. Nunes

   In 2018 25th Asia-Pacific Software Engineering Conference (APSEC), Dec 2018

   DOI
2. Structured Information on State and Evolution of Dockerfiles on Github

   Gerald Schermann, Sali Zumberi, and Jürgen Cito

   In Proceedings of the 15th International Conference on Mining Software Repositories, Gothenburg, Sweden, Dec 2018

   DOI
3. Search-Based Scheduling of Experiments in Continuous Deployment

   G. Schermann and P. Leitner

   In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), Sep 2018

   DOI
4. Estimating Cloud Application Performance Based on Micro-Benchmark Profiling

   Joel Scheuner and Philipp Leitner

   In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), Jul 2018

   DOI
5. An Evaluation of Open-source Software Microbenchmark Suites for Continuous Performance Assessment

   Christoph Laaber and Philipp Leitner

   In Proceedings of the 15th International Conference on Mining Software Repositories, Gothenburg, Sweden, Jul 2018

   DOI
6. We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation

   Gerald Schermann, Jürgen Cito, Philipp Leitner, Uwe Zdun, and Harald C. Gall

   Journal of Information and Software Technology, Jul 2018

   DOI
7. A Cloud Benchmark Suite Combining Micro and Applications Benchmarks

   Joel Scheuner and Philipp Leitner

   In Companion of the 2018 ACM/SPEC International Conference on Performance Engineering, Berlin, Germany, Jul 2018

   DOI
8. PerformanceHat: Augmenting Source Code with Runtime Performance Traces in the IDE

   Jürgen Cito, Philipp Leitner, Christian Bosshard, Markus Knecht, Genc Mazlami, and Harald C. Gall

   In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, Gothenburg, Sweden, Jul 2018

   DOI
9. Continuous Experimentation: Challenges, Implementation Techniques, and Current Research

   G. Schermann, J. Cito, and P. Leitner

   IEEE Software, Mar 2018

   DOI

2017

1. Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

   Andre Hoorn, Pooyan Jamshidi, Philipp Leitner, and Ingo Weber

   CoRR, Mar 2017
2. Optimized IoT service placement in the fog

   Olena Skarlat, Matteo Nardelli, Stefan Schulte, Michael Borkowski, and Philipp Leitner

   Service Oriented Computing and Applications, Dec 2017

   DOI
3. A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

   Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella

   In Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME), Dec 2017

   DOI
4. Extraction of Microservices from Monolithic Software Architectures

   Genc Mazlami, Jürgen Cito, and Philipp Leitner

   In Proceedings of the 24th IEEE International Conference on Web Services (ICWS) - Applications Track, Dec 2017

   DOI
5. An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software

   Thomas Rausch, Waldemar Hummer, Philipp Leitner, and Stefan Schulte

   In Proceedings of the 14th International Conference on Mining Software Repositories, Buenos Aires, Argentinia, Dec 2017
6. An Empirical Analysis of the Docker Container Ecosystem on GitHub

   Jürgen Cito, Gerald Schermann, Erik Wittern, Philipp Leitner, Sali Zumberi, and Harald Gall

   In Proceedings of the 14th International Conference on Mining Software Repositories, Buenos Aires, Argentinia, Dec 2017
7. (h | g)opper: Performance History Mining and Analysis

   Christoph Laaber and Philipp Leitner

   In Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE) - Poster and Demonstration Track, L’Aquila, Italy, Dec 2017

   DOI
8. An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications

   Christian Davatz, Christian Inzinger, Joel Scheuner, and Philipp Leitner

   In Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, Madrid, Spain, Dec 2017

   DOI
9. Context-Based Analytics – Establishing Explicit Links between Runtime Traces and Source Code

   Jürgen Cito, Faìbio Oliveira, Philipp Leitner, Priya Nagpurkar, and Harald Gall

   In Proceedings of the 38th International Conference on Software Engineering (ICSE’17) - Software Engineering in Practice (SEIP) Track, Dec 2017
10. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects

    Philipp Leitner and Cor-Paul Bezemer

    In Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE), Dec 2017

    DOI

2016

1. Probr – A Generic and Passive WiFi Tracking System

   Joel Scheuner, Genc Mazlami, Dominik Schöni, Sebastian Stephan, Alessandro De Carli, Thomas Bocek, and Burkhard Stiller

   In Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN), Dec 2016

   DOI
2. Probr Demonstration – Visualizing Passive WiFi Data

   Joel Scheuner, Genc Mazlami, Dominik Schöni, Sebastian Stephan, Alessandro De Carli, Thomas Bocek, and Burkhard Stiller

   In Proceedings of the 41st IEEE Conference on Local Computer Networks (LCN), Dec 2016
3. Resource Provisioning for IoT Services in the Fog

   Olena Skarlat, Stefan Schulte, Michael Borkowski, and Philipp Leitner

   In 9th IEEE International Conference on Service-Oriented Computing and Applications (SOCA), Dec 2016

   DOI
4. Modelling and Managing Deployment Costs of Microservice-Based Cloud Applications

   Philipp Leitner, Jürgen Cito, and Emanuel Stöckli

   In Proceedings of the 9th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), Dec 2016
5. Bifrost - Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies

   Gerald Schermann, Dominik Schöni, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 2016 ACM/IFIP/USENIX Middleware Conference, Best Student Paper Award, Trento, Italy, Dec 2016
6. TemPerf: Temporal Correlation Between Performance Metrics and Source Code

   Jürgen Cito, Genc Mazlami, and Philipp Leitner

   In Proceedings of the 2nd International Workshop on Quality-Aware DevOps, Saarbrücken, Germany, Dec 2016

   DOI
7. Towards Quality Gates in Continuous Delivery and Deployment

   Gerald Schermann, Jürgen Cito, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 24TH IEEE International Conference on Program Comprehension (ICPC), Best Short Paper Award, Austin, Texas, Dec 2016
8. TOIT

   

   Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds

   Philipp Leitner and Jürgen Cito

   ACM Transactions on Internet Technology, Apr 2016

   DOI PDF

2015

1. All the Services Large and Micro: Revisiting Industrial Practice in Services Computing

   Gerald Schermann, Jürgen Cito, and Philipp Leitner

   In Proceedings of the 11th International Workshop on Engineering Service Oriented Applications (WESOA’15), Goa, India, Apr 2015
2. A Framework for a Cost-Efficient Cloud Ecosystem

   Eliot Salant, Philipp Leitner, Karl Wallbom, and James Athes

   In eChallenges e-2015 Conference, Vilnius, Lithuania, Apr 2015
3. Bursting With Possibilities – an Empirical Study of Credit-Based Bursting Cloud Instance Types

   Philipp Leitner and Joel Scheuner

   In Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), Limassol, Cyprus, Apr 2015
4. Intent, Tests, and Release Dependencies: Pragmatic Recipes for Source Code Integration

   Martin Brandtner, Philipp Leitner, and Harald C. Gall

   In Proceedings of the 15th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM), Bremen, Germany, Apr 2015
5. CloudWave - Leveraging DevOps for Cloud Management and Application Development

   Dario Bruneo, Aryan Dadashi, Philipp Leitner, Boris Moltchanov, Francesco Javier Nieto De-Santos, Avi Miron, Eliot Salant, Amir Molzam Sharifloo, Karl Wallbom, and Chris Woods

   In Proceedings of the Fourth European Conference on Service-Oriented and Cloud Computing (ESOCC’15) - EU Projects Track, Taormina, Italy, Apr 2015
6. JCloudScale: Closing the Gap Between IaaS and PaaS

   Rostyslav Zabolotnyi, Philipp Leitner, Waldemar Hummer, and Schahram Dustdar

   ACM Transactions on Internet Technology, Jul 2015

   DOI
7. The Making of Cloud Applications – An Empirical Study on Software Development for the Cloud

   Jürgen Cito, Philipp Leitner, Thomas Fritz, and Harald C. Gall

   In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE), Bergamo, Italy, Jul 2015
8. Onward! @ SPLASH 2015

   

   Runtime Metric Meets Developer - Building Better Cloud Applications Using Feedback

   Jürgen Cito, Philipp Leitner, Harald C. Gall, Aryan Dadashi, Anne Keller, and Andreas Roth

   In Proceedings of the 2015 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2015), Pittsburgh, Pennsylvania, United States, Jul 2015

   PDF
9. Identifying Web Performance Degradations Through Synthetic and Real-user Monitoring

   Jürgen Cito, Devan Gotowka, Philipp Leitner, Ryan Pelette, Dritan Suljoti, and Schahram Dustdar

   Journal of Web Engineering, Nov 2015
10. SPEEDL - A Declarative Event-Based Language for Cloud Scaling Definition

    Rostyslav Zabolotnyi, Philipp Leitner, Stefan Schulte, and Schahram Dustdar

    In Proceedings of the Future of Software Engineering For and In Cloud, Visionary Track of IEEE Services, Nov 2015
11. Profile-based View Composition in Development Dashboards

    Martin Brandtner, Philipp Leitner, and Harald Gall

    In Proceedings of the 17th Workshop Software-Reengineering & Evolution (WSRE), Nov 2015
12. Discovering Loners and Phantoms in Commit and Issue Data

    Gerald Schermann, Martin Brandtner, Sebastiano Panichella, Philipp Leitner, and Harald C. Gall

    In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, Florence, Italy, Nov 2015
13. Cloud WorkBench: Benchmarking IaaS Providers Based on Infrastructure-as-Code

    Joel Scheuner, Jürgen Cito, Philipp Leitner, and Harald C. Gall

    In Proceedings of the 24th International Conference on World Wide Web, Florence, Italy, Nov 2015

    DOI
14. SQA-Profiles: Rule-Based Activity Profiles for Continuous Integration Environments

    Martin Brandtner, Sebastian C. Müller, Philipp Leitner, and Harald Gall

    In Proceedings of the 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER’15), Nov 2015
15. Comparing and Combining Predictive Business Process Monitoring Techniques

    Andreas Metzger, Philipp Leitner, Dragan Ivanovic, Eric Schmieders, Rod Franklin, Manuel Carro, Schahram Dustdar, and Klaus Pohl

    IEEE Transactions on Systems, Man, and Cybernetics: Systems, Feb 2015

    DOI

2014

1. CloudCom 2014

   

   Cloud WorkBench - Infrastructure-as-Code Based Cloud Benchmarking

   Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall

   In Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’14), Feb 2014

   DOI PDF
2. WPress: Benchmarking Infrastructure-as-a-Service Cloud Computing Systems for On-line Transaction Processing Applications

   Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung

   In Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC), Feb 2014
3. Profiling-Based Task Scheduling for Factory-Worker Applications in Infrastructure-as-a-Service Clouds

   Rostyslav Zabolotnyi, Philipp Leitner, and Schahram Dustdar

   In Proceedings of the 40th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Feb 2014
4. CloudWave: where Adaptive Cloud Management Meets DevOps

   Dario Bruneo, Thomas Fritz, Sharon Keidar-Barner, Philipp Leitner, Francesco Longo, Clarissa Marquezan, Andreas Metzger, Klaus Pohl, Antonio Puliafito, Danny Raz, Andreas Roth, Eliot Salant, Itai Segall, Massimo Villari, Yaron Wolfsthal, and Chris Woods

   In Proceedings of the Fourth International Workshop on Management of Cloud Systems (MoCS 2014), Feb 2014
5. Identifying Root Causes of Web Performance Degradation Using Changepoint Analysis

   Jürgen Cito, Dritan Suljoti, Philipp Leitner, and Schahram Dustdar

   In Web Engineering: 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings, Feb 2014

   DOI
6. A Note on Software Tools and Techniques for Monitoring and Prediction of Cloud Services

   Rajiv Ranjan, Rajkumar Buyya, Philipp Leitner, Armin Haller, and Stefan Tai

   Software: Practice and Experience, Jul 2014

   DOI