can LLMs and agents make real software faster, and how would we know?
Active since 2024.
Functional correctness got a cheap verifier. Efficiency did not.
Unit tests are fast, binary, and trustworthy enough to run inside a training loop or inside an agent’s inner loop, which is why the correctness of LLM-generated code has improved so quickly. Performance has no equivalent. Benchmarks are slow, statistically fragile, expensive to run, and frequently do not even exercise the code path whose cost changed. As a result there is essentially no reward signal for execution cost anywhere in the stack: not in training, not in agent scaffolding, and not in the CI pipelines of most projects.
We think this makes the efficiency gap in machine-generated code structural rather than a temporary artifact of model scale, and that it will not close on its own.
Our first study in this theme asks the direct question: can LLMs optimize real software? (Yi et al., 2025) mines 65 optimization tasks from performance-critical open-source Java projects and validates every proposed patch against developer-written JMH benchmarks, comparing against the improvement the human developer actually achieved. The answer is more interesting than either camp expects.
Distribution of speedup scores for LLM-generated patches, by prompting strategy and by model. The red line is the original code. The blue line is the median developer solution.
Models solve a surprisingly large share of these hard engineering tasks, and most of their patches do make the code faster. But the spread is enormous, and on average they remain (far) behind the human developers whose commits we compare against, across every model and prompting strategy we tried. AI not only writes slow code, it writes unpredictable code, which for a production system is often even worse.
Effect sizes against the original code (circles) and against the developer's own patch (triangles). Every configuration beats the original. None beats the developer.
We also find that the popular algorithmic-puzzle benchmarks give a substantially more optimistic picture of LLM optimization ability than real production code does, which matters because most of what the field currently believes about machine efficiency rests on those benchmarks.
In our ongoing work in this theme, we are particularly interested in:
Execution-cost oracles. Can a performance signal be made fast, cheap, and trustworthy enough to sit inside a generation loop? This continues our work on predicting execution time without executing (Samoaa et al., 2022) and on making benchmark suites dramatically cheaper to run (Laaber et al., 2020). An oracle that cannot say “I don’t know” is worse than no oracle at all, so quantifying its uncertainty is part of the problem.
Whether the instruments can be trusted. Recent audits of the standard efficiency benchmarks suggest that a substantial share of their reference solutions are not actually faster than the code they replace. Validating a cost oracle is itself a research contribution.
Avoidance rather than optimization. Every benchmark in this space poses the same task: here is slow code, make it fast. Almost nobody asks the industrial question: an agent makes fifty commits to a repository over three months. Does the system get slower? That is harder, because there is no identified hotspot and no reference patch, and the effects are diffuse and cumulative.
Large Language Models (LLMs) can generate code, but can they generate fast code for complex, real-world software systems? In this study, we investigate this question using a dataset of 65 tasks mined from performance-critical open-source Java projects. Unlike prior studies, which focused on algorithmic puzzles, we conduct experiments on actual performance-sensitive production code and employ developer-written JMH benchmarks to rigorously validate performance gains against human baselines. Our results reveal a nuanced reality – although LLMs demonstrate a surprisingly high capability to solve these complex engineering problems, their solutions suffer from extreme volatility and still lag behind human developers on average. Consequently, we find that the current benchmarks based on algorithmic tasks yields an overly optimistic assessment of LLM capabilities. We trace this real-world performance gap to two primary limitations: first, LLMs struggle to autonomously pinpoint performance hotspots, and second, even with explicit guidance, they often fall short of synthesizing optimal algorithmic improvements. Our results highlight the need to move beyond static code generation towards more complex agent-based systems that are able to profile and observe runtime behavior for performance improvement.
Predicting the performance of production code prior to actual execution is known to be highly challenging. In this paper, we propose a predictive model, dubbed TEP-GNN, which demonstrates that high-accuracy performance prediction is possible for the special case of predicting unit test execution times. TEP-GNN uses FA-ASTs, or flow-augmented ASTs, as a graph-based code representation approach, and predicts test execution times using a powerful graph neural network (GNN) deep learning model. We evaluate TEP-GNN using four real-life Java open source programs, based on 922 test files mined from the projects’ public repositories. We find that our approach achieves a high Pearson correlation of 0.789, considerable outperforming a baseline deep learning model. Our work demonstrates that FA-ASTs and GNNs are a feasible approach for predicting absolute performance values, and serves as an important intermediary step towards being able to predict the performance of arbitrary code prior to execution.
Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality
Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner
In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, 2020
