Today's machine learning systems are growing wildly in size every year. Of course, most of this code is not actually required for the intended use cases of the system - it is bloat. This code bloat can have a significant impact on the performance of ML systems, as it can increase the memory footprint, the startup time, and the energy consumption of the system - and, of course, the larger the code base, the larger the attack surface for security vulnerabilities.
In our research, we study the existence of code bloat in machine learning systems, and how it can be measured and reduced (Zhang et al., 2024). We find that most machine learning containers in most use cases exhibit 50% or more bloat.
Example growth of a real-life machine learning system
In collaboration with the Computer and Network Systems research unit at Chalmers, we built BLAFS, a bloat-aware container filesystem that removes unused files at runtime while guaranteeing the debloated container still works correctly (Zhang et al., 2026). Accessed files move into a debloating layer as the container runs, and, similar to garbage collection, files that are never accessed get removed. An optional reloading layer fetches anything mistakenly removed from a remote cache on demand.
Across the top 20 most-downloaded Docker Hub containers, four ML containers, and a 10-function serverless benchmark suite, BLAFS reduces container image sizes by up to 95% and cold starts by up to 68%. In a security-hardened mode it removes up to 89% of known CVEs, where the two state-of-the-art debloating tools we compared against, including docker-slim, largely fail on the same workloads.
Containers have become the standard for deploying applications in many cloud systems due to its convenience. However, this convenience leads to significant container bloat, i.e., unused files that inflate container image sizes, increase provisioning times, waste resources and introduce security vulnerabilities. Bloat is particularly problematic in serverless and edge computing scenarios, where resources are constrained, and performance is critical, and for microservice applications where rapid scaling is key to meet performance targets. However, existing container debloating tools are often limited in both effectiveness and robustness. In this paper, we propose BLAFS, a bloat-aware container filesystem that removes bloat while guaranteeing the correct operation of the debloated containers. BLAFS addresses bloat at the filesystem level by introducing new layers in the filesystem to enable debloating. During runtime, accessed files are moved to the debloating layers, and then similar to garbage collection mechanisms, BLAFS removes files that are not accessed during runtime. An optional reloading layer fetches files from a remote cloud cache on-demand if the files are mistakenly removed. We discuss how BLAFS can be used in different deployment scenarios and for different use-cases including container security-hardened and a dynamic deployment mode where the target is improved provisioning performance. We evaluate BLAFS performance using the top 20 downloaded containers from DockerHub, four ML containers, and SEBS, a Serverless Benchmark containing 10 serverless functions and compare its performance against two state-of-the-art debloating tools. Our evaluation shows that BLAFS reduces container sizes by up to 95% and cold-starts by up to 68%. In the security-hardened mode, BLAFS removes up to 89% of CVEs while the two state-of-the-art debloating tools fail on most of the workloads. We identify their limitations, and show how BLAFS provides a more principled approach to debloating. Additionally, when combined with lazy-loading snapshotters, BLAFS improves provisioning efficiency, reducing conversion times by up to 93% and provisioning times by up to 19%.
Today’s software is bloated with both code and features that are not used by most users. This bloat is prevalent across the entire software stack, from operating systems and applications to containers. Containers are lightweight virtualization technologies used to package code and dependencies, providing portable, reproducible and isolated environments. For their ease of use, data scientists often utilize machine learning containers to simplify their workflow. However, this convenience comes at a cost: containers are often bloated with unnecessary code and dependencies, resulting in very large sizes. In this paper, we analyze and quantify bloat in machine learning containers. We develop MMLB, a framework for analyzing bloat in software systems, focusing on machine learning containers. MMLB measures the amount of bloat at both the container and package levels, quantifying the sources of bloat. In addition, MMLB integrates with vulnerability analysis tools and performs package dependency analysis to evaluate the impact of bloat on container vulnerabilities. Through experimentation with 15 machine learning containers from TensorFlow, PyTorch, and Nvidia, we show that bloat accounts for up to 80% of machine learning container sizes, increasing container provisioning times by up to 370% and exacerbating vulnerabilities by up to 99%.
