Maintaining magma – A ground-truth fuzzing benchmark

This project aims to improve Magma, a benchmark used to evaluate bug-finding tools in software. Our role was to find proofs, called Proof-of-Concept (PoC), confirming that the bugs used in the evaluation were reproducible. Reliable benchmarks are essential in software testing research, since they allow tools to be compared on a fair and consistent basis. Since Magma’s accuracy depends on the assumption that every bug in its library is reachable, finding PoCs that confirm this is crucial. Without reproducible PoCs, this assumption may not hold, leading to misleading results and limiting the reliability of the benchmark. Ensuring reproducibility strengthens the trustworthiness of evaluations and increases their value for all users. To create these PoCs, we first extracted potential PoCs in the form of input files by running tests on my targets. We then manually verified whether these files trigger the wanted bug. If not, we either constructed a PoC manually by modifying the previously found input file or we removed the bug from the benchmark, as it would mean that it is unreproducible. This process was often challenging and time-consuming, since some bugs lacked clear trigger conditions or required large modifications for the file to be adapted . This project focuses on improving Magma, a widely used benchmark for evaluating fuzzing tools, by enhancing the reproducibility of its injected bugs. Since Magma’s reliability depends on the assumption that all bugs are reachable, the absence of reproducible Proof-of-Concepts (PoCs) undermines its accuracy and can lead to misleading evaluation results. To address this, we developed a systematic process for creating and validating PoCs. Candidate inputs were first extracted through fuzzing campaigns and then tested against target programs. Inputs that failed to trigger the expected bug were either refined through manual debugging or, when necessary, discarded. This iterative process ensured that only reproducible bugs remained in the benchmark. In total, 7 PoCs were successfully implemented, with 5 more under development. These contributions strengthen the trustworthiness of Magma, enabling fairer and more consistent comparisons between fuzzing tools. Future work will focus on expanding the PoC collection and exploring automation, potentially with the help of large language models, to accelerate PoC generation and integration. In total, 7 PoCs were implemented, and 5 more are currently in progress. By validating and filtering Magma’s bug library, this work contributes to more accurate and reliable evaluations for bug-finding tools. The next steps of this research are further expanding the PoC collection and automating their construction and implementation using large-language models (LLMs), which would drastically accelerate this process.