Scaling Sustainable Computing: Advanced ML Techniques for Enhancing the Efficiency in the SKA Regional Data Centers

The SKA Regional Centres Network (SRCNet) of globally distributed data centers will have to find innovative ways to store and disseminate some 600Pbytes/year of incoming data products from SKAO while providing the necessary federated computing resources to the research community to analyze it. As the amount SKAO data continues to grow over its 50+ years lifespan, the salability of SRCNet's capabilities to cater data and computing services to the scientific community will be critical to the meet SKAO’s Science and Sustainability Goals.

In this talk, we will present new ML-based techniques to exploit trade-offs between execution time and resource utilization toward more scalable and sustainable cloud systems. These techniques address the challenge of accurate performance prediction and run-time optimization of resources in servers and data centers for maximizing energy efficiency. In particular, CloudProphet will be presented as a robust ML-based method that identifies each type of running application in VMs and predicts the performance level of concurrently executed cloud applications to optimize the use of resources in the data center. Then, GreenDVFS uses the inputs of CloudProphet to successfully determine the best tuning points for the DVFS knobs in cloud servers and racks to maximize energy efficiency under performance constraints. Finally, we can combine the results of GreenDVFS with novel renewable energy sources for multi-location public clouds and infrastructures through a new workflow with a novel Green-performance degradation manager called ECOGreen.

This new ML-based flow effectively exploits trade-offs between execution time and resource utilization toward more scalable and sustainable cloud systems (20% less energy, 35% less temperature, and 48% increase of renewables). These techniques, together with the development of novel hardware architectures designed specifically for energy efficiency in the radio-interferometry domain, have the potential to help achieve the sustainability goals of SKAO while guaranteeing QoS for its users.