DAI: How Pre-computation Speeds up Data Analysis

As data sizes continue to expand, the challenge of conducting meaningful analysis grows. Utilizing I/O (Input/Output) libraries, such as HDF5 (Hierarchical Data Format) and ADIOS2 (Adaptable IO System), facilitates the filtering of raw data, with prior research highlighting the advantages of dissecting these formats for enhanced metadata management. Our study introduces a novel data management technique aimed at boosting query performance for HPC analysis applications through the automatic precomputation of commonly used data characteristics, as identified by our user survey. The Data Analysis Interface (DAI), developed for the JULEA storage framework, not only enables querying this enriched metadata but also shows how domain-specific features can be integrated, demonstrating a potential improvement in query times by up to 22,000 times.