Publication

Prototyping a Bulk Data Management System for CTAO with Rucio

2025

Abstract

Bulk Data Management, including the long-term archiving of massive datasets, is critical for advancing high-energy gamma-ray astrophysics research by ensuring data accessibility and scientific reproducibility. Within the Cherenkov Telescope Array Observatory (CTAO) [1], managing and preserving petabyte-scale data poses unique challenges. To address these challenges, we present our prototyping efforts for the Bulk Data Management System (BDMS), a key sub-system of CTAO’s Data Processing and Preservation System (DPPS) designed for long-term preservation. BDMS leverages Rucio [2] — an open-source data management system developed at CERN. BDMS manages the ingestion of data products on-site, replication of data between CTAO Data centers, ensure their long-term preservation, and provide an interface to ingest, query, and retrieve. We provide details on the BDMS architecture and its main functional blocks, namely: Ingest (including replication), Data Management (track preservation, and monitoring), Archival Storage, File Query and Access, and BDMS Administration. Our prototyping contributions include containerised deployment using Helm charts and continuous integration tests on a Kubernetes (K8s) cluster provided by DESY Computing/Data center; metadata handling by implementing a setup to extract and store metadata from raw (DL0: Data level 0) data products, thereby enabling high-level dataset queries. Finally, we provide details on current status and outline our future plans.

Official source

https://infoscience.epfl.ch/entities/publication/8afc8e76-a133-441c-aed0-20d34875b361

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

2025

Abstract

Official source

https://infoscience.epfl.ch/entities/publication/8afc8e76-a133-441c-aed0-20d34875b361

About this result

Related concepts (31)

Data management

Data management comprises all disciplines related to handling data as a valuable resource. The concept of data management arose in the 1980s as technology moved from sequential processing (first punched cards, then magnetic tape) to random access storage. Since it was now possible to store a discrete fact and quickly access it using random access disk technology, those suggesting that data management was more important than business process management used arguments such as "a customer's home address is stored in 75 (or some other large number) places in our computer systems.

Research data archiving

Research data archiving is the long-term storage of scholarly research data, including the natural sciences, social sciences, and life sciences. The various academic journals have differing policies regarding how much of their data and methods researchers are required to store in a public archive, and what is actually archived varies widely between different disciplines. Similarly, the major grant-giving institutions have varying attitudes towards public archival of data.

Digital preservation

In library and archival science, digital preservation is a formal endeavor to ensure that digital information of continuing value remains accessible and usable. It involves planning, resource allocation, and application of preservation methods and technologies, and it combines policies, strategies and actions to ensure access to reformatted and "born-digital" content, regardless of the challenges of media failure and technological change. The goal of digital preservation is the accurate rendering of authenticated content over time.

Related publications (14)

Prototyping a Bulk Data Management System for CTAO with Rucio

Chat with Graph Search

Supporting data: Soft-X-ray momentum microscopy of nonlinear magnon interactions

Euclid Quick Data Release (Q1): I. Data release overview

Training "Ethics in Research Data Management: Anonymization and Best Practices"