Session

DescriptionVarious data-intense scientific domains must deal with exabytes of data before they reach the exaflop. Data management at these extreme scales is challenging and covers not only pre-processing, data production, and data analysis workflows. While there are many research approaches and science databases that aim to manage data and improved their limits over time, practitioners still struggle to manage their data in the petabyte era. For instance, achieving high performance and providing means to easily localize data upon request. With billions of files, the scalability of the manual and fine-grained data management in HPC environments reaches its limitations. Various domain-specific solutions have been developed that mitigate performance and management issues enabling data management in the petabyte era. However, due to new storage technologies and heterogeneous environments, the challenges increase and so does the development effort for individual solutions. In this minisymposium, speakers from environmental science (Met Office and ECMWF), CERN, and the Square Kilometre Array will address this matter for different domains; each speaker will present the challenges faced in their scientific domain today, give an outlook for the future, and present state-of-the-art approaches the community follows to mitigate the data deluge.