Joe Masters (National Radio Astronomy Observatory), Anastasia Alexov (University of Amsterdam), Mike Folk (HDF Group), Robert Hanisch (VAO), Gerd Heber (HDF Group), Michael Wise (ASTRON)
Abstract
We would like to update the astronomy community on our effort to deal with the demands of ever-increasing astronomical data size and complexity, using the Hierarchical Data Format, version 5 (HDF5) format.
HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable, extensible, and parallelizable, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes a 20-year history of tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.
NRAO, LOFAR and VOA have joined forces with the HDF Group to write an NSF grant, requesting funding to assist in this effort. We will discuss the proposal outline. The proposal will be reviewed in the coming year; our hope is to be awarded funding for a full time software engineer starting Oct. 2012, for three years. That developer, along with staff from NRAO, LOFAR, the IVOA and the HDF Group, will focus their activities on: (1) Developing a data model and specification derived from the work done on the LOFAR Interface Control Documents and IVOA standards, specifically for the Image Cube, (2) Developing a middle layer (and simplified API) to work with astronomical data, based on the HDF5 Library, (3) Implementing converter tools to/from HDF5 and FITS, (4) working with the Oak Ridge National Laboratory on expanding VisIt to adapt their 3d visualization tool for use by the astronomy community.
Topics of discussion will include: input from the community on their experiences with HDF5; input on data I/O statistics, data storage structure efficiency; toolsets which exist and/or can be adapted for HDF5; and, any general lessons learned from working with large and complex data. The organizers welcome contriutions to the BoF agenda on HDF5 and other file formats which can accommodate streaming data encapsulation on the order of 23TB/hr! We hope our effort paves the way towards new astronomical data encapsulation, which can be used by future ground and space-based projects.Paper ID: B2
|
