Comunicato Stampa

VLT Data Flow System Begins Operation

Building a Terabyte Archive at the ESO Headquarters

21 Giugno 1999

The site at Cerro Paranal in the dry Atacama desert in Northern Chile is one of the best locations for astronomical observations from the surface of the Earth. Each of the four 8.2-m telescopes is a technological marvel with self-adjusting optics placed in a gigantic mechanical structure of the utmost precision, continuously controlled by advanced soft- and hardware. A multitude of extremely complex instruments with sensitive detectors capture the faint light from distant objects in the Universe and record the digital data fast and efficiently as images and spectra, with a minimum of induced noise. And now the next crucial link in this chain is in place. A few nights ago, following an extended test period, the VLT Data Flow System began providing the astronomers with a steady stream of high-quality, calibrated image and spectral data, ready to be interpreted. The VLT project has entered into a new phase with a larger degree of automation. Indeed, the first 8.2-m Unit Telescope, ANTU, with the FORS1 and ISAAC instruments, has now become a true astronomy machine.

A smooth flow of data through the entire system

Already from the start of this project more than ten years ago, the ESO Very Large Telescope was conceived as a complex digital facility to explore the Universe. In order for astronomers to be able to use this marvellous research tool in the most efficient manner possible, the VLT computer software and hardware systems must guarantee a smooth flow of scientific information through the entire system.

This process starts when the astronomers submit well-considered proposals for observing time and it ends with large volumes of valuable astronomical data being distributed to the international astronomical community. For this, ESO has produced an integrated collection of software and hardware, known as the VLT Data Flow System (DFS) , that manages and facilitates the flow of scientific information within the VLT Observatory. Early information about this new concept was published as eso9612 and extensive tests were first carried out at ESOs 3.5-m New Technology Telescope (NTT) at La Silla, cf. eso9703 [1].

The VLT DFS is a complete (end-to-end) system that guarantees the highest data quality by optimization of the observing process and repeated checks that identify and eliminate any problems. It also introduces automatic calibration of the data, i.e. the removal of external effects introduced by the atmospheric conditions at the time of the observations, as well as the momentary state of the telescope and the instruments.

From Proposals to Observations

In order to obtain observing time with ESO telescopes, also with the VLT, astronomers must submit a detailed observing proposal to the ESO Observing Programmes Committee (OPC) . It meets twice a year and ranks the proposals according to scientific merit. More than 1000 proposals are submitted each year, mostly by astronomers from the ESO members states and Chile; the competition is fierce and only a fraction of the total demand for observing time can be fulfilled.

During the submission of observing proposals, DFS software tools available over the World Wide Web enable the astronomers to simulate their proposed observations and provide accurate estimates of the amount of telescope time they will need to complete their particular scientific programme.

Once the proposals have been reviewed by the OPC and telescope time is awarded by the ESO management according to the recommendation by this Committee, the successful astronomers begin to assemble detailed descriptions of their intended observations (e.g. position in the sky, time and duration of the observation, the instrument mode, etc.) in the form of computer files called Observation Blocks (OBs) . The software to make OBs is distributed by ESO and used by the astronomers at their home institutions to design their observing programs well before the observations are scheduled at the telescope.

The OBs can then be directly executed by the VLT and result in an increased efficiency in the collection of raw data (images, spectra) from the science instruments on the VLT. The activation (execution) of OBs can be done by the astronomer at the telescope on a particular set of dates (visitor mode operation) or it can be done by ESO science operations astronomers at times which are optimally suited for the particular scientific programme (service mode operation).

An enormous VLT Data Archive

The raw data generated at the telescope are stored by an archive system that sends these data regularly back to ESO headquarters in Garching (Germany) in the form of CD and DVD ROM disks. While the well-known Compact Disks (CD ROMs) store about 600 Megabytes (600,000,000 bytes) each, the new Digital Versatile Disks (DVD ROMs) - of the same physical size - can store up 3.9 Gigabytes (3,900,000,000 bytes) each, or over 6 times more.

The VLT will eventually produce more than 20 Gigabytes (20,000,000,000 bytes) of astronomical data every night, corresponding to about 10 million pages of text [2]. Some of these data also pass through "software pipelines" that automatically remove the instrumental effects on the data and deliver data products to the astronomer that can more readily be turned into scientific results.

Ultimately these data are stored in a permanent Science Archive Facility at ESO headquarters which is jointly operated by ESO and the Space Telescope European Coordinating Facility (ST-ECF). From here, data are distributed to astronomers on CD ROMs and over the World Wide Web. The archive facility is being developed to enable astronomers to "mine" the large volumes of data that will be collected from the VLT in the coming years. Within the first five years of operations the VLT is expected to produce around 100 Terabytes (100,000,000,000,000 bytes) of data.

It is difficult to visualize this enormous amount of information. However, it corresponds to the content of 50 million books of 1000 pages each; they would occupy some 2,500 kilometres of bookshelves!

The VLT Data Flow System enters into operation

Science operations with the first VLT 8.2-m telescope (ANTU) began on April 1, 1999. Following the first call for proposals to use the VLT in October 1998, the OPC met in December and the observing schedule was finalized early 1999. The related Observation Blocks were prepared by the astronomers in February and March. Service-mode observations began in April and by late May the first scientific programs conducted by ESO science operations were completed.

Raw data, instrument calibration information and the products of pipeline processing from these programs have now been assembled and packed onto CD ROMs by ESO science operations staff. On June 15 the first CD ROMs were delivered to astronomers in the ESO community.

This event marks the closing of the data flow loop at the VLT for the first time and the successful culmination of more than 5 years of hard work by ESO engineers and scientists to implement a system for efficient and effective scientific data flow. This was achieved by a cross-organization science operations team involving staff in Chile and Europe.

With the VLT Data Flow System, a wider research community will have access to the enormous wealth of data from the VLT. It will help astronomers to keep pace with the new technologies and extensive capabilities of the VLT and so obtain world-first scientific results and new insights into the universe.

Note

[1] A more technical description of the VLT Data Flow System is available in Chapter 10 of the VLT Whitebook.

[2] By definition, one "normal printed page" contains 2,000 characters.

Connect with ESO on social media