ACS provides support for the transport of bulk data [RD01 - 7.1. Bulk data transfer], to be used for science data.
A data supplier produces a continuous stream of data for a specific bulk data consumerComponent. Typical example is the Correlator sending continuous streams of data to the Archive.
The basic operations are as follows:
The supplier gets access to the consumer Component
Opens a connection.
Data is sent by the supplier to the consumer until
The supplier closes the connection
A data supplier produces bulks of data as separate, non continuous streaming, entities. Typical example is a data processing Component sending images to another Component for further processing. Everything works like in the previous case, but for the fact that the communication protocol must be able to identify separate bulks of data, like a single image.
An application needs streaming data published by a Stream Provider Component. There can be multiple clients. Typical example is a GUI connecting to a CCD camera. Multiple GUIs can display the image from the same Camera.
The basic operations are a follows:
The supplier Component is available for publishing data
The client opens a connection with the supplier
The supplier sends data to the client(s) until
The client closes the connection
An application needs to retrieve discrete bulks of data as separate, non continuous streaming, entities. Typical example is retrieval of images from the archive. Everything works like in the previous case, but for the fact that the communication protocol must be able to identify separate bulks of data, like a single image.
It shall be possible to handle the actual transfer of data with communication protocols more efficient than CORBA IIOP, in particular for high volume streams.
When there are multiple clients for the data published we have implemented a service architecture in which the data supplier sends the data to just one Distributor which, in turn, sends them to a number of connecting clients. This decouples the load due to the increasing number of clients from the supplier.
Precise performance requirements have been collected and verified with the ACS Bulk Data system implementation based on the CORBA Audio Video streaming service. Tests have demonstrated that the ACS implementation can deliver 700 – 800 Mbit/s to three consumers simultaneously.
Iterators on normal IDL methods
Notification Channel
Audio Video Streaming Service
The first two options are based on the IIOP transport protocol and therefore suffer from performance limitations, although tests available in the literature show that properly designed buffering limits these problems. Option 1 is better suited for discrete bulk data while option 2 is better suited for streaming.
Option 3 is based on the Audio Video Streaming Service [RD42] defined by CORBA. This specification aims at the streaming and transfer of large amounts of data and satisfies the requirements expressed in [RD01 - 7.1.1 Image pipeline]. The handshaking protocol is defined using CORBA IDL Media Control interfaces, but the actual data transfer goes out of band and does not use (but could use) CORBA to transport data. TAO provides an implementation and provides transport over TCP and UDP with excellent performance[RD43].
Stream
A stream represents
continuous media transfer between two or more entities.
Flow
A stream can be composed
of many flows.
For example a video camera stream has a video and
an audio flow.
As an example strictly related to ALMA, the
Correlator will have one stream to send bulk data to the Archive,
but this would be split in up to 8 flows with at least 2 always
present: 1 per subarray, 1 for spectral data and 1 for channel
average.
Each flow can have independent quality of service
parameters and each flow can have an independent direction (upload
or download).
Frames
A frame is a unit of data sent over a flow.
A
frame of a video is a typical example.
A set of data published by
the Correlator can be mapped to a frame as well.
The basic A/V
service does not have the concept of frame. The SFP (Simple Flow
Protocol), on top of the physical transport protocol, introduces the
concept of frame in the A/V service and protocol messages to
identify and number the frames are added to the actual data.
There is no implementation of CORBA A/V service for Java or Python. The producers and consumers of the bulk data stream (Correlator, Control, TelCal, Pipeline and Archive) all use C++ implementations of the bulk data senders and receivers as ACS components. Java and Python applications can access these components via the normal ACS/CORBA interfaces. There is therefore no requirement for implementation of CORBA A/V in languages other than C++. For example, image data processing typically takes place between C++ Components without requiring high performance image transfer to Java or Python Components.