NGAS is not only an archive system for online archiving of data. It provides the entire infrastructure needed to handle and process data during the entire life-time of the data. In other words: NGAS is an Archive Logistics System. Since the system is very adaptable, it is possible to combine it with different kind of storage technologies below or even other archive systems, still having the same high-level interface to the archive system.

An NG/AMS Server is a multithreaded application, which can execute many requests (HTTP request) in parallel. It is for instance possible to execute Archive Push and Pull Requests in parallel with Retrieve/Processing Requests and to request information about the system at the same time. Of course, if too many parallel requests are executed, the performance of the system will go down. A configuration parameter is used to specify the maximal number of parallel requests that can be handled. If more are received, additional requests will be rejected.

The 'infrastructure' used internally by NG/AMS for managing the data files and other files to be handled is shown in the figure above. Some information will be stored on the system disk(s) of the NGAS Node. This is the Back-Log Buffer, the Processing Area, the Global Bad Files Directory, log files and temporary files used e.g. in connection with the Data Consistency Checking or Janitor Thread for the DB synchronization based on the DB Snapshot Feature. The connected storage media is typically organized in pairs of two disks, a Main Disk and a Replication Disk. On the Main Disk a Staging Area can be found where files are stored until they are moved to their final destination. Also a Bad Files Area is available for files, which were classified as bad. Also the storage area is of course located on the disk. The filename of the files in the DB is given relative to the mount point of a storage disk. On the Main Disk also the NgasDiskInfo XML document can be found. It contains the information about the disk from the DB. It is used by the system to mark the disk as a registered NGAS Disk. Also the directory ".db" is found on the disk (not shown). It contains the information in connection with the DB Snapshot feature. The Replication Disk has the same structure as the Main Disk, although no Staging Area and Bad Files Area are located on this disk.

Since the communication protocol of NG/AMS is based on HTTP, it is possible to communicate with the server with any WEB browser or HTTP client. Even telnet can be used to interact with the server. Certain commands require a special handling like e.g. the Archive Push Request where the data of the file must be packed in the body of the HTTP request.

The synchronization of the DB holdings constitute a special problem for a distributed system like NGAS. A unidirectional DB replication from Chile (La Silla/Paranal) is provided. Problem is that replication is only implemented from Chile to Garching. I.e., when there are changes in the DB in Garching, this is not reflected in the DBs in Chile. This problem has been solved by implementing an internal DB synchronization scheme within The NG/AMS Server keeps a DB Snapshot file on each disk (<Mount Point>/.db/NgasFiles.bsddb). This is a file based DB (Berkeley DB/DBM). It contains the DB information about the files archived on that disk. Whenever  changes are introduced in the DB concerning files (e.g. when files are deleted), the DB Snapshot is updated accordingly. When the disk is put online at another NGAS site, the DB there is updated according to the DB Snapshot. I.e., if files are registered in the local DB but not found in the DB Snapshot + not stored on the disk, the entries are deleted from the DB. This is used e.g. for the disk recycling where files/disks are removed from the DB in Garching and the disk re-appears at the observatory site. The DB Snapshot can also be used as a sort of table of contents for the files stored on the disk.

- The NG/AMS Server receives the HTTP request and creates a new thread to handle this. - Based on the type of the data the best suitable Target Disk Set for the file is found. - From the type of data the Data Archiving Plug-In (DAPI) to handle this kind of data is determined. - The DAPI checks the file, extracts the information needed by NG/AMS and returns this to the calling thread. - It replicates the files (if specified) and updates the information about the Replication File in the DB. - If a response was not generated at the beginning of the command handling this is generated now. When updating the DB the DB access class generates temporary snapshot files reflecting the changes in the DB. The Janitor Thread updates the DB Snapshot in background.

The configuration used by an NG/AMS Server is specified by the parameter "-cfg". This configuration must at least contain the definition of the DB connection if the rest of the configuration is stored in the For the configuration stored in the DB it is possible to allocate an ID for each group of parameters (XML element). These IDs can then be combined to form a complete configuration referred to by one name. The overall names + list of Configuration Group IDs are stored in the table ngas_cfg.

In order to ensure that the data in the archive is in a good condition, it is necessary to constantly monitor the condition of the data. The Data Consistency Checking (DCC) Thread takes care of this task. It is executed periodically (defined by a configuration parameter) and carries out the check of the data. It is checked for each file if it is accessible, if the size is as expected and or if the checksum is given if the checksum is correct. It is also checked if files are found on the disk, which are not supposed to be there. If discrepancies are found, an Email Notification Message is send to the subscribers of this specified in the configuration file. It is possible to specify how much load the DCC Thread should consume (Priority) and to specify the maximum number of checking sub-threads that can run simultaneously. In principle one such sub-thread per disk is created.

An important feature of the NG/AMS Server is the ability to execute in a cluster environment. The same SW is used both for the online archiving and for the archive cluster facility. If a private network is used within the cluster, it is possible to define one of the nodes as master. This will then be the main contact point for external clients when e.g. retrieving data. The proxy server will retrieve the file from its actual location and return it to the  requestor. In principle the file can be stored on any NGAS Node accessible via HTTP. I.e., also files stored on nodes at the observatory sites can be retrieved by a proxy server. In the example above, the file is stored on a sub-node in a sub-cluster within the  NGAS Cluster. The contacted node identifies the best suitable file for the request. In this case it identifies that the file is stored in another private network and contacts the Main Node of the sub-cluster. The sub-cluster main node locates the file and requests this from the sub-node. The file is subsequently send back to the contacted main node and streamed through to the requestor. Before deciding which file to choose for a query, it is checked if the file is actually accessible.

A simpler example of the operation in cluster mode is to use still a main node, which does not act as proxy. The main node will then send back HTTP re-direction responses to the requestor which subsequently must pick up the file with a RETRIEVE Request sent directly to the node hosting the file. I.e., there is no specific structure in the cluster and any node could in principle act as file locater. Disadvantage is that every node needs to be accessible for the clients. The advantage is a better performance. It is also quite easy to set up such a cluster, as any computer connected to the network with the NG/AMS Server running on it (and registered in the NG/AMS DB) may be transformed into an NGAS Node. It is possible to have a combination of the 'proxy mode cluster' and the ‘flat’ cluster structure.

A more HW-oriented view of the NGAS Cluster is shown in this slide. It is probably the structure, which will be used in the near future for the Garching NGAS Cluster. Now a combination of the two concepts above is used. The cluster has one entry point as such. However, two Main Nodes are available for higher availability and for better performance. The client APIs can multiplex between the Main Nodes. The data server nodes (NCUs) are divided into sub-clusters, although this partition is more logical than physical. The archiving capability is provided as an Archiving Cluster. The Main Nodes can multiplex between the Archiving Units to distribute the load better and for higher availability when acting as proxy servers for handling incoming Archive Requests. Backup of NGAS data now done by keeping an offline replication archive NGAS Disks at the observatory sites. This will be replaced by an offline tape based replication archive.

when a RETRIEVE Command is send to an NG/AMS Server it is possible to specify that the NG/AMS Server should execute a Data Processing Plug-In (DPPI) on the file. If this is the case, only the result/output of the DPPI will be send back to the requestor rather than the complete file. The actual processing of the file is carried out on the node where the file is stored. I.e., if a Main Node is handling the request and acting as proxy, it will forward the Retrieve Request to the node hosting the file. The processing will take place on that node. Client applications may issue parallel Retrieve Requests and in this way use the NGAS Cluster as a parallel number cruncher. In addition, less network traffic is needed.

The complete list of commands provided shown in this slide. A command is in principle like a  CGI script for a 'normal' HTTP server. The actual command (CGI script) is executed as a sub-thread in the NG/Server rather than e.g. as an external CGI script.

The NGAS WEB Interfaces are an essential part of the operation system. These are used extensively by the operators at each site for monitoring the system and for finding various items (data files, disks, nodes). For now not much interaction with the system is possible. The WEB interfaces should be enhanced.

The NGAS Project is a technical challenge, but is maybe even more a logistic challenge. In the slide above a simplified view of the operation environment is show.