NGAS Release Cycle

NGAS Release Cycle

ESO Garching, 2007-10-05

J.Knudstrup

Revision: 1.0

Introduction

This document contains a description of how an NGAS release cycle is carried out (or was carried out until V3.1.1).

The release cycle has evolved over time, for instance in the beginning Linux and NGAS were installed manually and a snapshot ISO image created from this via Mindi/Mondo and installed on all machines. Now the OS is installed via a kickstart installation (from DVD) and NGAS via an installation tool from CDROM, USB stick etc, or from a WEB address.

This document describes the latest status of the release preparation as it has been carried out for NGAS V3.X.

The document does not contain a description of the NGAS environment per se, but this information could be added if of interest. Also not contained in this document are the preparation and handling of new configuration files and updating of existing configuration parameters.

Note, there was an earlier initiative to write down the installation procedure and to document the NGAS operational environment, but this was never completed due to the change from using Mondo/Mindi to a kickstart based installation. This draft document is named: “NGAS System Installation & Configuration Manual/VLT-MAN-19400-3104”, but is mostly obsolete.

Preparing the Distribution

To prepare an NGAS distribution, do as follows:

Log into an NGAS Node installed with the latest Linux distribution installed with the appropriate kickstart file (usually prepared by SEG/OTS-SOS).
Ensure that the correct version of the following modules have been checked out from CVS (SDD/DFS CVS repository) and are located in the “/opsw/packages/” directory:

ngams: The module containing the core system; consult http://www.eso.org/projects/dfs/dfs-shared/doc/19400/VLT-MAN-ESO-19400-2739-V3.PDF for more details.
ngasCfg: Module containing the various XML configuration file used by the operational NGAS installations.
ngasInstall: Contains tools used to managing the release cycle and installation of NGAS.
ngasPlugIns: Contains a number of plug-ins used for the operational scenario. Note, some plug-ins are also contained in “ngams/ngamsPlugIns”.
ngasSys: Contains various system tools and data. This includes the Zope DB and various resource files.
ngasUtils: Contains a set of utilities used by the operators to interact with the system; see also http://ngasdev3.hq.eso.org:7575/ngasUtils.src.html. (note, this machine may not be online continuesly).
pcc: Contains various basic tools, like the logging system used by the NGAS SW.

Execute the NGAS unit tests and check the result:

$ngasmgr> cd /opsw/packages/ngams/ngamsTest/

$ngasmgr> python ngamsTest.py -notifemail jknudstr@eso.org

Executing the test will take approximately 1 hour. Examine the result of the low level functional tests; the report will be printed out stdout and will be sent in the form of a report via email to the given address.

An example of such a test report is shown below:

Date: Mon, 18 Jun 2007 10:37:37 GMT

From: NGAS user <ngas@eso.org>

Message-Id: <200706181037.l5IAbbTt012829@ngasdev3.hq.eso.org>

Subject: NG/AMS FUNCTIONAL TESTS REPORT

To: undisclosed-recipients:;

NG/AMS FUNCTIONAL TESTS - TEST REPORT

Date: 2007-06-18T09:55:16.585

Host: ngasdev3

NG/AMS Version: v3.1.1/Beta1/2007-06-15T17:00:00

Running Test Suite: ngamsArchiveClientTest (#1/30) - 14.9 s - SUCCESS

Running Test Suite: ngamsArchiveCmdTest (#2/30) - 488.5s - SUCCESS

Running Test Suite: ngamsArchiveStressTest (#3/30) - 84.3 s - SUCCESS

Running Test Suite: ngamsAuthorizationTest (#4/30) - 22.2 s - SUCCESS

Running Test Suite: ngamsCClientTest (#5/30) - 287.8s - SUCCESS

Running Test Suite: ngamsCheckFileCmdTest (#6/30) - 44.0 s - SUCCESS

Running Test Suite: ngamsCloneCmdTest (#7/30) - 312.5s - SUCCESS

Running Test Suite: ngamsConfigCmdTest (#8/30) - 8.6 s - SUCCESS

Running Test Suite: ngamsConfigHandlingTest (#9/30) - 39.6 s - SUCCESS

Running Test Suite: ngamsDataCheckingThreadTest (#10/30) - 18.4 s - SUCCESS

Running Test Suite: ngamsDbSnapShotTest (#11/30) - 97.1 s - SUCCESS

Running Test Suite: ngamsDiscardCmdTest (#12/30) - 76.3 s - SUCCESS

Running Test Suite: ngamsEmailNotificationTest (#13/30) - 1.2 s - SUCCESS

Running Test Suite: ngamsExitCmdTest (#14/30) - 11.5 s - SUCCESS

Running Test Suite: ngamsHelpCmdTest (#15/30) - 11.4 s - SUCCESS

Running Test Suite: ngamsIdleSuspensionTest (#16/30) - 293.2s - SUCCESS

Running Test Suite: ngamsInitCmdTest (#17/30) - 15.3 s - SUCCESS

Running Test Suite: ngamsJanitorThreadTest (#18/30) - 1.2 s - SUCCESS

Running Test Suite: ngamsLabelCmdTest (#19/30) - 1.2 s - SUCCESS

Running Test Suite: ngamsOfflineCmdTest (#20/30) - 8.6 s - SUCCESS

Running Test Suite: ngamsOnlineCmdTest (#21/30) - 10.2 s - SUCCESS

Running Test Suite: ngamsPClientTest (#22/30) - 217.7s - SUCCESS

Running Test Suite: ngamsRegisterCmdTest (#23/30) - 9.3 s - SUCCESS

Running Test Suite: ngamsRemDiskCmdTest (#24/30) - 76.7 s - SUCCESS

Running Test Suite: ngamsRemFileCmdTest (#25/30) - 62.2 s - SUCCESS

Running Test Suite: ngamsRestoreCmdTest (#26/30) - 1.2 s - SUCCESS

Running Test Suite: ngamsRetrieveCmdTest (#27/30) - 269.6s - SUCCESS

Running Test Suite: ngamsServerTest (#28/30) - 1.2 s - SUCCESS

Running Test Suite: ngamsStatusCmdTest (#29/30) - 53.9 s - SUCCESS

Running Test Suite: ngamsSubscriptionTest (#30/30) - 1.2 s - SUCCESS

Executed 30 Test Suites in 2541.162 seconds

If tests fail, the detailed information about the cause of the problem will be contained in the report

Tagging the Release

In order to tag an NGAS release, the tool “ngasFreezeRelease” is used. It should be invoked as follows:

$ngasmgr> python /opsw/packages/ngasInstall/src/ngasFreezeRelease.py

It will prompt for a release tag for the pertinent modules. This could be e.g. “NGAS-V3_1_1Beta1”; the latest release tag will be displayed and can be chosen if desirable. If the same tag is chosen, the CVS module in question will be retagged.

Creating and Publishing the Distribution Tarball

After having tagged all CVS modules constituting an NGAS distribution, the release tarball should be created. This is done by means of the tool “ngasCreateDist”. This should be invoked as follows:

$ngasmgr> python /opsw/packages/ngasInstall/src/ngasCreateDist.py

It will prompt the user for some information about the release (name etc.).

The resulting release tarball will appear as, e.g.:

/opsw/packages/ngasInstall/tar-balls/V3_1_1-2007-07-12T08_42_49-Linux_SL43.tar.gz

This should then be published or stored in the appropriate location.

JKN: until now for convenience, I published it under my private ESO WEB area:

http://www.eso.org/~jknudstr/NGAS/RELEASES/V3_1_1/

The contents under http://www.eso.org/~jknudstr/NGAS/RELEASES/, was synchronised to SOS’ more official repository for the NGAS distributions:

http://aclxint.hq.eso.org/addons/ngas/ngas_software/CURRENT/ [1]

From this site, SOS was installing NGAS on the operational nodes. At LS and PL, NGAS was installed from a CDROM (the NGAS installation tool supports both).

Just for completenes, the crontab to mirror the NGAS release area onto the SOS NGAS SW area, was as follows:

30 * * * * rm -f /home/jknudstr/public_html/NGAS/RELEASES/*~; \

/usr/local/bin/rsync -ave ssh –delete \

/home/jknudstr/public_html/NGAS/RELEASES/ \

ngas@aclxint:/home/ngas/ngas_software/ > /dev/null

Installing an NGAS Node

In order to install an NGAS Node, the tool “ngasInstallSys” is used; the latest/appropriate version is available within the NGAS distribution as a standalone tool:

http://www.eso.org/~jknudstr/NGAS/RELEASES/V3_1_1/ngasInstallSys

It can be executed without the availability of further NGAS SW. It only needs Python to execute.

In order to facilitate the installation procedure, pre-defined installation commands are provided within the release, e.g.:

http://www.eso.org/~jknudstr/NGAS/RELEASES/V3_1_1/INSTALL-CMD-LS-AHU

As can be seen from the name, imaginatively, this command is used to install the Archive Handling Unit (AHU) at La Silla.

The content of such a command is e.g. (from the example above)[2]:

###############################################################################

# Installation UNIX shell script to install an NGAS system of type:

#   LS, AHU

###############################################################################

###############################################################################

# Base URL from where to obtain the distribution.

###############################################################################

#SRC_URL=http://aclxint.hq.eso.org/addons/ngas/ngas_software/CURRENT/

SRC_URL=file://$PWD/

###############################################################################

# Various parameters to update for the specific context.

###############################################################################

INST_EMAIL=ngast@eso.org

WATCHDOG_EMAIL=ngast@eso.org

NGAMS_CFG_FILE_SRC=/opsw/packages/ngasCfg/cfg/NgamsCfg_LS_AHU.xml

NGAMS_CFG_FILE=/opsw/packages/cfg/ngamsServer.conf

NGAMS_DB_CFG_ID=FILE-BASED-CFG

PAR_ZOPE_URLS=http://$HOSTNAME:8080/NGAS/OLASLS

#ALIASES_FILE=/opsw/packages/ngasSys/common/LS_etc_aliases

NGAS_UTILS_RESOURCE_FILE=/opsw/packages/ngasSys/common/ngas_DHA_LS

###############################################################################

…

As can be seen from the “SRC_URL” above, the LS and PL systems are installed from a CDROM. I.e., the data for the distribution is picked up locally. The other URL is the one used at HQ to access the distribution files.

The other parameters are as follows:

·         INST_EMAIL: Email recipient that will receive the final report indiacting the status of the installation procedure and all actions executed.

·         WATCHDOG_EMAIL: Email recipient that will receive notification when the NGAS Server Watchdog restarts the NGAS Server.

·         NGAMS_CFG_FILE_SRC: Source configuration file to be used. Must be specified and must at least contain the parameters for connecting to the RDBMS.

·         NGAMS_CFG_FILE: The configuration file used as input parameter to the “ngamsServer” application; this is a link pointing to the configuration file above.

·         NGAMS_DB_CFG_ID: Indicates if the configuration information is stored in the DB or in a local file. In case of former, this should be the ID of the DB configuration. If only a local configuration file is used, this should be “FILE-BASED-CFG”.

·         PAR_ZOPE_URLS: Indicates the parameters for the NGAS Zope Watchdog to monitor.

·         NGAS_UTILS_RESOURCE_FILE: Resource file used by the NGAS Utilities.

Usually it should be possible to use one of the predefined installation commands. If necessary, the best suitable one can be taken and updated to adapt the system for specific purposes.

As example of an installation report is shown below:

Subject: NGAS INSTALLATION REPORT

From: ngasmgr@eso.org

Date: Tue, 26 Jun 2007 12:45:44 +0200 (MEST)

To: undisclosed-recipients:;

================================================================================

NGAS INSTALLATION UTILITY

--------------------------------------------------------------------------------

Date: 2007-06-26T10:43:11 UTC

User: ngasmgr

Host: ngasdev3

Status: OK

================================================================================

INFO> Executing shell command: ps -efww | grep ngamsServer | grep /opsw/util/python/bin/python | grep -v root

INFO> Running installation tool with command line options: --installation-root=/opsw/packages/, --src-url=http://www.eso.org/~jknudstr/NGAS/RELEASES/V3_1_1/, --post-installation-pi=Install_Post_Inst, --notif-email=jknudstr@eso.org, --Install_Configure_NGAS=NGAMS_CFG_FILE_SRC=/opsw/packages/ngasCfg/cfg/NgamsCfg.Example.xml,NGAMS_CFG_FILE=/opsw/packages/cfg/ngamsServer.conf,NOTIF_EMAIL=jknudstr@eso.org, --Install_Configure_Users, --Install_Configure_Zope=PAR_ZOPE_URLS=http://ngasdev3.hq.eso.org:8080/NGAS/ESOECF, --Install_Configure_aliases=/opsw/packages/ngasSys/common/DEV_etc_aliases, --Install_Configure_pydoc, --Install_cfitsio, --Install_chksum, --Install_frameIngest, --Install_ngams_bins, --Install_ngasPlugIns, --Install_Configure_ngasUtils=RESOURCE_FILE=/opsw/packages/ngasSys/common/ngas_DEV

INFO> Installing NGAS Package ...

INFO> Selected NGAS Package Name: NGAS-CURRENT

INFO> Executing shell command: rm -rf /tmp/NGAS-CURRENT.tar*

INFO> Retrieving package from URL: http://www.eso.org/~jknudstr/NGAS/RELEASES/V3_1_1//NGAS-CURRENT.tar.gz into target file: /tmp/NGAS-CURRENT.tar.gz ...

INFO> Executing shell command: gunzip /tmp/NGAS-CURRENT.tar.gz

INFO> Entering path: /opsw/packages/

INFO> Executing shell command: rm -rf /opsw/packages//ngasInstall/plug-ins/*.py

INFO> Executing shell command: tar xvf /tmp/NGAS-CURRENT.tar

INFO> Entering path: /opsw/packages/ngams

INFO> Executing shell command: ./bootstrap

INFO> Executing shell command: ./configure --prefix=/opsw/packages/

INFO> Entering path: ngamsCClient

INFO> Executing shell command: make clean all install

INFO> Installed NGAS Package

INFO> Entering runInstallPlugIns() ...

INFO> Executing Installation Plug-In: Install_Configure_Users

INFO> Executed Installation Plug-In: Install_Configure_Users

INFO> Not executing Installation Plug-In Install_Python

INFO> Executing Installation Plug-In: Install_Configure_aliases

INFO> Executed Installation Plug-In: Install_Configure_aliases

INFO> Executing Installation Plug-In: Install_Configure_ngasUtils

INFO> Executed Installation Plug-In: Install_Configure_ngasUtils

INFO> Executing Installation Plug-In: Install_Configure_pydoc

INFO> Executed Installation Plug-In: Install_Configure_pydoc

INFO> Executing Installation Plug-In: Install_frameIngest

INFO> Executed Installation Plug-In: Install_frameIngest

INFO> Executing Installation Plug-In: Install_ngams_bins

INFO> Executed Installation Plug-In: Install_ngams_bins

INFO> Not executing Installation Plug-In Install_SysFiles

INFO> Not executing Installation Plug-In Install_Sybase

INFO> Executing Installation Plug-In: Install_chksum

INFO> Executed Installation Plug-In: Install_chksum

INFO> Executing Installation Plug-In: Install_cfitsio

INFO> Executed Installation Plug-In: Install_cfitsio

INFO> Executing Installation Plug-In: Install_ngasPlugIns

INFO> Executed Installation Plug-In: Install_ngasPlugIns

INFO> Executing Installation Plug-In: Install_Configure_NGAS

INFO> Executed Installation Plug-In: Install_Configure_NGAS

INFO> Not executing Installation Plug-In Install_Configure_fstab

INFO> Executing Installation Plug-In: Install_Configure_Zope

INFO> Executed Installation Plug-In: Install_Configure_Zope

INFO> Not executing Installation Plug-In Install_Post_Inst

INFO> Not executing Installation Plug-In Install_Zope

INFO> Leaving runInstallPlugIns()

INFO> Executing Installation Plug-In: Install_Post_Inst

INFO> Executed Installation Plug-In: Install_Post_Inst

INFO> Executed system installation/configuration

INFO> Entering submitReport() ...

================================================================================

Verifying the Installation

After the node has been installed it can be verified manually in the following way:

Reboot the node or execute:

$ngasmgr> sudo /etc/init.d/ngamsServer start

Execute an NGAS STATUS Command on the server:

$ngasmgr> ngamsStatus

Status of request:

Host: ngasdev3

Port: 7777

Command: STATUS

Date: 2007-10-05T12:39:53.832

Error Code: 0

Host ID: ngasdev3

Message: Successfully handled command STATUS

Status: SUCCESS

State: ONLINE

Sub-State: IDLE

NG/AMS Version: v3.1.1/2007-07-12T10:30:00

Other commands could be executed for instance to archive data and retrieve files for the sake of making a rough verification of the node.

The installation can also be verified by executing the automatic NGAS tests; see above (“Preparing the Distribution”). However, in order to execute the tests, the operational DB servers must be commented out from the Sybase interfaces file. Also, the alias in “/etc/aliases” for the user “ngasmgr” must be unset. The test tool will inform about this if one or both preconditions are not fulfilled.

[1] If of interest, the contents of the NGAS distribution folder can be described.

[2] Only the part of interest for the person installing the system is shown. I.e., the parameters that can be adjusted to reflect various changes in the configuration.