External Web>PackagingScript (2022-06-16, DirkPetry)Raw edit |

ALMA QA2 Packaging Script

(DEPRECATED - use new page at https://confluence.alma.cl/display/MDR/QA+Packager )

ALMA QA2 Packaging Script
- Running the script

The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

from QA2_Packaging_module import *

Packaging of standard script-generator-assisted analysis products

For a standard script-generator-assisted analysis, you can continue to run the script in the following way:

QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py', style='cycle8-nopipe', mode='copy')

If you just want to test, set mode = 'fake'.

The parameters mean the following:

origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
packpath: the path to the destination folder (which should have the project code as name)
paths should not end in /
readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
PIscript: should be the path to the latest version of the scriptForPI.py (from the JAO CVS repository or a mirror)
style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
mode: The copying mode:
- mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
- mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
- mode = ’copy’ : The files are copied in the normal way.
- mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
- mode = ’manimgonly’ : only the products of manual imaging (recognized by the string "manual" in the file names) and the corresponding scripts will be packaged using "copy"; supplemental full pol. calibration will be picked up in the case of PL intensity calibration)
append (optional): The appending mode (see below).
- append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
- append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
- append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
style: alternative (and recommended) way to specify a number of option at once.
- style = "cycle0-nopipe": corresponds to noms=False
- style = "cycle1-nopipe": corresponds to noms=True
- style = "cycle2-nopipe": corresponds to noms=True
- style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
- style = "cycle3-nopipe": same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
- style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
- style = "cycle4-nopipe": so far (v1.49) same as cycle3-nopipe
- style = "cycle4-pipe": so far (v1.49) same as cycle3-pipe1
- style = "cycle5-nopipe", "cycle6-nopipe": same as cycle4-nopipe
- style = "cycle5-pipe", "cycle6-pipe", "cycle7-pipe", "cycle8-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
- default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle7-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')

Packaging including Pipeline products

If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.

Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI, the packager enables you to do that as well.

Both use cases are described below:

Case 1a: packaging for external manual imaging

An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='<hard, copy, fake>',  # default is 'copy'
                                      PIscript="<path to scriptForPI.py>", # needs to be included in the package
                                      cycle=<num>) # where num is 0,1,2,3,4,5,6,7 and decides on the packaging style (pipeline version used in that cycle)

    Example:
       QA_Pipeline_Stager('2019.100456.S-2020-08-08T123456', '2019.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=7)

Case 1b: incremental packaging of only the manual imaging products

An SB was processed by the Cycle 7 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the manual imaging products into the archive.

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='man-img-only',  
                                      cycle=<num>) # where num is 7

    Example:
       QA_Pipeline_Stager('2019.100456.S-2020-08-08T123456', '2019.100456.S', mode='man-img-only', cycle=7)

NOTE: If you only have a "working" and no "products" directory, create a link to the "working" directory under the name "products" on the same level as the "working" directory, i.e. run

    cd <top dir of pipeline results>/S*OUS*/G*OUS*/M*OUS*/
    ln -sf working products
    cd ../../../../..

before you run the stager.

Case 2a: packaging after external manual imaging including calibration and imaging products

A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the whole data is to be packaged for delivery.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='<hard,copy,fake,ticket>',
                               style='cycle7-pipe', # this style needs to the chosen in this case
                               PIscript="./scriptForPI.py")  # overwrites the original scriptForPI.py if needed

   Example:
       QA_Packager(origpath='./2019.100456.S-analysis',
                               packpath='./2019.100456.S',
                               readme='myREADME.txt',
                               mode='hard',
                               style='cycle7-pipe',
                               PIscript="./scriptForPI.py")

NOTE: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE

See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.

Case 2b: packaging after external imaging including only manual imaging products

A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).

However, the calibration products have already been ingested in the archive.

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the manual imaging results are to be packaged for delivery.

NOTE: this also covers the case of full polarisation calibration after PL intensity calibration. The supplemental scriptForPolCalibration and the full pol. diagnostic plots will be picked up.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='man-img-only',
                               style='cycleX-pipe', # a -pipe style needs to the chosen in this case
                               PIscript="./scriptForPI.py")  # overwrites the original scriptForPI.py if needed

   Example:
       QA_Packager(origpath='./2018.100456.S-analysis',
                               packpath='./2018.100456.S',
                               readme='myREADME.txt',
                               mode='man-img-only',
                               style='cycle8-pipe',
                               PIscript="./scriptForPI.py")

NOTE again: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE

Packaging of TP data

Since version 1.43 of the packager, handling of Total Power data is supported.

The typical call to package manually calibrated TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-nopipe', mode='hard')

The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py . It will then override other modes and switch to TP mode. The typical output tree is shown here.

Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them.

The typical call to stage pipeline-calibrated TP data looks like this:

QA_Pipeline_Stager('2013.1.00911.S_2015_08_07T16_10_05.732', '2013.1.00911.S', mode='copy', PIscript="./scriptForPI.py")

The typical call to package pipeline-calibrated TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-pipe1', mode='hard', PIscript="./scriptForPI.py")

Note: the latter expects fits images in the "calibrated" directory.

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Usage:

     tar xvzf QA2_Packaging_module.unit-test.tgz
     cd QA2_Packaging_module.unit-test
     cp ../QA2_Packaging_module.py .
     start casapy and run
         execfile('QA2_Packaging_module.unit-test.py')