Difference: PackagingScript (1 vs. 39)

Revision 382022-02-10 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 18 to 18
  For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
Changed:
<
<
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle7-nopipe', mode='copy')
>
>
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle8-nopipe', mode='copy')
  If you just want to test, set mode = 'fake'.
Line: 35 to 35
 
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
Changed:
<
<
    • mode = ’manimgonly’ : only the products of manual imaging (recognized by the string "manual" in the file names) and the corresponding scripts will be packaged using "copy")
>
>
    • mode = ’manimgonly’ : only the products of manual imaging (recognized by the string "manual" in the file names) and the corresponding scripts will be packaged using "copy"; supplemental full pol. calibration will be picked up in the case of PL intensity calibration)
 
  • append (optional): The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
Line: 50 to 50
 
    • style = "cycle4-nopipe": so far (v1.49) same as cycle3-nopipe
    • style = "cycle4-pipe": so far (v1.49) same as cycle3-pipe1
    • style = "cycle5-nopipe", "cycle6-nopipe": same as cycle4-nopipe
Changed:
<
<
    • style = "cycle5-pipe", "cycle6-pipe", "cycle7-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
>
>
    • style = "cycle5-pipe", "cycle6-pipe", "cycle7-pipe", "cycle8-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Line: 135 to 135
  PIscript="./scriptForPI.py")
Changed:
<
<
NOTE: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE
>
>
NOTE: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE
 

See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.

Line: 154 to 154
  The result is QA2_PASS and now the manual imaging results are to be packaged for delivery.
Added:
>
>
NOTE: this also covers the case of full polarisation calibration after PL intensity calibration. The supplemental scriptForPolCalibration and the full pol. diagnostic plots will be picked up.
 Then run the packager as follows:
       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='man-img-only',
Changed:
<
<
style='cycle6-pipe', # this style needs to the chosen in this case
>
>
style='cycleX-pipe', # a -pipe style needs to the chosen in this case
  PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed

Example:

Line: 168 to 170
  packpath='./2018.100456.S', readme='myREADME.txt', mode='man-img-only',
Changed:
<
<
style='cycle6-pipe',
>
>
style='cycle8-pipe',
  PIscript="./scriptForPI.py")
Changed:
<
<
NOTE again: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE
>
>
NOTE again: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE
 

Packaging of TP data

Line: 218 to 220
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script.
Changed:
<
<
-- Dirk Petry - 2018-10-10
>
>
-- Dirk Petry - 2022-02-10
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 372019-10-02 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 18 to 18
  For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
Changed:
<
<
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle6-nopipe', mode='copy')
>
>
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle7-nopipe', mode='copy')
  If you just want to test, set mode = 'fake'.
Line: 50 to 50
 
    • style = "cycle4-nopipe": so far (v1.49) same as cycle3-nopipe
    • style = "cycle4-pipe": so far (v1.49) same as cycle3-pipe1
    • style = "cycle5-nopipe", "cycle6-nopipe": same as cycle4-nopipe
Changed:
<
<
    • style = "cycle5-pipe", "cycle6-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
>
>
    • style = "cycle5-pipe", "cycle6-pipe", "cycle7-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle6-nopipe', PIscript='/science/qa2/scriptForPI.py')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle7-nopipe', PIscript='/science/qa2/scriptForPI.py')
 

Packaging including Pipeline products

Line: 78 to 78
  staged_root='', mode='<hard, copy, fake>', # default is 'copy' PIscript="", # needs to be included in the package
Changed:
<
<
cycle=) # where num is 0,1,2,3,4,5,6 and decides on the packaging style (pipeline version used in that cycle)
>
>
cycle=) # where num is 0,1,2,3,4,5,6,7 and decides on the packaging style (pipeline version used in that cycle)
  Example:
Changed:
<
<
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3)
>
>
QA_Pipeline_Stager('2019.100456.S-2020-08-08T123456', '2019.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=7)
 

Case 1b: incremental packaging of only the manual imaging products

Changed:
<
<
An SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the manual imaging products into the archive.
>
>
An SB was processed by the Cycle 7 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the manual imaging products into the archive.
  Then run the packager as follows:
      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='man-img-only',  
Changed:
<
<
cycle=) # where num is 6
>
>
cycle=) # where num is 7
  Example:
Changed:
<
<
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='man-img-only', cycle=6)
>
>
QA_Pipeline_Stager('2019.100456.S-2020-08-08T123456', '2019.100456.S', mode='man-img-only', cycle=7)
 

NOTE: If you only have a "working" and no "products" directory, create a link to the "working" directory under the name "products" on the same level as the "working" directory, i.e. run

Line: 123 to 123
  packpath='', readme='', mode='<hard,copy,fake,ticket>',
Changed:
<
<
style='cycle6-pipe', # this style needs to the chosen in this case
>
>
style='cycle7-pipe', # this style needs to the chosen in this case
  PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed

Example:

Changed:
<
<
QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S',
>
>
QA_Packager(origpath='./2019.100456.S-analysis', packpath='./2019.100456.S',
  readme='myREADME.txt', mode='hard',
Changed:
<
<
style='cycle6-pipe',
>
>
style='cycle7-pipe',
  PIscript="./scriptForPI.py")

Revision 362019-01-10 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 98 to 98
  QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='man-img-only', cycle=6)
Added:
>
>
NOTE: If you only have a "working" and no "products" directory, create a link to the "working" directory under the name "products" on the same level as the "working" directory, i.e. run
    cd <top dir of pipeline results>/S*OUS*/G*OUS*/M*OUS*/
    ln -sf working products
    cd ../../../../..
before you run the stager.
 

Case 2a: packaging after external manual imaging including calibration and imaging products

A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).

Revision 352018-10-10 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 84 to 84
  QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3)
Changed:
<
<

Case 1b: incremental packaging of only the science imaging products

An SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the science imaging products into the archive.
>
>

Case 1b: incremental packaging of only the manual imaging products

An SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the manual imaging products into the archive.
  Then run the packager as follows:
      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
Changed:
<
<
mode='sci-img-only',
>
>
mode='man-img-only',
  cycle=) # where num is 6

Example:

Changed:
<
<
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='sci-img-only', cycle=6)
>
>
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='man-img-only', cycle=6)
 

Case 2a: packaging after external manual imaging including calibration and imaging products

Line: 132 to 132
  See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
Changed:
<
<

Case 2b: packaging after external imaging including only science imaging products

>
>

Case 2b: packaging after external imaging including only manual imaging products

  A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).
Line: 144 to 144
  The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.
Changed:
<
<
The result is QA2_PASS and now the science imaging results are to be packaged for delivery.
>
>
The result is QA2_PASS and now the manual imaging results are to be packaged for delivery.
  Then run the packager as follows:
       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
Changed:
<
<
mode='sci-img-only',
>
>
mode='man-img-only',
  style='cycle6-pipe', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Line: 159 to 159
  QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S', readme='myREADME.txt',
Changed:
<
<
mode='sci-img-only',
>
>
mode='man-img-only',
  style='cycle6-pipe', PIscript="./scriptForPI.py")
Line: 210 to 210
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script.
Changed:
<
<
-- Dirk Petry - 2018-10-09
>
>
-- Dirk Petry - 2018-10-10
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 342018-10-09 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 6 to 6
  The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.
Deleted:
<
<
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database

A pdf document with a more detailed description of the packaging script: documentation.pdf

As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.

As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline.

As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1".

As of 25 Aug 2016, the packager also accepts a tgz extension in addition to the tar.gz extension.

 

Running the script

Line: 29 to 18
  For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
Changed:
<
<
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle2-nopipe', mode='fake')
>
>
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle6-nopipe', mode='copy')
 
Changed:
<
<
This will just test the packaging (mode = 'fake').
>
>
If you just want to test, set mode = 'fake'.
  The parameters mean the following:
Line: 39 to 28
 
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
Changed:
<
<
  • PIscript added Nov 2013
>
>
  • PIscript: should be the path to the latest version of the scriptForPI.py (from the JAO CVS repository or a mirror)
 
  • style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
Added:
>
>
    • mode = ’manimgonly’ : only the products of manual imaging (recognized by the string "manual" in the file names) and the corresponding scripts will be packaged using "copy")
 
  • append (optional): The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
Line: 59 to 49
 
    • style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
    • style = "cycle4-nopipe": so far (v1.49) same as cycle3-nopipe
    • style = "cycle4-pipe": so far (v1.49) same as cycle3-pipe1
Added:
>
>
    • style = "cycle5-nopipe", "cycle6-nopipe": same as cycle4-nopipe
    • style = "cycle5-pipe", "cycle6-pipe": taking into account naming changes (manifest, ppr) from the CASA 5.1.1 pipeline onwards
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle3-nopipe', PIscript='/science/qa2/scriptForPI.py')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle6-nopipe', PIscript='/science/qa2/scriptForPI.py')
 

Packaging including Pipeline products

Line: 77 to 69
  Both use cases are described below:
Changed:
<
<

Case 1:

>
>

Case 1a: packaging for external manual imaging

 An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

Line: 86 to 78
  staged_root='', mode='<hard, copy, fake>', # default is 'copy' PIscript="", # needs to be included in the package
Changed:
<
<
cycle=) # where num is 0,1,2,3,4 and decides on the packaging style (pipeline version used in that cycle)
>
>
cycle=) # where num is 0,1,2,3,4,5,6 and decides on the packaging style (pipeline version used in that cycle)
  Example: QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3)
Changed:
<
<

Case 2:

A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).
>
>

Case 1b: incremental packaging of only the science imaging products

An SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the science imaging products into the archive.

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='sci-img-only',  
                                      cycle=<num>) # where num is 6 

    Example:
       QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='sci-img-only', cycle=6)

Case 2a: packaging after external manual imaging including calibration and imaging products

A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).
  The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.
Line: 109 to 115
  packpath='', readme='', mode='<hard,copy,fake,ticket>',
Changed:
<
<
gzip_caltables=True, style='cycle4-pipe', # this style needs to the chosen in this case
>
>
style='cycle6-pipe', # this style needs to the chosen in this case
  PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed

Example:

Changed:
<
<
QA_Packager(origpath='./2016.100456.S-analysis', packpath='./2016.100456.S',
>
>
QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S',
  readme='myREADME.txt', mode='hard',
Changed:
<
<
gzip_caltables=True, style='cycle4-pipe',
>
>
style='cycle6-pipe',
  PIscript="./scriptForPI.py")
Line: 128 to 132
  See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
Added:
>
>

Case 2b: packaging after external imaging including only science imaging products

A package as created in Case 1a above has arrived at the ARC (or a JAO analyst).

However, the calibration products have already been ingested in the archive.

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the science imaging results are to be packaged for delivery.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='sci-img-only',
                               style='cycle6-pipe', # this style needs to the chosen in this case
                               PIscript="./scriptForPI.py")  # overwrites the original scriptForPI.py if needed

   Example:
       QA_Packager(origpath='./2018.100456.S-analysis',
                               packpath='./2018.100456.S',
                               readme='myREADME.txt',
                               mode='sci-img-only',
                               style='cycle6-pipe',
                               PIscript="./scriptForPI.py")

NOTE again: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE

 

Packaging of TP data

Since version 1.43 of the packager, handling of Total Power data is supported.

Line: 169 to 207
 

Acknowledgements

Changed:
<
<
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.
>
>
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script.
 
Changed:
<
<

-- DirkPetry - 25 Aug 2016
>
>
-- Dirk Petry - 2018-10-09
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 332016-09-20 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 85 to 85
  QA_Pipeline_Stager(pipeline_root='', staged_root='', mode='<hard, copy, fake>', # default is 'copy'
Changed:
<
<
PIscript="") # needs to be included in the package
>
>
PIscript="", # needs to be included in the package cycle=) # where num is 0,1,2,3,4 and decides on the packaging style (pipeline version used in that cycle)
  Example:
Changed:
<
<
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py")
>
>
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3)
 

Case 2:

Line: 109 to 110
  readme='', mode='<hard,copy,fake,ticket>', gzip_caltables=True,
Changed:
<
<
style='cycle3-pipe1', # this style needs to the chosen in this case
>
>
style='cycle4-pipe', # this style needs to the chosen in this case
  PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed

Example:

Changed:
<
<
QA_Packager(origpath='./2013.100456.S-analysis', packpath='./2013.100456.S',
>
>
QA_Packager(origpath='./2016.100456.S-analysis', packpath='./2016.100456.S',
  readme='myREADME.txt', mode='hard', gzip_caltables=True,
Changed:
<
<
style='cycle3-pipe1',
>
>
style='cycle4-pipe',
  PIscript="./scriptForPI.py")

Revision 322016-08-25 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 16 to 16
  As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1".
Added:
>
>
As of 25 Aug 2016, the packager also accepts a tgz extension in addition to the tar.gz extension.

 

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

Line: 55 to 57
 
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
    • style = "cycle3-nopipe": same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
    • style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
Added:
>
>
    • style = "cycle4-nopipe": so far (v1.49) same as cycle3-nopipe
    • style = "cycle4-pipe": so far (v1.49) same as cycle3-pipe1
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='/science/qa2/scriptForPI.py')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle3-nopipe', PIscript='/science/qa2/scriptForPI.py')
 

Packaging including Pipeline products

Line: 84 to 88
  PIscript="") # needs to be included in the package

Example:

Changed:
<
<
QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")
>
>
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py")
 

Case 2:

Line: 105 to 109
  readme='', mode='<hard,copy,fake,ticket>', gzip_caltables=True,
Changed:
<
<
style='cycle2-pipe1', # this style needs to the chosen in this case
>
>
style='cycle3-pipe1', # this style needs to the chosen in this case
  PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed

Example:

Line: 114 to 118
  readme='myREADME.txt', mode='hard', gzip_caltables=True,
Changed:
<
<
style='cycle2-pipe1',
>
>
style='cycle3-pipe1',
  PIscript="./scriptForPI.py")
Line: 167 to 171
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.
Changed:
<
<

-- DirkPetry - 8 Dec 2015
>
>

-- DirkPetry - 25 Aug 2016
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 312016-02-17 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 118 to 118
  PIscript="./scriptForPI.py")
Added:
>
>
NOTE: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE
 See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.

Packaging of TP data

Revision 302015-12-08 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 14 to 14
  As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline.
Added:
>
>
As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1".
 

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

Line: 51 to 53
 
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
Changed:
<
<
    • style = "cycle3-nopipe": presently the same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
    • style = "cycle3-pipe1": presently the same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py
>
>
    • style = "cycle3-nopipe": same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
    • style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Line: 162 to 164
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.
Changed:
<
<

-- DirkPetry - 1 Nov 2015
>
>

-- DirkPetry - 8 Dec 2015
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 292015-11-02 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 12 to 12
  As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.
Added:
>
>
As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline.
 

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

Line: 49 to 51
 
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
Changed:
<
<
    • style = "cycle3-nopipe": presently the same as cycle2-nopipe
    • style = "cycle3-pipe1": presently the same as cycle2-pipe1
>
>
    • style = "cycle3-nopipe": presently the same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
    • style = "cycle3-pipe1": presently the same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Line: 160 to 162
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.
Changed:
<
<

-- DirkPetry - 30 Oct 2015
>
>

-- DirkPetry - 1 Nov 2015
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 282015-10-30 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 49 to 49
 
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
Added:
>
>
    • style = "cycle3-nopipe": presently the same as cycle2-nopipe
    • style = "cycle3-pipe1": presently the same as cycle2-pipe1
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Line: 118 to 120
  Since version 1.43 of the packager, handling of Total Power data is supported.
Changed:
<
<
The typical call to package TP data looks like this:
>
>
The typical call to package manually calibrated TP data looks like this:
 
Changed:
<
<
QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2012.1.00501.S', style='cycle2-nopipe', mode='hard')
>
>
QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-nopipe', mode='hard')
 

The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py .

Line: 128 to 130
  Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them.
Added:
>
>
The typical call to stage pipeline-calibrated TP data looks like this:
QA_Pipeline_Stager('2013.1.00911.S_2015_08_07T16_10_05.732', '2013.1.00911.S', mode='copy', PIscript="./scriptForPI.py")

The typical call to package pipeline-calibrated TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-pipe1', mode='hard', PIscript="./scriptForPI.py")
Note: the latter expects fits images in the "calibrated" directory.
 

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Line: 146 to 160
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.
Changed:
<
<

-- DirkPetry - 20 Aug 2015
>
>

-- DirkPetry - 30 Oct 2015
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 272015-08-20 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

ALMA QA2 Packaging Script

Line: 114 to 114
  See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
Added:
>
>

Packaging of TP data

Since version 1.43 of the packager, handling of Total Power data is supported.

The typical call to package TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2012.1.00501.S', style='cycle2-nopipe', mode='hard')

The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py . It will then override other modes and switch to TP mode. The typical output tree is shown here.

Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them.

 

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Line: 130 to 144
 

Acknowledgements

Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script.
Changed:
<
<
Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
>
>
Thanks to Ivan Marti-Vidal (Onsala) for continued support.
 
Changed:
<
<

-- DirkPetry - 9 Jan 2015
>
>

-- DirkPetry - 20 Aug 2015
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 262015-01-09 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Changed:
<
<

Early Science Packaging Script

>
>

ALMA QA2 Packaging Script

  The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.
Line: 65 to 67
  Both use cases are described below:
Changed:
<
<
Case 1:
>
>

Case 1:

 An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

Line: 79 to 81
  QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")
Changed:
<
<
Case 2:
>
>

Case 2:

 A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

Line: 126 to 128
 
Changed:
<
<
Acknowledgements
>
>

Acknowledgements

  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
Changed:
<
<

-- DirkPetry - 9 Sept 2014
>
>

-- DirkPetry - 9 Jan 2015
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 252014-10-22 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 110 to 110
  PIscript="./scriptForPI.py")
Added:
>
>
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
 

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Revision 242014-09-12 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 130 to 130
 
-- DirkPetry - 9 Sept 2014
Changed:
<
<
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1409748559" name="documentation.pdf" path="documentation.pdf" size="71961" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp4306" user="DirkPetry" version="4"
>
>
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="September 2014 version of the packager documentation" date="1410523633" name="documentation.pdf" path="packaging_documentation.pdf" size="74934" stream="packaging_documentation.pdf" tmpFilename="/usr/tmp/CGItemp15204" user="DirkPetry" version="5"

Revision 232014-09-09 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Changed:
<
<
The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py
>
>
The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.
  The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database
Line: 128 to 128
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
Changed:
<
<

-- DirkPetry - 2 Sept 2014
>
>

-- DirkPetry - 9 Sept 2014
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1409748559" name="documentation.pdf" path="documentation.pdf" size="71961" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp4306" user="DirkPetry" version="4"

Revision 222014-09-03 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 130 to 130
 
-- DirkPetry - 2 Sept 2014
Changed:
<
<
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1409649567" name="documentation.pdf" path="documentation.pdf" size="71833" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp13704" user="DirkPetry" version="3"
>
>
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1409748559" name="documentation.pdf" path="documentation.pdf" size="71961" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp4306" user="DirkPetry" version="4"

Revision 212014-09-02 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 38 to 38
 
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
Deleted:
<
<
In order to perform the final packaging, run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
 
  • append (optional): The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
Changed:
<
<
* style: alternative way to specify a number of option at once.
>
>
  • style: alternative (and recommended) way to specify a number of option at once.
 
    • style = "cycle0-nopipe": corresponds to noms=False
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
Line: 57 to 50
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Added:
>
>
In order to perform the final packaging, run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')

 

Packaging including Pipeline products

If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.

Line: 129 to 128
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
Changed:
<
<

-- DirkPetry - 22 Aug 2014
>
>

-- DirkPetry - 2 Sept 2014
 
Changed:
<
<
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1408722334" name="documentation.pdf" path="documentation.pdf" size="71824" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp15563" user="DirkPetry" version="2"
>
>
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1409649567" name="documentation.pdf" path="documentation.pdf" size="71833" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp13704" user="DirkPetry" version="3"

Revision 202014-08-22 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 131 to 131
 
-- DirkPetry - 22 Aug 2014
Changed:
<
<
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"
>
>
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="August 2014 version of the packager documentation" date="1408722334" name="documentation.pdf" path="documentation.pdf" size="71824" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp15563" user="DirkPetry" version="2"

Revision 192014-08-22 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 21 to 21
  For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',PIscript='/science/qa2/scriptForPI.py', noms=True, append='',mode='fake')
>
>
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='/science/qa2/scriptForPI.py', style='cycle2-nopipe', mode='fake')
  This will just test the packaging (mode = 'fake').
Line: 32 to 32
 
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • PIscript added Nov 2013
Changed:
<
<
  • noms: do not include the MSs
>
>
  • style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
 
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
Line: 42 to 42
 In order to perform the final packaging, run the script with

Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S', append='',mode='hard', noms=True, PIscript='/science/qa2/scriptForPI.py')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='/science/qa2/scriptForPI.py')
 
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
Changed:
<
<
  • append: The appending mode (see below).
>
>
  • append (optional): The appending mode (see below).
 
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed). * style: alternative way to specify a number of option at once.
    • style = "cycle0-nopipe": corresponds to noms=False
    • style = "cycle1-nopipe": corresponds to noms=True
Added:
>
>
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Line: 96 to 97
  packpath='', readme='', mode='<hard,copy,fake,ticket>',
Deleted:
<
<
append='',
  gzip_caltables=True, style='cycle2-pipe1', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Line: 106 to 106
  packpath='./2013.100456.S', readme='myREADME.txt', mode='hard',
Deleted:
<
<
append='',
  gzip_caltables=True, style='cycle2-pipe1', PIscript="./scriptForPI.py")
Line: 130 to 129
  Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
Changed:
<
<

-- DirkPetry - 14 Aug 2014
>
>

-- DirkPetry - 22 Aug 2014
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"

Revision 182014-08-20 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 21 to 21
  For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='/science/qa2/scriptForPI.py', noms=True, append='',mode='fake')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',PIscript='/science/qa2/scriptForPI.py', noms=True, append='',mode='fake')
  This will just test the packaging (mode = 'fake').
Line: 42 to 42
 In order to perform the final packaging, run the script with

Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',
 append='',mode='hard', noms=True, PIscript='/science/qa2/scriptForPI.py')
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.

Revision 172014-08-14 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Changed:
<
<

Early Science Packaging Script

>
>

Early Science Packaging Script

  The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py
Line: 8 to 8
  A pdf document with a more detailed description of the packaging script: documentation.pdf
Changed:
<
<

Running the script

>
>
As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.

Running the script

  The script should be run from within casa. It can be run from any directory. The import is done with

from QA2_Packaging_module import *
Changed:
<
<
Then run the script with
>
>

Packaging of standard script-generator-assisted analysis products

For a standard script-generator-assisted analysis, you can continue to run the script in the following way:

 
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='<yourjaomirror>/science/qa2/scriptForPI.py', noms=True, append='',mode='fake')
Added:
>
>
This will just test the packaging (mode = 'fake').

The parameters mean the following:

 
  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
Line: 29 to 38
 
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
Changed:
<
<
    • Run the script with

>
>
In order to perform the final packaging, run the script with

 QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', append='',mode='hard', noms=True, PIscript='/science/qa2/scriptForPI.py')
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
Line: 45 to 56
 
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Added:
>
>

Packaging including Pipeline products

If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.

Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI, the packager enables you to do that as well.

Both use cases are described below:

Case 1:
An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='<hard, copy, fake>',  # default is 'copy'
                                      PIscript="<path to scriptForPI.py>") # needs to be included in the package

    Example:
       QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")

Case 2:
A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the whole data is to be packaged for delivery.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='<hard,copy,fake,ticket>',
                               append='',
                               gzip_caltables=True,
                               style='cycle2-pipe1', # this style needs to the chosen in this case
                               PIscript="./scriptForPI.py")  # overwrites the original scriptForPI.py if needed

   Example:
       QA_Packager(origpath='./2013.100456.S-analysis',
                               packpath='./2013.100456.S',
                               readme='myREADME.txt',
                               mode='hard',
                               append='',
                               gzip_caltables=True,
                               style='cycle2-pipe1',
                               PIscript="./scriptForPI.py")

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Usage:

     tar xvzf QA2_Packaging_module.unit-test.tgz
     cd QA2_Packaging_module.unit-test
     cp ../QA2_Packaging_module.py .
     start casapy and run
         execfile('QA2_Packaging_module.unit-test.py')
 
Acknowledgements
Changed:
<
<
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.
>
>
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.
 
Changed:
<
<

-- DirkPetry - 20 Feb 2014
>
>

-- DirkPetry - 14 Aug 2014
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"

Revision 162014-04-10 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 44 to 44
 
    • style = "cycle1-nopipe": corresponds to noms=True
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
Deleted:
<
<

Examples

For example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run:

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='fake', noms=True)

This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files.

The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. The latter should be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file.

If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory:

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='ticket', noms=True)

Appending a member

To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='member',mode='fake')

NOTE: The following documentation was valid for cycle 0 style delivery and will shortly be updated to Cycle 1 style

Example data reduction directory structure

The packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following:

project123 - parent directory for the data reduction of this project
   SB1 - directory for SB number 1
      Xab1 - separate directories for each of the asdm, named after the last few characters for the asdm
      Xabc
      calibrated.ms - the combined  ms of this SB
      the imaging and flux calibration scripts
      fits files      
      optional qa2 notes for the PI
   SB2 - directory for SB number 2
      Xbb1
      Xbbc

For single asdms per SB, the structure would look like this:

project123 - parent directory for the data reduction of this project
   SB1 - directory for SB number 1
      Xab1 - a directory for the asdm, named after the last few characters for the asdm
         including also the imaging script and fits files etc      
   SB2 - directory for SB number 2
      Xbb1
 

or, alternatively:

project123 - parent directory for the data reduction of this project
   Xab1 - a directory for the asdm, named after the last few characters for the asdm
      including also the imaging script and fits files etc      
   Xbb1
 

The directories named Xab1 etc should contain uid___xxx.scriptForCalibration.py, uid___xxx.split, (and uid___xxx.split.cal), all calibration tables and the checklist. The only subdirectory in here should be qa2. Please do not create a separate 'Calibration' subdirectory in here. Note that there is no requirement for these directories to start with 'X'. You can call them 'Calibration_Xab1', if you prefer.

Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories.

For example, for project 229, which contains four SBs, the data reduction directory looks like this:
project229=
=|-- IRC10216_setup3_b6_run_x2</verbatim>
- first SB
| |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)
| |-- X8c - first execution of this SB
| | |-- checklist.txt - checklist (does not need to be prepended by asdm name)
| | |-- qa2 - qa2 directory
| | | |-- qa2_part1.png
| | | |-- textfile.txt
| | | |-- etc
| | |-- uid___A002_X3c9295_X8c.ms
| | |-- uid___A002_X3c9295_X8c.ms.antpos
| | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
| | |-- uid___A002_X3c9295_X8c.ms.split
| | |-- uid___A002_X3c9295_X8c.ms.split.cal
| | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass
| | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots
| | |-- etc
| |-- X528
| | |-- uid___A002_X3dc0e7_X528.ms
| | |-- etc
| |-- Combined - Optional directory to put the combined products in
| |-- calibrated.ms - combined ms. Will not be packaged!
| |-- calibrated_all_setup3_b6.clean.fits - a fits file
| |-- scriptForFluxCalibration.py
| |-- scriptForImaging.py
|-- IRC10216_setup4_b6_run_x2
| |-- X3c9295_X1e0
| | |-- uid___A002_X3c9295_X1e0.ms
| | |-- etc
</verbatim>

Notes

  • File names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed.
  • QA2 files will be found at the SB level, at the asdm level, and in the qa2 directory. They should match one of: *.jpg, *qa2.txt, *qa2-summ.txt, *.qa2.pdf, *textfile.txt', *qa2_part?.png
  • Checklists should match one of: *checklist, *checklist.txt, *Checklist, *Checklist.txt [script need updating]

Example output directory structure

After running
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='fake')
the following package was created:
|-- 2011.0.00229.S/
|   |-- sg_ouss_id/
|   |   |-- group_ouss_id/
|   |   |   |-- member_ouss_id/
|   |   |   |   |-- raw/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.split/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.split/
|   |   |   |   |-- qa/
|   |   |   |   |   |-- X528.checklist.txt
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part1.png
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__textfile.txt
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part1.png
|   |   |   |   |   |-- X8c.checklist.txt
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part3.png
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part2.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part2.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part3.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__textfile.txt
|   |   |   |   |-- script/
|   |   |   |   |   |-- scriptForImaging.py
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.scriptForCalibration.py
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
|   |   |   |   |   |-- scriptForCalibration.py
|   |   |   |   |   |-- scriptForAprioriCalibration.py
|   |   |   |   |   |-- scriptForFluxCalibration.py
|   |   |   |   |-- log/
|   |   |   |   |   |-- ipython.log
|   |   |   |   |   |-- casapy-2012-05-03T115433.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__casapy-20120502-102144.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__ipython.log
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__ipython-20120430-142604.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__ipython-20120502-102151.log
|   |   |   |   |-- product/
|   |   |   |   |   |-- calibrated_all_setup3_b6.clean.fits
|   |   |   |   |-- calibrated/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.split.cal/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.split.cal/
|   |   |   |   |-- calibration/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.calibration.plots/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.calibration/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.calibration/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.calibration.plots/

Then

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='member',mode='fake')
produces a second member:
|-- 2011.0.00229.S/
|   |-- sg_ouss_id/
|   |   |-- group_ouss_id/
|   |   |   |-- member_ouss_id/
|   |   |   |-- member_ouss_id2/

Tarring up the package

Once a new project is packaged and approved for delivery, the data have to be tarred up, be delivered to the PI and then sent to JAO for ingestion into the Archive (from where they get mirrored to the ARCs). This is the current procedure which might change as we go along.

In order to make data delivery to archival researchers in the future easy, please use tarsplit.py from cvs AIV/science/DSO to create the tarfiles. E.g.

tarsplit.py -n 2012-06-20 2011.01234.S

This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler.

README

Atacama Large Millimeter/submillimeter Array (ALMA)

#####

Cycle: 0 (Early science)
Project code:
SB name:
PI name:
Project title:
Configuration:
Proposed rms:
CASA version used for reduction: 3.4
Comments from Reducer:

#####

This file describes the content of the tar file you have received. The
full data structure is inserted below.

At this stage, we are releasing data after completion of one SB (excuted
multiple times if required), so you will find only one member_ouss_id
directory.  This directory contains this README file and the following
directories: raw, calibrated, calibration, script, qa2, log, product.

- 'raw' contains the apriori calibrated ms for each execution block,
after being split into the science spectral windows.  This calibration
includes: WVR, Tsys and antenna position corrections and apriori
flagging.
- 'calibrated' contains the fully calibrated ms for each execution block.
- 'calibration' contains the files needed for calibration starting from
the initial ms to the fully calibrated data.   Plots are included.
- 'script' contains the reduction scripts used to process the initial ms
to calibrated data, but also to obtain concatenated data (if more than
one execution) and imaging products.  There are usually several scripts
dealing with different parts of the processing.
- 'product' contains the fits files of the selected image products. 
These will not include all images of scientific value, but will indicate
the quality of the calibration and images.
- 'qa' contains the qa2 reports that show plots and text information
needed to assess the quality of the processing.  The resultant image
rms, compared with that proposed, is given.
- 'log' contains the casa log files.

#####
 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.

Revision 152014-02-20 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 28 to 28
 
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
Changed:
<
<
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
>
>
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
 
    • Run the script with

      QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
Line: 39 to 39
 
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
Added:
>
>
* style: alternative way to specify a number of option at once.
    • style = "cycle0-nopipe": corresponds to noms=False
    • style = "cycle1-nopipe": corresponds to noms=True
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead
 

Examples

Line: 238 to 242
 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.
Changed:
<
<

-- DirkPetry - 07 Feb 2014
>
>

-- DirkPetry - 20 Feb 2014
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"

Revision 142014-02-07 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 16 to 16
  Then run the script with
Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='./uid___A002_X71e4ae_X317.ms.scriptForCalibration.py', append='',mode='fake')
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='/science/qa2/scriptForPI.py', noms=True, append='',mode='fake')
 
  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • PIscript added Nov 2013
Added:
>
>
  • noms: do not include the MSs
 
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
Changed:
<
<
    • BEFORE PACKAGING WITH 'HARD' type:
    • execfile('/diska/work/software/jao-mirror/AIV/science/qa2/QA2_Packaging_module.py')

      Then run the script with

>
>

Run the script with

 QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
Changed:
<
<
append='',mode='hard')
>
>
append='',mode='hard', noms=True, PIscript='/science/qa2/scriptForPI.py')
 
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
  • append: The appending mode (see below).
Line: 46 to 45
 For example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run:
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
Changed:
<
<
packpath='./2011.0.00229.S',append='',mode='fake')
>
>
packpath='./2011.0.00229.S',append='',mode='fake', noms=True)
 

This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files.

Line: 56 to 55
 If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory:
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
Changed:
<
<
packpath='./2011.0.00229.S',append='',mode='ticket')
>
>
packpath='./2011.0.00229.S',append='',mode='ticket', noms=True)
 

Appending a member

Line: 65 to 64
 packpath='./2011.0.00229.S',append='member',mode='fake')
Added:
>
>

NOTE: The following documentation was valid for cycle 0 style delivery and will shortly be updated to Cycle 1 style

 

Example data reduction directory structure

The packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following:

Line: 236 to 238
 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.
Changed:
<
<

-- MartinZwaan - 20 May 2012
>
>

-- DirkPetry - 07 Feb 2014
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"

Revision 132013-11-14 - AnitaRichards

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 16 to 16
  Then run the script with
Changed:
<
<
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
>
>
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='./uid___A002_X71e4ae_X317.ms.scriptForCalibration.py',
 append='',mode='fake')

  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
Added:
>
>
  • PIscript added Nov 2013
 
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).

Revision 122013-11-08 - AnitaRichards

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 30 to 30
 
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
    • BEFORE PACKAGING WITH 'HARD' type:
Changed:
<
<
    • <font size="2"><span style="font-size: 10pt">execfile('/diska/work/software/jao-mirror/AIV/science/qa2/QA2_Packaging_module.py')</span></font>

      Then run the script with

>
>
    • execfile('/diska/work/software/jao-mirror/AIV/science/qa2/QA2_Packaging_module.py')

      Then run the script with

 QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', append='',mode='hard')
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).

Revision 112013-11-07 - AnitaRichards

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 25 to 25
 
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
Added:
>
>
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
 
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
Line: 49 to 50
  This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files.
Changed:
<
<
The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. These are the files that can be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file.
>
>
The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. The latter should be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file.
  If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory:

Revision 102013-06-05 - DirkPetry

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Changed:
<
<
The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py
>
>
The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py
  The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database

Revision 92012-12-04 - ElizabethHumphreys

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Changed:
<
<
The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py
>
>
The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py
 
Changed:
<
<
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database
>
>
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database
  A pdf document with a more detailed description of the packaging script: documentation.pdf
Line: 26 to 24
 
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • mode: The copying mode:
Changed:
<
<
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
>
>
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
    • BEFORE PACKAGING WITH 'HARD' type:
    • <font size="2"><span style="font-size: 10pt">execfile('/diska/work/software/jao-mirror/AIV/science/qa2/QA2_Packaging_module.py')</span></font>

      Then run the script with

      QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
      append='',mode='hard')
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
 
  • append: The appending mode (see below).
Changed:
<
<
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
>
>
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
 

Examples

Line: 54 to 57
 packpath='./2011.0.00229.S',append='',mode='ticket')
Deleted:
<
<
 

Appending a member

Changed:
<
<
To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':
>
>
To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':
 QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake')
Line: 103 to 104
  Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories.
Changed:
<
<
For example, for project 229, which contains four SBs, the data reduction directory looks like this:
=project229=
|-- IRC10216_setup3_b6_run_x2 - first SB
|   |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)
|   |-- X8c - first execution of this SB
|   |   |-- checklist.txt - checklist (does not need to be prepended by asdm name)
|   |   |-- qa2 - qa2 directory
|   |   |   |-- qa2_part1.png
|   |   |   |-- textfile.txt
|   |   |   |-- etc
|   |   |-- uid___A002_X3c9295_X8c.ms
|   |   |-- uid___A002_X3c9295_X8c.ms.antpos
|   |   |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
|   |   |-- uid___A002_X3c9295_X8c.ms.split
|   |   |-- uid___A002_X3c9295_X8c.ms.split.cal
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots
|   |   |-- etc
|   |-- X528
|   |   |-- uid___A002_X3dc0e7_X528.ms
|   |   |-- etc
|   |-- Combined - Optional directory to put the combined products in
|   |-- calibrated.ms - combined ms. Will not be packaged!
|   |-- calibrated_all_setup3_b6.clean.fits - a fits file
|   |-- scriptForFluxCalibration.py
|   |-- scriptForImaging.py
|-- IRC10216_setup4_b6_run_x2
|   |-- X3c9295_X1e0
|   |   |-- uid___A002_X3c9295_X1e0.ms
|   |   |-- etc
>
>
For example, for project 229, which contains four SBs, the data reduction directory looks like this:
project229=
=|-- IRC10216_setup3_b6_run_x2</verbatim>
- first SB
| |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)
| |-- X8c - first execution of this SB
| | |-- checklist.txt - checklist (does not need to be prepended by asdm name)
| | |-- qa2 - qa2 directory
| | | |-- qa2_part1.png
| | | |-- textfile.txt
| | | |-- etc
| | |-- uid___A002_X3c9295_X8c.ms
| | |-- uid___A002_X3c9295_X8c.ms.antpos
| | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
| | |-- uid___A002_X3c9295_X8c.ms.split
| | |-- uid___A002_X3c9295_X8c.ms.split.cal
| | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass
| | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots
| | |-- etc
| |-- X528
| | |-- uid___A002_X3dc0e7_X528.ms
| | |-- etc
| |-- Combined - Optional directory to put the combined products in
| |-- calibrated.ms - combined ms. Will not be packaged!
| |-- calibrated_all_setup3_b6.clean.fits - a fits file
| |-- scriptForFluxCalibration.py
| |-- scriptForImaging.py
|-- IRC10216_setup4_b6_run_x2
| |-- X3c9295_X1e0
| | |-- uid___A002_X3c9295_X1e0.ms
| | |-- etc
</verbatim>
  Notes
  • File names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed.
Line: 141 to 112
 
  • Checklists should match one of: *checklist, *checklist.txt, *Checklist, *Checklist.txt [script need updating]

Example output directory structure

Changed:
<
<
After running
>
>
After running
 QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake')
Changed:
<
<
the following package was created:
>
>
the following package was created:
 |-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/
Line: 197 to 164
 
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='member',mode='fake')
Changed:
<
<
produces a second member:
>
>
produces a second member:
 |-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/
Line: 267 to 231
 #####
Deleted:
<
<

 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.
Changed:
<
<


-- MartinZwaan - 20 May 2012

>
>

-- MartinZwaan - 20 May 2012
 
META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"

Revision 82012-10-06 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 24 to 24
 
  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
Changed:
<
<
  • readme: The path to an ascii file with the text of the README header
>
>
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
 
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’copy’ : The files are copied in the normal way.
Line: 219 to 219
  This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler.
Added:
>
>

README

Atacama Large Millimeter/submillimeter Array (ALMA)

#####

Cycle: 0 (Early science)
Project code:
SB name:
PI name:
Project title:
Configuration:
Proposed rms:
CASA version used for reduction: 3.4
Comments from Reducer:

#####

This file describes the content of the tar file you have received. The
full data structure is inserted below.

At this stage, we are releasing data after completion of one SB (excuted
multiple times if required), so you will find only one member_ouss_id
directory.  This directory contains this README file and the following
directories: raw, calibrated, calibration, script, qa2, log, product.

- 'raw' contains the apriori calibrated ms for each execution block,
after being split into the science spectral windows.  This calibration
includes: WVR, Tsys and antenna position corrections and apriori
flagging.
- 'calibrated' contains the fully calibrated ms for each execution block.
- 'calibration' contains the files needed for calibration starting from
the initial ms to the fully calibrated data.   Plots are included.
- 'script' contains the reduction scripts used to process the initial ms
to calibrated data, but also to obtain concatenated data (if more than
one execution) and imaging products.  There are usually several scripts
dealing with different parts of the processing.
- 'product' contains the fits files of the selected image products. 
These will not include all images of scientific value, but will indicate
the quality of the calibration and images.
- 'qa' contains the qa2 reports that show plots and text information
needed to assess the quality of the processing.  The resultant image
rms, compared with that proposed, is given.
- 'log' contains the casa log files.

#####
 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.

Revision 72012-09-14 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 148 to 148
  the following package was created:
Changed:
<
<
|-- 2011.0.00229.S_setup3_b6/
>
>
|-- 2011.0.00229.S/
 | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/
Line: 200 to 200
  produces a second member:
Changed:
<
<
|-- 2011.0.00229.S_setup3_b6/
>
>
|-- 2011.0.00229.S/
 | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/

Revision 62012-08-01 - AnitaRichards

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

The latest version of the packaging script has been committed to cvs here:

Changed:
<
<
AIV/science/qa2/QA2_Packaging_module.py
>
>
AIV/science/qa2/QA2_Packaging_module.py
  The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database

Revision 52012-06-21 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 135 to 135
 |   |   |-- etc
Changed:
<
<
Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed.
>
>
Notes
  • File names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed.
  • QA2 files will be found at the SB level, at the asdm level, and in the qa2 directory. They should match one of: *.jpg, *qa2.txt, *qa2-summ.txt, *.qa2.pdf, *textfile.txt', *qa2_part?.png
  • Checklists should match one of: *checklist, *checklist.txt, *Checklist, *Checklist.txt [script need updating]
 

Example output directory structure

After running

Revision 42012-06-21 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 62 to 62
 packpath='./2011.0.00229.S',append='member',mode='fake')
Changed:
<
<

Example directory structure of input and output

For project 229, which contains four SBs, the data reduction directory looks like this:
>
>

Example data reduction directory structure

The packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following:

 
Changed:
<
<
. |-- IRC10216_setup3_b6_run_x2 | |-- IRC10216_setup3_b6_runx2.imaging.txt | |-- X3c9295_X8c | | |-- X3c9295_X8c.checklist.txt | | |-- qa2 | | | |-- qa2_part1.png | | | |-- textfile.txt | | | |-- etc | | |-- uid___A002_X3c9295_X8c.ms | | |-- uid___A002_X3c9295_X8c.ms.antpos | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | |-- uid___A002_X3c9295_X8c.ms.split | | |-- uid___A002_X3c9295_X8c.ms.split.cal | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots | | |-- etc | |-- X3dc0e7_X528 | | |-- uid___A002_X3dc0e7_X528.ms | | |-- etc | |-- calibrated.ms | |-- calibrated_all_setup3_b6.clean.fits | |-- scriptForFluxCalibration.py | |-- scriptForImaging.py |-- IRC10216_setup4_b6_run_x2 | |-- X3c9295_X1e0 | | |-- uid___A002_X3c9295_X1e0.ms | | |-- etc
>
>
project123 - parent directory for the data reduction of this project SB1 - directory for SB number 1 Xab1 - separate directories for each of the asdm, named after the last few characters for the asdm Xabc calibrated.ms - the combined ms of this SB the imaging and flux calibration scripts fits files optional qa2 notes for the PI SB2 - directory for SB number 2 Xbb1 Xbbc

For single asdms per SB, the structure would look like this:

project123 - parent directory for the data reduction of this project
   SB1 - directory for SB number 1
      Xab1 - a directory for the asdm, named after the last few characters for the asdm
         including also the imaging script and fits files etc      
   SB2 - directory for SB number 2
      Xbb1
 

or, alternatively:

project123 - parent directory for the data reduction of this project
   Xab1 - a directory for the asdm, named after the last few characters for the asdm
      including also the imaging script and fits files etc      
   Xbb1
 

The directories named Xab1 etc should contain uid___xxx.scriptForCalibration.py, uid___xxx.split, (and uid___xxx.split.cal), all calibration tables and the checklist. The only subdirectory in here should be qa2. Please do not create a separate 'Calibration' subdirectory in here. Note that there is no requirement for these directories to start with 'X'. You can call them 'Calibration_Xab1', if you prefer.

Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories.

For example, for project 229, which contains four SBs, the data reduction directory looks like this:
=project229=
|-- IRC10216_setup3_b6_run_x2 - first SB
|   |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)
|   |-- X8c - first execution of this SB
|   |   |-- checklist.txt - checklist (does not need to be prepended by asdm name)
|   |   |-- qa2 - qa2 directory
|   |   |   |-- qa2_part1.png
|   |   |   |-- textfile.txt
|   |   |   |-- etc
|   |   |-- uid___A002_X3c9295_X8c.ms
|   |   |-- uid___A002_X3c9295_X8c.ms.antpos
|   |   |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
|   |   |-- uid___A002_X3c9295_X8c.ms.split
|   |   |-- uid___A002_X3c9295_X8c.ms.split.cal
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots
|   |   |-- etc
|   |-- X528
|   |   |-- uid___A002_X3dc0e7_X528.ms
|   |   |-- etc
|   |-- Combined - Optional directory to put the combined products in
|   |-- calibrated.ms - combined ms. Will not be packaged!
|   |-- calibrated_all_setup3_b6.clean.fits - a fits file
|   |-- scriptForFluxCalibration.py
|   |-- scriptForImaging.py
|-- IRC10216_setup4_b6_run_x2
|   |-- X3c9295_X1e0
|   |   |-- uid___A002_X3c9295_X1e0.ms
|   |   |-- etc

 
Changed:
<
<
Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X3c9295_X8c.checklist.txt' will not be renamed.
>
>
Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed.
 
Added:
>
>

Example output directory structure

 After running
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
Line: 112 to 153
 | | | | | |-- uid___A002_X3dc0e7_X528.ms.split/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split/ | | | | |-- qa/
Changed:
<
<
| | | | | |-- X3dc0e7_X528.checklist.txt
>
>
| | | | | |-- X528.checklist.txt
 | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part1.png | | | | | |-- uid___A002_X3dc0e7_X528__textfile.txt | | | | | |-- uid___A002_X3c9295_X8c__qa2_part1.png
Changed:
<
<
| | | | | |-- X3c9295_X8c.checklist.txt
>
>
| | | | | |-- X8c.checklist.txt
 | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part3.png | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part2.png

Revision 32012-06-20 - MartinZwaan

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 164 to 164
 
Added:
>
>

Tarring up the package

Once a new project is packaged and approved for delivery, the data have to be tarred up, be delivered to the PI and then sent to JAO for ingestion into the Archive (from where they get mirrored to the ARCs). This is the current procedure which might change as we go along.

In order to make data delivery to archival researchers in the future easy, please use tarsplit.py from cvs AIV/science/DSO to create the tarfiles. E.g.

tarsplit.py -n 2012-06-20 2011.01234.S

This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler.

 
Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.

Revision 22012-05-25 - AnitaRichards

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Early Science Packaging Script

Line: 23 to 23
 
  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
Added:
>
>
  • paths should not end in /
 
  • readme: The path to an ascii file with the text of the README header
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.

Revision 12012-05-20 - MartinZwaan

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

Early Science Packaging Script

The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py

The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database

A pdf document with a more detailed description of the packaging script: documentation.pdf

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

from QA2_Packaging_module import *

Then run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
append='',mode='fake')

  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • readme: The path to an ascii file with the text of the README header
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging.
    • mode = ’move’ :The files are moved from the origin to the destination. Then, symbolic links are created at the origin. The symbolic links are never made to whole folders, but only to files (to avoid, for instance, an accidental deletion of whole folders at the destination path by removing the content of a linked folder).
    • mode = ’ticket’ : Similar to ’fake’, but the files to be added to the JIRA ticket are hard-linked. This way, the packager will generate valid ticket tar/zip files, but the measurement sets and tables will not be copied. Recommended for creating the jira ticket file.
  • append: The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).

Examples

For example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run:

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='fake')

This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files.

The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. These are the files that can be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file.

If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory:

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='ticket')

Appending a member

To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='member',mode='fake')

Example directory structure of input and output

For project 229, which contains four SBs, the data reduction directory looks like this:
.
|-- IRC10216_setup3_b6_run_x2
|   |-- IRC10216_setup3_b6_runx2.imaging.txt
|   |-- X3c9295_X8c
|   |   |-- X3c9295_X8c.checklist.txt
|   |   |-- qa2
|   |   |   |-- qa2_part1.png
|   |   |   |-- textfile.txt
|   |   |   |-- etc
|   |   |-- uid___A002_X3c9295_X8c.ms
|   |   |-- uid___A002_X3c9295_X8c.ms.antpos
|   |   |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
|   |   |-- uid___A002_X3c9295_X8c.ms.split
|   |   |-- uid___A002_X3c9295_X8c.ms.split.cal
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass
|   |   |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots
|   |   |-- etc
|   |-- X3dc0e7_X528
|   |   |-- uid___A002_X3dc0e7_X528.ms
|   |   |-- etc
|   |-- calibrated.ms
|   |-- calibrated_all_setup3_b6.clean.fits
|   |-- scriptForFluxCalibration.py
|   |-- scriptForImaging.py
|-- IRC10216_setup4_b6_run_x2
|   |-- X3c9295_X1e0
|   |   |-- uid___A002_X3c9295_X1e0.ms
|   |   |-- etc

Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X3c9295_X8c.checklist.txt' will not be renamed.

After running

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='',mode='fake')
the following package was created:
|-- 2011.0.00229.S_setup3_b6/
|   |-- sg_ouss_id/
|   |   |-- group_ouss_id/
|   |   |   |-- member_ouss_id/
|   |   |   |   |-- raw/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.split/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.split/
|   |   |   |   |-- qa/
|   |   |   |   |   |-- X3dc0e7_X528.checklist.txt
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part1.png
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__textfile.txt
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part1.png
|   |   |   |   |   |-- X3c9295_X8c.checklist.txt
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part3.png
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__qa2_part2.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part2.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__qa2_part3.png
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__textfile.txt
|   |   |   |   |-- script/
|   |   |   |   |   |-- scriptForImaging.py
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.scriptForCalibration.py
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py
|   |   |   |   |   |-- scriptForCalibration.py
|   |   |   |   |   |-- scriptForAprioriCalibration.py
|   |   |   |   |   |-- scriptForFluxCalibration.py
|   |   |   |   |-- log/
|   |   |   |   |   |-- ipython.log
|   |   |   |   |   |-- casapy-2012-05-03T115433.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__casapy-20120502-102144.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__ipython.log
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528__ipython-20120430-142604.log
|   |   |   |   |   |-- uid___A002_X3c9295_X8c__ipython-20120502-102151.log
|   |   |   |   |-- product/
|   |   |   |   |   |-- calibrated_all_setup3_b6.clean.fits
|   |   |   |   |-- calibrated/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.ms.split.cal/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.ms.split.cal/
|   |   |   |   |-- calibration/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.calibration.plots/
|   |   |   |   |   |-- uid___A002_X3c9295_X8c.calibration/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.calibration/
|   |   |   |   |   |-- uid___A002_X3dc0e7_X528.calibration.plots/

Then

QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt',
packpath='./2011.0.00229.S',append='member',mode='fake')
produces a second member:
|-- 2011.0.00229.S_setup3_b6/
|   |-- sg_ouss_id/
|   |   |-- group_ouss_id/
|   |   |   |-- member_ouss_id/
|   |   |   |-- member_ouss_id2/


Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.


-- MartinZwaan - 20 May 2012

META FILEATTACHMENT attachment="documentation.pdf" attr="" comment="" date="1337498449" name="documentation.pdf" path="documentation.pdf" size="62922" stream="documentation.pdf" tmpFilename="/usr/tmp/CGItemp34232" user="MartinZwaan" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ARC TWiki? Send feedback