Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Added: | ||||||||
> > | (DEPRECATED - use new page at https://confluence.alma.cl/display/MDR/QA+Packager![]() | |||||||
The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 18 to 18 | ||||||||
For a standard script-generator-assisted analysis, you can continue to run the script in the following way: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript=' | |||||||
If you just want to test, set mode = 'fake'. | ||||||||
Line: 35 to 35 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 50 to 50 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 135 to 135 | ||||||||
PIscript="./scriptForPI.py") | ||||||||
Changed: | ||||||||
< < | NOTE: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
> > | NOTE: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options. | ||||||||
Line: 154 to 154 | ||||||||
The result is QA2_PASS and now the manual imaging results are to be packaged for delivery. | ||||||||
Added: | ||||||||
> > | NOTE: this also covers the case of full polarisation calibration after PL intensity calibration. The supplemental scriptForPolCalibration and the full pol. diagnostic plots will be picked up. | |||||||
Then run the packager as follows:
QA_Packager(origpath='<top dir of the analysis dir>', packpath='<project id = top dir of output package>', readme='<path to readme file>', mode='man-img-only', | ||||||||
Changed: | ||||||||
< < | style='cycle6-pipe', # this style needs to the chosen in this case | |||||||
> > | style='cycleX-pipe', # a -pipe style needs to the chosen in this case | |||||||
PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Example: | ||||||||
Line: 168 to 170 | ||||||||
packpath='./2018.100456.S', readme='myREADME.txt', mode='man-img-only', | ||||||||
Changed: | ||||||||
< < | style='cycle6-pipe', | |||||||
> > | style='cycle8-pipe', | |||||||
PIscript="./scriptForPI.py") | ||||||||
Changed: | ||||||||
< < | NOTE again: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
> > | NOTE again: where it says "top dir of the analysis dir", what meant here is the directory that contains the "science_goal" directory, e.g. origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
Packaging of TP data | ||||||||
Line: 218 to 220 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script. | ||||||||
Changed: | ||||||||
< < | -- ![]() | |||||||
> > | -- ![]() | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 98 to 98 | ||||||||
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='man-img-only', cycle=6) | ||||||||
Added: | ||||||||
> > | NOTE: If you only have a "working" and no "products" directory, create a link to the "working" directory under the name "products" on the same level as the "working" directory, i.e. run
cd <top dir of pipeline results>/S*OUS*/G*OUS*/M*OUS*/ ln -sf working products cd ../../../../..before you run the stager. | |||||||
Case 2a: packaging after external manual imaging including calibration and imaging productsA package as created in Case 1a above has arrived at the ARC (or a JAO analyst). |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 84 to 84 | ||||||||
QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3) | ||||||||
Changed: | ||||||||
< < | Case 1b: incremental packaging of only the science imaging productsAn SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the science imaging products into the archive. | |||||||
> > | Case 1b: incremental packaging of only the manual imaging productsAn SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the manual imaging products into the archive. | |||||||
Then run the packager as follows:
QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>', staged_root='<project ID == top dir of output package>', | ||||||||
Changed: | ||||||||
< < | mode='sci-img-only', | |||||||
> > | mode='man-img-only', | |||||||
cycle= Example: | ||||||||
Changed: | ||||||||
< < | QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='sci-img-only', cycle=6) | |||||||
> > | QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='man-img-only', cycle=6) | |||||||
Case 2a: packaging after external manual imaging including calibration and imaging products | ||||||||
Line: 132 to 132 | ||||||||
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options. | ||||||||
Changed: | ||||||||
< < | Case 2b: packaging after external imaging including only science imaging products | |||||||
> > | Case 2b: packaging after external imaging including only manual imaging products | |||||||
A package as created in Case 1a above has arrived at the ARC (or a JAO analyst). | ||||||||
Line: 144 to 144 | ||||||||
The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory. | ||||||||
Changed: | ||||||||
< < | The result is QA2_PASS and now the science imaging results are to be packaged for delivery. | |||||||
> > | The result is QA2_PASS and now the manual imaging results are to be packaged for delivery. | |||||||
Then run the packager as follows:
QA_Packager(origpath='<top dir of the analysis dir>', packpath='<project id = top dir of output package>', readme='<path to readme file>', | ||||||||
Changed: | ||||||||
< < | mode='sci-img-only', | |||||||
> > | mode='man-img-only', | |||||||
style='cycle6-pipe', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed | ||||||||
Line: 159 to 159 | ||||||||
QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S', readme='myREADME.txt', | ||||||||
Changed: | ||||||||
< < | mode='sci-img-only', | |||||||
> > | mode='man-img-only', | |||||||
style='cycle6-pipe', PIscript="./scriptForPI.py") | ||||||||
Line: 210 to 210 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script. | ||||||||
Changed: | ||||||||
< < | -- ![]() | |||||||
> > | -- ![]() | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 6 to 6 | ||||||||
The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository![]() | ||||||||
Deleted: | ||||||||
< < | The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() A pdf document with a more detailed description of the packaging script: documentation.pdf As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline. As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline. As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1". As of 25 Aug 2016, the packager also accepts a tgz extension in addition to the tar.gz extension. | |||||||
Running the script | ||||||||
Line: 29 to 18 | ||||||||
For a standard script-generator-assisted analysis, you can continue to run the script in the following way: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript=' | |||||||
Changed: | ||||||||
< < | This will just test the packaging (mode = 'fake'). | |||||||
> > | If you just want to test, set mode = 'fake'. | |||||||
The parameters mean the following: | ||||||||
Line: 39 to 28 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Line: 59 to 49 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
In order to perform the final packaging, run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle3-nopipe', PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle6-nopipe', PIscript=' | |||||||
Packaging including Pipeline products | ||||||||
Line: 77 to 69 | ||||||||
Both use cases are described below: | ||||||||
Changed: | ||||||||
< < | Case 1: | |||||||
> > | Case 1a: packaging for external manual imaging | |||||||
An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere
Then run the packager as follows: | ||||||||
Line: 86 to 78 | ||||||||
staged_root=' | ||||||||
Changed: | ||||||||
< < | cycle= | |||||||
> > | cycle= | |||||||
Example: QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3) | ||||||||
Changed: | ||||||||
< < | Case 2:A package as created in Case 1 above has arrived at the ARC (or a JAO analyst). | |||||||
> > | Case 1b: incremental packaging of only the science imaging productsAn SB was processed by the Cycle 6 pipeline and its calibration was already ingested. Now, additional manual imaging was done in situ and an incremental package is to be created for ingestion of the science imaging products into the archive.Then run the packager as follows: QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>', staged_root='<project ID == top dir of output package>', mode='sci-img-only', cycle=<num>) # where num is 6 Example: QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='sci-img-only', cycle=6)
Case 2a: packaging after external manual imaging including calibration and imaging productsA package as created in Case 1a above has arrived at the ARC (or a JAO analyst). | |||||||
The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case. | ||||||||
Line: 109 to 115 | ||||||||
packpath=' | ||||||||
Changed: | ||||||||
< < | gzip_caltables=True, style='cycle4-pipe', # this style needs to the chosen in this case | |||||||
> > | style='cycle6-pipe', # this style needs to the chosen in this case | |||||||
PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Example: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='./2016.100456.S-analysis', packpath='./2016.100456.S', | |||||||
> > | QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S', | |||||||
readme='myREADME.txt', mode='hard', | ||||||||
Changed: | ||||||||
< < | gzip_caltables=True, style='cycle4-pipe', | |||||||
> > | style='cycle6-pipe', | |||||||
PIscript="./scriptForPI.py") | ||||||||
Line: 128 to 132 | ||||||||
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options. | ||||||||
Added: | ||||||||
> > | Case 2b: packaging after external imaging including only science imaging productsA package as created in Case 1a above has arrived at the ARC (or a JAO analyst). However, the calibration products have already been ingested in the archive. The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case. The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI. The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory. The result is QA2_PASS and now the science imaging results are to be packaged for delivery. Then run the packager as follows: QA_Packager(origpath='<top dir of the analysis dir>', packpath='<project id = top dir of output package>', readme='<path to readme file>', mode='sci-img-only', style='cycle6-pipe', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed Example: QA_Packager(origpath='./2018.100456.S-analysis', packpath='./2018.100456.S', readme='myREADME.txt', mode='sci-img-only', style='cycle6-pipe', PIscript="./scriptForPI.py") NOTE again: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
Packaging of TP dataSince version 1.43 of the packager, handling of Total Power data is supported. | ||||||||
Line: 169 to 207 | ||||||||
Acknowledgements | ||||||||
Changed: | ||||||||
< < | Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support. | |||||||
> > | Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the original first useful version of the script. | |||||||
Changed: | ||||||||
< < | -- DirkPetry - 25 Aug 2016 | |||||||
> > | -- ![]() | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 85 to 85 | ||||||||
QA_Pipeline_Stager(pipeline_root=' | ||||||||
Changed: | ||||||||
< < | PIscript=" | |||||||
> > | PIscript=" | |||||||
Example: | ||||||||
Changed: | ||||||||
< < | QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py") | |||||||
> > | QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py", cycle=3) | |||||||
Case 2: | ||||||||
Line: 109 to 110 | ||||||||
readme=' | ||||||||
Changed: | ||||||||
< < | style='cycle3-pipe1', # this style needs to the chosen in this case | |||||||
> > | style='cycle4-pipe', # this style needs to the chosen in this case | |||||||
PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Example: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='./2013.100456.S-analysis', packpath='./2013.100456.S', | |||||||
> > | QA_Packager(origpath='./2016.100456.S-analysis', packpath='./2016.100456.S', | |||||||
readme='myREADME.txt', mode='hard', gzip_caltables=True, | ||||||||
Changed: | ||||||||
< < | style='cycle3-pipe1', | |||||||
> > | style='cycle4-pipe', | |||||||
PIscript="./scriptForPI.py") |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 16 to 16 | ||||||||
As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1". | ||||||||
Added: | ||||||||
> > | As of 25 Aug 2016, the packager also accepts a tgz extension in addition to the tar.gz extension. | |||||||
Running the scriptThe script should be run from within casa. It can be run from any directory. The import is done with | ||||||||
Line: 55 to 57 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
In order to perform the final packaging, run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle3-nopipe', PIscript=' | |||||||
Packaging including Pipeline products | ||||||||
Line: 84 to 88 | ||||||||
PIscript=" Example: | ||||||||
Changed: | ||||||||
< < | QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py") | |||||||
> > | QA_Pipeline_Stager('2015.100456.S-2016-08-08T123456', '2015.100456.S', mode='copy', PIscript="./scriptForPI.py") | |||||||
Case 2: | ||||||||
Line: 105 to 109 | ||||||||
readme=' | ||||||||
Changed: | ||||||||
< < | style='cycle2-pipe1', # this style needs to the chosen in this case | |||||||
> > | style='cycle3-pipe1', # this style needs to the chosen in this case | |||||||
PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
Example: | ||||||||
Line: 114 to 118 | ||||||||
readme='myREADME.txt', mode='hard', gzip_caltables=True, | ||||||||
Changed: | ||||||||
< < | style='cycle2-pipe1', | |||||||
> > | style='cycle3-pipe1', | |||||||
PIscript="./scriptForPI.py") | ||||||||
Line: 167 to 171 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 8 Dec 2015 | |||||||
> > | -- DirkPetry - 25 Aug 2016 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 118 to 118 | ||||||||
PIscript="./scriptForPI.py") | ||||||||
Added: | ||||||||
> > | NOTE: where it says "top dir of output package", what meant here is the directory that looks like this: origpath='analysis/2015.1.01247.S.MOUS.uid___A001_X2f6_X66.SBNAME.ULASJ1243_a_03_TE | |||||||
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
Packaging of TP data |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 14 to 14 | ||||||||
As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline. | ||||||||
Added: | ||||||||
> > | As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1". | |||||||
Running the scriptThe script should be run from within casa. It can be run from any directory. The import is done with | ||||||||
Line: 51 to 53 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 162 to 164 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 1 Nov 2015 | |||||||
> > | -- DirkPetry - 8 Dec 2015 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 12 to 12 | ||||||||
As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline. | ||||||||
Added: | ||||||||
> > | As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline. | |||||||
Running the scriptThe script should be run from within casa. It can be run from any directory. The import is done with | ||||||||
Line: 49 to 51 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 160 to 162 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 30 Oct 2015 | |||||||
> > | -- DirkPetry - 1 Nov 2015 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 49 to 49 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Line: 118 to 120 | ||||||||
Since version 1.43 of the packager, handling of Total Power data is supported. | ||||||||
Changed: | ||||||||
< < | The typical call to package TP data looks like this: | |||||||
> > | The typical call to package manually calibrated TP data looks like this: | |||||||
Changed: | ||||||||
< < | QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2012.1.00501.S', style='cycle2-nopipe', mode='hard') | |||||||
> > | QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-nopipe', mode='hard') | |||||||
The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py . | ||||||||
Line: 128 to 130 | ||||||||
Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them. | ||||||||
Added: | ||||||||
> > | The typical call to stage pipeline-calibrated TP data looks like this:
QA_Pipeline_Stager('2013.1.00911.S_2015_08_07T16_10_05.732', '2013.1.00911.S', mode='copy', PIscript="./scriptForPI.py") The typical call to package pipeline-calibrated TP data looks like this: QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-pipe1', mode='hard', PIscript="./scriptForPI.py")Note: the latter expects fits images in the "calibrated" directory. | |||||||
Packager Unit TestIn the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working. | ||||||||
Line: 146 to 160 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 20 Aug 2015 | |||||||
> > | -- DirkPetry - 30 Oct 2015 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
ALMA QA2 Packaging Script | ||||||||
Line: 114 to 114 | ||||||||
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options. | ||||||||
Added: | ||||||||
> > | Packaging of TP dataSince version 1.43 of the packager, handling of Total Power data is supported. The typical call to package TP data looks like this: QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2012.1.00501.S', style='cycle2-nopipe', mode='hard') The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py . It will then override other modes and switch to TP mode. The typical output tree is shown here. Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them. | |||||||
Packager Unit TestIn the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working. | ||||||||
Line: 130 to 144 | ||||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. | ||||||||
Changed: | ||||||||
< < | Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | |||||||
> > | Thanks to Ivan Marti-Vidal (Onsala) for continued support. | |||||||
Changed: | ||||||||
< < | -- DirkPetry - 9 Jan 2015 | |||||||
> > | -- DirkPetry - 20 Aug 2015 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Changed: | ||||||||
< < | Early Science Packaging Script | |||||||
> > | ALMA QA2 Packaging Script
| |||||||
The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository![]() | ||||||||
Line: 65 to 67 | ||||||||
Both use cases are described below: | ||||||||
Changed: | ||||||||
< < | Case 1: | |||||||
> > | Case 1: | |||||||
An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere
Then run the packager as follows: | ||||||||
Line: 79 to 81 | ||||||||
QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py") | ||||||||
Changed: | ||||||||
< < | Case 2: | |||||||
> > | Case 2: | |||||||
A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).
The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case. | ||||||||
Line: 126 to 128 | ||||||||
Changed: | ||||||||
< < | Acknowledgements | |||||||
> > | Acknowledgements | |||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 9 Sept 2014 | |||||||
> > | -- DirkPetry - 9 Jan 2015 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 110 to 110 | ||||||||
PIscript="./scriptForPI.py") | ||||||||
Added: | ||||||||
> > | See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options. | |||||||
Packager Unit TestIn the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 130 to 130 | ||||||||
-- DirkPetry - 9 Sept 2014 | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Changed: | ||||||||
< < | The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
> > | The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository![]() | |||||||
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() | ||||||||
Line: 128 to 128 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 2 Sept 2014 | |||||||
> > | -- DirkPetry - 9 Sept 2014 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 130 to 130 | ||||||||
-- DirkPetry - 2 Sept 2014 | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 38 to 38 | ||||||||
| ||||||||
Deleted: | ||||||||
< < |
In order to perform the final packaging, run the script with
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')
| |||||||
| ||||||||
Changed: | ||||||||
< < | * style: alternative way to specify a number of option at once. | |||||||
> > |
| |||||||
| ||||||||
Line: 57 to 50 | ||||||||
| ||||||||
Added: | ||||||||
> > | In order to perform the final packaging, run the script with
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py') | |||||||
Packaging including Pipeline productsIf you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package. | ||||||||
Line: 129 to 128 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 22 Aug 2014 | |||||||
> > | -- DirkPetry - 2 Sept 2014 | |||||||
Changed: | ||||||||
< < |
| |||||||
> > |
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 131 to 131 | ||||||||
-- DirkPetry - 22 Aug 2014 | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 21 to 21 | ||||||||
For a standard script-generator-assisted analysis, you can continue to run the script in the following way: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript=' | |||||||
This will just test the packaging (mode = 'fake'). | ||||||||
Line: 32 to 32 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 42 to 42 | ||||||||
In order to perform the final packaging, run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',
append='',mode='hard', noms=True, PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript=' | |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Line: 96 to 97 | ||||||||
packpath=' | ||||||||
Deleted: | ||||||||
< < | append='', | |||||||
gzip_caltables=True, style='cycle2-pipe1', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed | ||||||||
Line: 106 to 106 | ||||||||
packpath='./2013.100456.S', readme='myREADME.txt', mode='hard', | ||||||||
Deleted: | ||||||||
< < | append='', | |||||||
gzip_caltables=True, style='cycle2-pipe1', PIscript="./scriptForPI.py") | ||||||||
Line: 130 to 129 | ||||||||
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 14 Aug 2014 | |||||||
> > | -- DirkPetry - 22 Aug 2014 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 21 to 21 | ||||||||
For a standard script-generator-assisted analysis, you can continue to run the script in the following way: | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript=' | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',PIscript=' | |||||||
This will just test the packaging (mode = 'fake'). | ||||||||
Line: 42 to 42 | ||||||||
In order to perform the final packaging, run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S', | |||||||
append='',mode='hard', noms=True, PIscript='
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Changed: | ||||||||
< < | Early Science Packaging Script | |||||||
> > | Early Science Packaging Script | |||||||
The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py![]() | ||||||||
Line: 8 to 8 | ||||||||
A pdf document with a more detailed description of the packaging script: documentation.pdf | ||||||||
Changed: | ||||||||
< < | Running the script | |||||||
> > | As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.
Running the script | |||||||
The script should be run from within casa. It can be run from any directory. The import is done with
from QA2_Packaging_module import * | ||||||||
Changed: | ||||||||
< < | Then run the script with | |||||||
> > |
Packaging of standard script-generator-assisted analysis productsFor a standard script-generator-assisted analysis, you can continue to run the script in the following way: | |||||||
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='<yourjaomirror>/science/qa2/scriptForPI.py', noms=True, append='',mode='fake') | ||||||||
Added: | ||||||||
> > | This will just test the packaging (mode = 'fake').
The parameters mean the following: | |||||||
| ||||||||
Line: 29 to 38 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > | In order to perform the final packaging, run the script with | |||||||
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
append='',mode='hard', noms=True, PIscript='
| ||||||||
Line: 45 to 56 | ||||||||
| ||||||||
Added: | ||||||||
> > | Packaging including Pipeline productsIf you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package. Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI, the packager enables you to do that as well. Both use cases are described below:
Case 1:An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhereThen run the packager as follows: QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>', staged_root='<project ID == top dir of output package>', mode='<hard, copy, fake>', # default is 'copy' PIscript="<path to scriptForPI.py>") # needs to be included in the package Example: QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")
Case 2:A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case. The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI. The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory. The result is QA2_PASS and now the whole data is to be packaged for delivery. Then run the packager as follows: QA_Packager(origpath='<top dir of the analysis dir>', packpath='<project id = top dir of output package>', readme='<path to readme file>', mode='<hard,copy,fake,ticket>', append='', gzip_caltables=True, style='cycle2-pipe1', # this style needs to the chosen in this case PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed Example: QA_Packager(origpath='./2013.100456.S-analysis', packpath='./2013.100456.S', readme='myREADME.txt', mode='hard', append='', gzip_caltables=True, style='cycle2-pipe1', PIscript="./scriptForPI.py")
Packager Unit TestIn the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working. Usage: tar xvzf QA2_Packaging_module.unit-test.tgz cd QA2_Packaging_module.unit-test cp ../QA2_Packaging_module.py . start casapy and run execfile('QA2_Packaging_module.unit-test.py') | |||||||
Acknowledgements | ||||||||
Changed: | ||||||||
< < | Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. | |||||||
> > | Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support. | |||||||
Changed: | ||||||||
< < | -- DirkPetry - 20 Feb 2014 | |||||||
> > | -- DirkPetry - 14 Aug 2014 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 44 to 44 | ||||||||
| ||||||||
Deleted: | ||||||||
< < | ExamplesFor example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run: QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake', noms=True) This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files. The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. The latter should be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file. If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory: QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='ticket', noms=True)
Appending a memberTo append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake')
NOTE: The following documentation was valid for cycle 0 style delivery and will shortly be updated to Cycle 1 style
Example data reduction directory structureThe packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following: project123 - parent directory for the data reduction of this project SB1 - directory for SB number 1 Xab1 - separate directories for each of the asdm, named after the last few characters for the asdm Xabc calibrated.ms - the combined ms of this SB the imaging and flux calibration scripts fits files optional qa2 notes for the PI SB2 - directory for SB number 2 Xbb1 Xbbc For single asdms per SB, the structure would look like this:
project123 - parent directory for the data reduction of this project SB1 - directory for SB number 1 Xab1 - a directory for the asdm, named after the last few characters for the asdm including also the imaging script and fits files etc SB2 - directory for SB number 2 Xbb1 or, alternatively:
project123 - parent directory for the data reduction of this project Xab1 - a directory for the asdm, named after the last few characters for the asdm including also the imaging script and fits files etc Xbb1
The directories named Xab1 etc should contain Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories.
For example, for project 229, which contains four SBs, the data reduction directory looks like this: Notes
Example output directory structureAfter runningQA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake')the following package was created: |-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ | | | | |-- raw/ | | | | | |-- uid___A002_X3dc0e7_X528.ms.split/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split/ | | | | |-- qa/ | | | | | |-- X528.checklist.txt | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part1.png | | | | | |-- uid___A002_X3dc0e7_X528__textfile.txt | | | | | |-- uid___A002_X3c9295_X8c__qa2_part1.png | | | | | |-- X8c.checklist.txt | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part3.png | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part3.png | | | | | |-- uid___A002_X3c9295_X8c__textfile.txt | | | | |-- script/ | | | | | |-- scriptForImaging.py | | | | | |-- uid___A002_X3dc0e7_X528.ms.scriptForCalibration.py | | | | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | | | | |-- scriptForCalibration.py | | | | | |-- scriptForAprioriCalibration.py | | | | | |-- scriptForFluxCalibration.py | | | | |-- log/ | | | | | |-- ipython.log | | | | | |-- casapy-2012-05-03T115433.log | | | | | |-- uid___A002_X3c9295_X8c__casapy-20120502-102144.log | | | | | |-- uid___A002_X3c9295_X8c__ipython.log | | | | | |-- uid___A002_X3dc0e7_X528__ipython-20120430-142604.log | | | | | |-- uid___A002_X3c9295_X8c__ipython-20120502-102151.log | | | | |-- product/ | | | | | |-- calibrated_all_setup3_b6.clean.fits | | | | |-- calibrated/ | | | | | |-- uid___A002_X3dc0e7_X528.ms.split.cal/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split.cal/ | | | | |-- calibration/ | | | | | |-- uid___A002_X3c9295_X8c.calibration.plots/ | | | | | |-- uid___A002_X3c9295_X8c.calibration/ | | | | | |-- uid___A002_X3dc0e7_X528.calibration/ | | | | | |-- uid___A002_X3dc0e7_X528.calibration.plots/ Then QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake')produces a second member: |-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ | | | |-- member_ouss_id2/
Tarring up the packageOnce a new project is packaged and approved for delivery, the data have to be tarred up, be delivered to the PI and then sent to JAO for ingestion into the Archive (from where they get mirrored to the ARCs). This is the current procedure which might change as we go along.In order to make data delivery to archival researchers in the future easy, please use tarsplit.py from cvs AIV/science/DSO to create the tarfiles. E.g.
tarsplit.py -n 2012-06-20 2011.01234.S This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler.
READMEAtacama Large Millimeter/submillimeter Array (ALMA) ##### Cycle: 0 (Early science) Project code: SB name: PI name: Project title: Configuration: Proposed rms: CASA version used for reduction: 3.4 Comments from Reducer: ##### This file describes the content of the tar file you have received. The full data structure is inserted below. At this stage, we are releasing data after completion of one SB (excuted multiple times if required), so you will find only one member_ouss_id directory. This directory contains this README file and the following directories: raw, calibrated, calibration, script, qa2, log, product. - 'raw' contains the apriori calibrated ms for each execution block, after being split into the science spectral windows. This calibration includes: WVR, Tsys and antenna position corrections and apriori flagging. - 'calibrated' contains the fully calibrated ms for each execution block. - 'calibration' contains the files needed for calibration starting from the initial ms to the fully calibrated data. Plots are included. - 'script' contains the reduction scripts used to process the initial ms to calibrated data, but also to obtain concatenated data (if more than one execution) and imaging products. There are usually several scripts dealing with different parts of the processing. - 'product' contains the fits files of the selected image products. These will not include all images of scientific value, but will indicate the quality of the calibration and images. - 'qa' contains the qa2 reports that show plots and text information needed to assess the quality of the processing. The resultant image rms, compared with that proposed, is given. - 'log' contains the casa log files. ##### | |||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 28 to 28 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 39 to 39 | ||||||||
| ||||||||
Added: | ||||||||
> > | * style: alternative way to specify a number of option at once.
| |||||||
Examples | ||||||||
Line: 238 to 242 | ||||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. | ||||||||
Changed: | ||||||||
< < | -- DirkPetry - 07 Feb 2014 | |||||||
> > | -- DirkPetry - 20 Feb 2014 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 16 to 16 | ||||||||
Then run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='./uid___A002_X71e4ae_X317.ms.scriptForCalibration.py', append='',mode='fake') | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript=' | |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > | Run the script with | |||||||
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', | ||||||||
Changed: | ||||||||
< < | append='',mode='hard') | |||||||
> > | append='',mode='hard', noms=True, PIscript=' | |||||||
| ||||||||
Line: 46 to 45 | ||||||||
For example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run:
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', | ||||||||
Changed: | ||||||||
< < | packpath='./2011.0.00229.S',append='',mode='fake') | |||||||
> > | packpath='./2011.0.00229.S',append='',mode='fake', noms=True) | |||||||
This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files. | ||||||||
Line: 56 to 55 | ||||||||
If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory:
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', | ||||||||
Changed: | ||||||||
< < | packpath='./2011.0.00229.S',append='',mode='ticket') | |||||||
> > | packpath='./2011.0.00229.S',append='',mode='ticket', noms=True) | |||||||
Appending a member | ||||||||
Line: 65 to 64 | ||||||||
packpath='./2011.0.00229.S',append='member',mode='fake') | ||||||||
Added: | ||||||||
> > |
NOTE: The following documentation was valid for cycle 0 style delivery and will shortly be updated to Cycle 1 style | |||||||
Example data reduction directory structureThe packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following: | ||||||||
Line: 236 to 238 | ||||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. | ||||||||
Changed: | ||||||||
< < | -- MartinZwaan - 20 May 2012 | |||||||
> > | -- DirkPetry - 07 Feb 2014 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 16 to 16 | ||||||||
Then run the script with | ||||||||
Changed: | ||||||||
< < | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', | |||||||
> > | QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',PIscript='./uid___A002_X71e4ae_X317.ms.scriptForCalibration.py', | |||||||
append='',mode='fake')
| ||||||||
Added: | ||||||||
> > |
| |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 30 to 30 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S',
append='',mode='hard')
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 25 to 25 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Line: 49 to 50 | ||||||||
This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files. | ||||||||
Changed: | ||||||||
< < | The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. These are the files that can be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file. | |||||||
> > | The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. The latter should be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file. | |||||||
If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory: |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Changed: | ||||||||
< < | The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
> > | The latest version of the packaging script is available via the EU ARC svn repository here: AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Changed: | ||||||||
< < | The latest version of the packaging script has been committed to cvs here:
AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
> > | The latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
Changed: | ||||||||
< < | The contents of the package should be consistent with the guidelines described here:
http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() | |||||||
> > | The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() | |||||||
A pdf document with a more detailed description of the packaging script: documentation.pdf | ||||||||
Line: 26 to 24 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Examples | ||||||||
Line: 54 to 57 | ||||||||
packpath='./2011.0.00229.S',append='',mode='ticket') | ||||||||
Deleted: | ||||||||
< < | ||||||||
Appending a member | ||||||||
Changed: | ||||||||
< < | To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member': | |||||||
> > | To append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member': | |||||||
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake') | ||||||||
Line: 103 to 104 | ||||||||
Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories. | ||||||||
Changed: | ||||||||
< < | For example, for project 229, which contains four SBs, the data reduction directory looks like this: =project229= |-- IRC10216_setup3_b6_run_x2 - first SB| |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)| |-- X8c - first execution of this SB | | |-- checklist.txt - checklist (does not need to be prepended by asdm name)| | |-- qa2 - qa2 directory| | | |-- qa2_part1.png | | | |-- textfile.txt | | | |-- etc | | |-- uid___A002_X3c9295_X8c.ms | | |-- uid___A002_X3c9295_X8c.ms.antpos | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | |-- uid___A002_X3c9295_X8c.ms.split | | |-- uid___A002_X3c9295_X8c.ms.split.cal | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots | | |-- etc | |-- X528 | | |-- uid___A002_X3dc0e7_X528.ms | | |-- etc | |-- Combined - Optional directory to put the combined products in | |-- calibrated.ms - combined ms. Will not be packaged! | |-- calibrated_all_setup3_b6.clean.fits - a fits file | |-- scriptForFluxCalibration.py | |-- scriptForImaging.py |-- IRC10216_setup4_b6_run_x2 | |-- X3c9295_X1e0 | | |-- uid___A002_X3c9295_X1e0.ms | | |-- etc | |||||||
> > | For example, for project 229, which contains four SBs, the data reduction directory looks like this:project229= - first SB| |-- IRC10216_setup3_b6_runx2.imaging.txt - notes for the PI (optional)| |-- X8c - first execution of this SB | | |-- checklist.txt - checklist (does not need to be prepended by asdm name)| | |-- qa2 - qa2 directory| | | |-- qa2_part1.png | | | |-- textfile.txt | | | |-- etc | | |-- uid___A002_X3c9295_X8c.ms | | |-- uid___A002_X3c9295_X8c.ms.antpos | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | |-- uid___A002_X3c9295_X8c.ms.split | | |-- uid___A002_X3c9295_X8c.ms.split.cal | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots | | |-- etc | |-- X528 | | |-- uid___A002_X3dc0e7_X528.ms | | |-- etc | |-- Combined - Optional directory to put the combined products in | |-- calibrated.ms - combined ms. Will not be packaged! | |-- calibrated_all_setup3_b6.clean.fits - a fits file | |-- scriptForFluxCalibration.py | |-- scriptForImaging.py |-- IRC10216_setup4_b6_run_x2 | |-- X3c9295_X1e0 | | |-- uid___A002_X3c9295_X1e0.ms | | |-- etc </verbatim> | |||||||
Notes
| ||||||||
Line: 141 to 112 | ||||||||
Example output directory structure | ||||||||
Changed: | ||||||||
< < | After running | |||||||
> > | After running | |||||||
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake') | ||||||||
Changed: | ||||||||
< < | the following package was created: | |||||||
> > | the following package was created: | |||||||
|-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | ||||||||
Line: 197 to 164 | ||||||||
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake') | ||||||||
Changed: | ||||||||
< < | produces a second member: | |||||||
> > | produces a second member: | |||||||
|-- 2011.0.00229.S/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | ||||||||
Line: 267 to 231 | ||||||||
##### | ||||||||
Deleted: | ||||||||
< < | ||||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > | -- MartinZwaan - 20 May 2012 | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 24 to 24 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 219 to 219 | ||||||||
This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler. | ||||||||
Added: | ||||||||
> > | READMEAtacama Large Millimeter/submillimeter Array (ALMA) ##### Cycle: 0 (Early science) Project code: SB name: PI name: Project title: Configuration: Proposed rms: CASA version used for reduction: 3.4 Comments from Reducer: ##### This file describes the content of the tar file you have received. The full data structure is inserted below. At this stage, we are releasing data after completion of one SB (excuted multiple times if required), so you will find only one member_ouss_id directory. This directory contains this README file and the following directories: raw, calibrated, calibration, script, qa2, log, product. - 'raw' contains the apriori calibrated ms for each execution block, after being split into the science spectral windows. This calibration includes: WVR, Tsys and antenna position corrections and apriori flagging. - 'calibrated' contains the fully calibrated ms for each execution block. - 'calibration' contains the files needed for calibration starting from the initial ms to the fully calibrated data. Plots are included. - 'script' contains the reduction scripts used to process the initial ms to calibrated data, but also to obtain concatenated data (if more than one execution) and imaging products. There are usually several scripts dealing with different parts of the processing. - 'product' contains the fits files of the selected image products. These will not include all images of scientific value, but will indicate the quality of the calibration and images. - 'qa' contains the qa2 reports that show plots and text information needed to assess the quality of the processing. The resultant image rms, compared with that proposed, is given. - 'log' contains the casa log files. ##### | |||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 148 to 148 | ||||||||
the following package was created: | ||||||||
Changed: | ||||||||
< < | |-- 2011.0.00229.S_setup3_b6/ | |||||||
> > | |-- 2011.0.00229.S/ | |||||||
| |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ | ||||||||
Line: 200 to 200 | ||||||||
produces a second member: | ||||||||
Changed: | ||||||||
< < | |-- 2011.0.00229.S_setup3_b6/ | |||||||
> > | |-- 2011.0.00229.S/ | |||||||
| |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging ScriptThe latest version of the packaging script has been committed to cvs here: | ||||||||
Changed: | ||||||||
< < | AIV/science/qa2/QA2_Packaging_module.py | |||||||
> > | AIV/science/qa2/QA2_Packaging_module.py![]() | |||||||
The contents of the package should be consistent with the guidelines described here:
http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database![]() |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 135 to 135 | ||||||||
| | |-- etc | ||||||||
Changed: | ||||||||
< < | Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed. | |||||||
> > | Notes
| |||||||
Example output directory structureAfter running |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 62 to 62 | ||||||||
packpath='./2011.0.00229.S',append='member',mode='fake') | ||||||||
Changed: | ||||||||
< < | Example directory structure of input and outputFor project 229, which contains four SBs, the data reduction directory looks like this: | |||||||
> > | Example data reduction directory structureThe packaging script follows the conventions described in the "How to reduce ALMA science data?" document. In case there are multiple executions for each SB, the recommended structure is the following: | |||||||
Changed: | ||||||||
< < | . |-- IRC10216_setup3_b6_run_x2 | |-- IRC10216_setup3_b6_runx2.imaging.txt | |-- X3c9295_X8c | | |-- X3c9295_X8c.checklist.txt | | |-- qa2 | | | |-- qa2_part1.png | | | |-- textfile.txt | | | |-- etc | | |-- uid___A002_X3c9295_X8c.ms | | |-- uid___A002_X3c9295_X8c.ms.antpos | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | |-- uid___A002_X3c9295_X8c.ms.split | | |-- uid___A002_X3c9295_X8c.ms.split.cal | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots | | |-- etc | |-- X3dc0e7_X528 | | |-- uid___A002_X3dc0e7_X528.ms | | |-- etc | |-- calibrated.ms | |-- calibrated_all_setup3_b6.clean.fits | |-- scriptForFluxCalibration.py | |-- scriptForImaging.py |-- IRC10216_setup4_b6_run_x2 | |-- X3c9295_X1e0 | | |-- uid___A002_X3c9295_X1e0.ms | | |-- etc | |||||||
> > | project123 - parent directory for the data reduction of this project
SB1 - directory for SB number 1
Xab1 - separate directories for each of the asdm, named after the last few characters for the asdm
Xabc
calibrated.ms - the combined ms of this SB
the imaging and flux calibration scripts
fits files
optional qa2 notes for the PI
SB2 - directory for SB number 2
Xbb1
Xbbc
For single asdms per SB, the structure would look like this:
project123 - parent directory for the data reduction of this project SB1 - directory for SB number 1 Xab1 - a directory for the asdm, named after the last few characters for the asdm including also the imaging script and fits files etc SB2 - directory for SB number 2 Xbb1 or, alternatively:
project123 - parent directory for the data reduction of this project Xab1 - a directory for the asdm, named after the last few characters for the asdm including also the imaging script and fits files etc Xbb1
The directories named Xab1 etc should contain Some data reducers may find it convenient to put results (images, cubes etc) in separate directories at the same level as the X directories. The packaging script will find the data products in those directories.
For example, for project 229, which contains four SBs, the data reduction directory looks like this: | |||||||
Changed: | ||||||||
< < | Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X3c9295_X8c.checklist.txt' will not be renamed. | |||||||
> > | Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X8c.checklist.txt' will not be renamed. | |||||||
Added: | ||||||||
> > | Example output directory structure | |||||||
After running
QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', | ||||||||
Line: 112 to 153 | ||||||||
| | | | | |-- uid___A002_X3dc0e7_X528.ms.split/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split/ | | | | |-- qa/ | ||||||||
Changed: | ||||||||
< < | | | | | | |-- X3dc0e7_X528.checklist.txt | |||||||
> > | | | | | | |-- X528.checklist.txt | |||||||
| | | | | |-- uid___A002_X3dc0e7_X528__qa2_part1.png | | | | | |-- uid___A002_X3dc0e7_X528__textfile.txt | | | | | |-- uid___A002_X3c9295_X8c__qa2_part1.png | ||||||||
Changed: | ||||||||
< < | | | | | | |-- X3c9295_X8c.checklist.txt | |||||||
> > | | | | | | |-- X8c.checklist.txt | |||||||
| | | | | |-- uid___A002_X3dc0e7_X528__qa2_part3.png | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part2.png |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 164 to 164 | ||||||||
Added: | ||||||||
> > | Tarring up the packageOnce a new project is packaged and approved for delivery, the data have to be tarred up, be delivered to the PI and then sent to JAO for ingestion into the Archive (from where they get mirrored to the ARCs). This is the current procedure which might change as we go along.In order to make data delivery to archival researchers in the future easy, please use tarsplit.py from cvs AIV/science/DSO to create the tarfiles. E.g.
tarsplit.py -n 2012-06-20 2011.01234.S This helps to split the data into chunks that allow for efficient network transfer to and from the ARCs. Also it makes sure that the tarfiles obey to the naming convention which helps with the tracking as well as the mini-datapacker that will run in the Request Handler. | |||||||
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Early Science Packaging Script | ||||||||
Line: 23 to 23 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Added: | ||||||||
> > |
Early Science Packaging ScriptThe latest version of the packaging script has been committed to cvs here: AIV/science/qa2/QA2_Packaging_module.py
The contents of the package should be consistent with the guidelines described here:
http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database A pdf document with a more detailed description of the packaging script: documentation.pdf
Running the scriptThe script should be run from within casa. It can be run from any directory. The import is done with
from QA2_Packaging_module import * Then run the script with QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./2011.0.00XXX.S', append='',mode='fake')
ExamplesFor example, for project 229, which contains four SBs, one of them is called 'IRC10216_setup3_b6_run_x2', you can run: QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake') This will create a dummy directory structure 2011.0.00229.S. It is recommended to run it in this mode first. If everything looks OK, run it again with mode='hard', which is the recommended mode for packaging. The fake mode is very fast, but the 'hard' mode may take a while because split will have to be run several times to split out the relevant data columns from .split and .split.cal files. The script will also produce two files with extension 'ticket.tar' and 'ticket.zip'. These are the files that can be attached to the data reduction ticket. They contain the README, the qa2 files, the scripts, and the expected size of the package. Attaching zip files to the jira ticket seems very convenient because these can be opened directly from the jira ticket, without having to download the whole file. If the script is run in fake mode, note that the .ticket files are also dummy files. In order to make real ticket files, run the script in 'hard' mode, or use mode='ticket' (recommended). The latter mode will create a valid ticket file, but will only create a dummy packaging directory: QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='ticket')
Appending a memberTo append another SB to the package, for example IRC10216_setup4_b6_run_x2, run with option append='member':QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake')
Example directory structure of input and outputFor project 229, which contains four SBs, the data reduction directory looks like this:. |-- IRC10216_setup3_b6_run_x2 | |-- IRC10216_setup3_b6_runx2.imaging.txt | |-- X3c9295_X8c | | |-- X3c9295_X8c.checklist.txt | | |-- qa2 | | | |-- qa2_part1.png | | | |-- textfile.txt | | | |-- etc | | |-- uid___A002_X3c9295_X8c.ms | | |-- uid___A002_X3c9295_X8c.ms.antpos | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | |-- uid___A002_X3c9295_X8c.ms.split | | |-- uid___A002_X3c9295_X8c.ms.split.cal | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass | | |-- uid___A002_X3c9295_X8c.ms.split.ap_pre_bandpass.plots | | |-- etc | |-- X3dc0e7_X528 | | |-- uid___A002_X3dc0e7_X528.ms | | |-- etc | |-- calibrated.ms | |-- calibrated_all_setup3_b6.clean.fits | |-- scriptForFluxCalibration.py | |-- scriptForImaging.py |-- IRC10216_setup4_b6_run_x2 | |-- X3c9295_X1e0 | | |-- uid___A002_X3c9295_X1e0.ms | | |-- etc Note that the file names that are not unique (do not start with uid name, or X...) will be prepended with the uid automatically. So 'qa2_part1.png' will be renamed 'uid___A002_X3dc0e7_X528__qa2_part1.png'. But 'X3c9295_X8c.checklist.txt' will not be renamed. After running QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup3_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='',mode='fake')the following package was created: |-- 2011.0.00229.S_setup3_b6/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ | | | | |-- raw/ | | | | | |-- uid___A002_X3dc0e7_X528.ms.split/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split/ | | | | |-- qa/ | | | | | |-- X3dc0e7_X528.checklist.txt | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part1.png | | | | | |-- uid___A002_X3dc0e7_X528__textfile.txt | | | | | |-- uid___A002_X3c9295_X8c__qa2_part1.png | | | | | |-- X3c9295_X8c.checklist.txt | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part3.png | | | | | |-- uid___A002_X3dc0e7_X528__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part2.png | | | | | |-- uid___A002_X3c9295_X8c__qa2_part3.png | | | | | |-- uid___A002_X3c9295_X8c__textfile.txt | | | | |-- script/ | | | | | |-- scriptForImaging.py | | | | | |-- uid___A002_X3dc0e7_X528.ms.scriptForCalibration.py | | | | | |-- uid___A002_X3c9295_X8c.ms.scriptForCalibration.py | | | | | |-- scriptForCalibration.py | | | | | |-- scriptForAprioriCalibration.py | | | | | |-- scriptForFluxCalibration.py | | | | |-- log/ | | | | | |-- ipython.log | | | | | |-- casapy-2012-05-03T115433.log | | | | | |-- uid___A002_X3c9295_X8c__casapy-20120502-102144.log | | | | | |-- uid___A002_X3c9295_X8c__ipython.log | | | | | |-- uid___A002_X3dc0e7_X528__ipython-20120430-142604.log | | | | | |-- uid___A002_X3c9295_X8c__ipython-20120502-102151.log | | | | |-- product/ | | | | | |-- calibrated_all_setup3_b6.clean.fits | | | | |-- calibrated/ | | | | | |-- uid___A002_X3dc0e7_X528.ms.split.cal/ | | | | | |-- uid___A002_X3c9295_X8c.ms.split.cal/ | | | | |-- calibration/ | | | | | |-- uid___A002_X3c9295_X8c.calibration.plots/ | | | | | |-- uid___A002_X3c9295_X8c.calibration/ | | | | | |-- uid___A002_X3dc0e7_X528.calibration/ | | | | | |-- uid___A002_X3dc0e7_X528.calibration.plots/
Then QA_Packager(origpath='/scratch04/arcproc/project229/IRC10216_setup4_b6_run_x2',readme='./README.header.txt', packpath='./2011.0.00229.S',append='member',mode='fake')produces a second member: |-- 2011.0.00229.S_setup3_b6/ | |-- sg_ouss_id/ | | |-- group_ouss_id/ | | | |-- member_ouss_id/ | | | |-- member_ouss_id2/
AcknowledgementsThanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the script.
|