Procedure for creating ALMA MSs upon user request

(version 2022-04-04)

Purpose

The ALMA project presently only stores raw data in its archive. Users are given software to calibrate the data after download. It was decided by ALMA management that a new service be offered by ESO where users can request the creation and staging of calibrated data for download.

Procedure Summary

The user requests the calibrated data via a helpdesk ticket. The helpdesk ticket should contain the MOUS UID(s) (up to 10) of the data to calibrate. The ticket is received by designated EU ARC personnel who hands on the MOUS UID together with the ALMA user id is by email to AOG.

Optionally, the user can also request that only the data for one or more calibrator fields is provided. In this case, the user needs to specify the exact name(s) of the fields and no entitlement check will be carried out since calibrator data are immediately public.

The EU ARC personnel creates one email per MOUS.

The emails are generated via a script on the cluster which at the same time also places a trigger file in a special AOG directory.

Cycle 0 data requests are rejected and users are told to retrieve that calibrated data directly from the archive.

AOG verifies that the user is entitled to access the data on the day of the request (i.e. the day when AOG receives it). If entitlement is verified, AOG inserts the MOUS UID into a standard command line which is run by AOG on almasciencedev.hq.eso.org as user "arcdistribution".

The command ("stagecalms" developed and maintained by D. Petry) connects to the EU ARC cluster and triggers the creation of the calibrated MS. It tars up the completed MS and moves it into a new directory on almasciencedev.hq.eso.org. When the command line completes, it prints to the terminal the newly created URL where the data has been staged.

If there is more than one MOUS to calibrate, AOG can run up to two staging commands in parallel. If there are more than two MOUSs, the commands need to be issued sequentially.

AOG takes the resulting URL(s) and communicates them to the user via a standard email using the user's email on record with the EU ARC personnel in CC who handled on the request.

The EU ARC personnel then closes the helpdesk ticket.

Detailed Procedure

Check for Cycle 0 projects

MOUSs with project code 2011.x.xxxxx.x are not included in this scheme since in Cycle 0, the calibrated data was directly ingested in the archive. If a user requests calibrated data for a Cycle 0 project, the helpdesk ticket owner should remind the user that the calibrated data is already available from the archive.

AOG should not be contacted in case of Cycle 0 projects.

Check for presence of QA0_PASS data

Before contacting AOG, it should also be verified that there is actually calibrated data which we can offer to the user, i.e. there are QA0_PASS EBs in the archive.

This step was integrated in the the call to triggerCalMSCreation.sh described in the following section.

Create email to AOG after receipt of a request via helpdesk

For each requested MOUS, the assigned EU ARC personnel should send one email to AOG via the special script "triggerCalMSCreation".

The email should have the following format:

To: aog@eso.org
CC: <EU ARC person>, dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Body: 
EU ARC contact <EU ARC person>
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X133d_X1244
project <project code>
calfield <calibrator field names separated by comma, no spaces permitted>
for user <official user name from helpdesk profile>
email <official user email from profile in helpdesk>
helpdesk ticket <helpdesk ticket number>

NOTE: the calfield line is optional!

In order to create the email, the EU ARC person logs in on the cluster and runs the script

  triggerCalMSCreation.sh <proj code> "<official user name from helpdesk profile>" "<official user email from helpdesk profile>" <helpdesk ticket number> <MOUS UID> [ "calibrator field name(s)"]

Examples:

triggerCalMSCreation.sh 2017.1.00727.S "Judit Szulagyi" judits@phys.ethz.ch 16140 uid://A001/X1296/X675
triggerCalMSCreation.sh 2021.1.01582.S "Franco Nero" "franco@gmail.com" 12345 uid://A001/X159a/X6b "J0510+1800,J0321+1234"

This script (developed and maintained by D. Petry) sends the above email to three addresses: the sending EU ARC person (cluster user), to AOG, and dpetry, and places a textfile with an equivalent of the email into a special directory on the cluster:

/opsw/work/datadelivery/stagecalmsTriggers

The file name is:

trigger-<MOUS UID>-YYYY-MM-DDThh-mm-ss.txt

Example:

trigger-uid___A001_X1296_X675-2019-09-02T14-00-27.txt

The content of the file looks like this (note that the line starting with "calfield" is optional):

EU ARC contact <EU ARC person email>
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X133d_X1244
project <project code>
calfield <calibrator field names separated by comma, no spaces permitted>
for user <official user name from helpdesk profile>
email <official user email from profile in helpdesk>
helpdesk ticket <helpdesk ticket number>

Example:

EU ARC contact evkampe@eso.org
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X1296_X675

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X1296_X675
project 2017.1.00727.S
calfield J0510+1800,J0321+1234
for user Judit Szulagyi
email judits@phys.ethz.ch
helpdesk ticket 16140

AOG is running a cronjob on an AOG machine to look for new triggers once per hour.

In case there are no QA0_PASS data, the script triggerCalMSCreation.sh will terminate with this message:

   ERROR: No QA0_PASS data found for this MOUS. Please double-check MOUS UID.
                    If confirmed, please inform the user that ALMA cannot offer calibrated data for this MOUS.

and not contact AOG.

Prevention of duplicate triggers

Once AOG has completely processed a trigger, AOG removes the corresponding trigger file from /opsw/work/datadelivery/stagecalmsTriggers . I.e. the trigger file is left in the directory until the staging is completed and the notification email to the ALMA user has gone out.

The script triggerCalMSCreation.sh checks existing "active" triggers and warns the user if a triggerfile for the given MOUS already exists in order to prevent unwanted duplicate triggers. The user can then abort the duplicate request.

Verification of entitlement and availability

After receiving the request email, AOG uses the contained user ID and the MOUS UID to determine whether the user is entitled at that moment to obtain this data. The database queries to achieve this were developed together with F. Stoehr. AOG is maintaining the queries.

If the calfield parameter is set (i.e. the user is only requesting calibrator data), the user is always entitled to obtain the data and the database query is skipped.

If the user is entitled (i.e. is the PI or a delegee as identified by the user email or only calibrators were requested), AOG also checks if the MOUS is in state delivered and if the MOUS is not in state QA3InProgress.

Only if these conditions are met, AOG goes ahead and starts the creation of the calibrated MS.

Reply to EU ARC personnel (helpdesk contact) in case the user is not entitled

If AOG determines that the user is not entitled at the moment to access the data, AOG replies to the request email and writes the text

NOT ENTITLED

above the quoted original email text.

This concludes AOG's involvement.

The EU ARC person (helpdesk contact) replies via helpdesk to the user using the following text:

You presently do not have access permission for the requested data.
Please wait until the data becomes public.

Reply to EU ARC person (helpdesk contact) in case the user or the MOUS does not exist

If AOG determines that the data is faulty, i.e. the MOUS UID is invalid (entitlement query results in zero rows), AOG replies to the request email and writes the text

INVALID MOUS UID

above the quoted original email text.

The EU ARC person can then communicate with the user and try to fix the information.

The corrected new request to AOG should be using a fresh template and no history of the previous emails!

Reply to EU ARC person (helpdesk contact) in case the MOUS is not in state delivered

If AOG determines that the MOUS UID is not in state delivered, AOG replies to the request email and writes the text

MOUS NOT DELIVERED

above the quoted original email text.

The EU ARC person (helpdesk contact) can then investigate further the status of the MOUS and tell the user what can be done.

Creation and staging of the MS(s)

With the information from the request email, AOG proceeds to run the MS creation on almasciencedev.hq.eso.org:

AOG logs in as arcdistribution@almasciencedev.hq.eso.org and types

stagecalms.sh PROJCODE MOUSUID HELPDESKTICKETNUM [X CALFIELD]

This will trigger jobs on the EU ARC cluster.

The optional X CALFIELD is only typed when a calfield line was provided in the request email.

Examples:

stagecalms.sh 2019.1.00337.S uid://A001/X14d7/X160 18704

stagecalms.sh 2019.1.00337.S uid://A001/X14d7/X160 18704 X "J0217-0820,J0238+1636"

Up to two stagecalms instances can be run at the same time.

The typical terminal output of the command looks like this

The staging directory will be accessible at http://almascience.eso.org/arcdistribution/preview/8672af71dac32e0ad9703dae3438fcac
Looking for cluster node with low load ...
DATAMACHINE chosen: arcp14
Working as arcproc@arcp14.hq.eso.org ...
(re-)creating directory /opsw/work/pipeproc/autopipeline/tmp/WORK23067
Now generating the calibrated MS(s) ...
Checking for pre-existing MSs for the following EBs ...
uid___A002_Xd33a12_X15be
MS for uid___A002_Xd33a12_X15be was not found among the already staged MSs. Will not search further ...
None or not all of the needed MSs found: Will regenerate all the MSs ...
Generated the job script. Now submitting and waiting for completion ...
Completed. Determining the list of generated MSs ...
Copying over the generated MSs ...
receiving incremental file list
uid___A002_Xd33a12_X15be.ms.split.cal.tar

sent 42 bytes  received 5765728719 bytes  64421550.40 bytes/sec
total size is 5764321280  speedup is 1.00
Deleting redundant logs ...
Deleting temporary work directory ...

Staging of calibrated MS(s) for 2018.1.01406.S uid___A001_X133d_X3cf3 helpdesk ticket #16267 completed
at http://almascience.eso.org/arcdistribution/preview/8672af71dac32e0ad9703dae3438fcac
Data will be deleted 28 days from Fri Sep 27 14:04:04 UTC 2019 unless you put a .donotremove into the directory.

There can be the additional information

Old project, using RHEL5 build of CASA.

which indicates that the required CASA version is so old that it needs to be run on a special RHEL5 machine. This message is just for information.

If the calfield was provided, the output starts with

*** Based on your selection, only data for field <calfield name(s)> will be staged. ***

Re-use of existing calibrated MSs on almasciencedev

stagecalms has the special feature that it will minimise the computing effort for the data calibration by first searching for existing copies of the requested calibrated MS on almasciencedev.

If a copy of the MS already exists (because a different user requested it less than 28 days before the present request), then, instead of recreating the MS again, stagecalms will simply create a hard link to the existing MS in the new download directory.

In case this happens, stagecalms completes very quickly and there are messages like

Found pre-existing /home/arcdistribution/arcdistribution/preview/8b6db8a851c558b1669e7a0d987e6f87/uid___A002_Xd15514_X8bfa.ms.split.cal.tar .
   Will try to use this instead of recreating it (making hard link) ...

on the terminal.

Otherwise, there is a message

MS for uid___A002_X9aa6ef_X15a1 was not found among the already staged MSs. Will not search further ...
None or not all of the needed MSs found: Will regenerate all the MSs ...

The messages are just for information. AOG does not have to take special action either way.

What to do in case stagecalms completes with an error

In case there is an error, stagecalms will produce a terminal output similar to

Failed. Please investigate  /var/www/html/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a and possibly remove incomplete data belonging to MOUS uid://A001/X11f/X9c
The working directory /opsw/work/pipeproc/autopipeline/tmp/WORK23486 on the EU ARC cluster was not removed.

The entire terminal output including the command line which was entered by AOG should be copied and pasted into an email to dpetry@eso.org for investigation.

dpetry will then give further instructions.

Email to the PI after successful completion of stagecalms

Once stagecalms has completed (which can take hours), AOG should create an email and send it to the user with the EU ARC person, AOG, and dpetry@eso.org in BCC.

The format of the email should be the following:

To: <alma user email address obtained via ALMA user id>
BCC: <EU ARC person (helpdesk contact)>, dpetry@eso.org, aog@eso.org
Reply-to: <no_reply_eu_data_delivery@eso.org>
Subject: "Re: ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244, helpdesk ticket <ticket number>"
Body: 
Your calibrated data is ready for download from
http://almascience.eso.org/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a

It will be deleted 28 days from now.

Kind regards,

The ESO ALMA Team

where the download link is copied and pasted from the terminal output of stagecalms in the previous step.

This concludes AOG's work for the given MOUS.

The staged data will be automatically deleted after 28 days by a cronjob on almasciencedev.

EU ARC person (helpdesk contact) closes helpdesk ticket

Once the helpdesk person receives the email with the download link from AOG, he./she copies and pastes the email into a last comment on the ticket and closes the ticket.

-- Dirk Petry - 2019-10-01, 2022-04-04

stats sheet

Comments