Procedure for creating ALMA MSs upon user request

(version 2019-09-20)

Purpose

The ALMA project presently only stores raw data in its archive. Users are given software to calibrate the data after download. It was decided by ALMA management that a new service be offered by ESO where users can request the creation and staging of calibrated data for download.

Procedure Summary

The user requests the calibrated data via a helpdesk ticket. The helpdesk ticket should contain the MOUS UID(s) (up to 10) of the data to calibrate. The ticket is received by designated EU ARC personnel who hands on the MOUS UID together with the ALMA user id is by email to AOG.

The EU ARC personnel creates one email per MOUS.

The emails are generated via a script on the cluster which at the same time also places a trigger file in a special AOG directory.

Cycle 0 data requests are rejected and users are told to retrieve that calibrated data directly from the archive.

AOG verifies that the user is entitled to access the data on the day of the request (i.e. the day when AOG receives it). If entitlement is verified, AOG inserts the MOUS UID into a standard command line which is run by AOG on almasciencedev.hq.eso.org as user "arcdistribution".

The command ("stagecalms" developed and tested by D. Petry) connects to the EU ARC cluster and triggers the creation of the calibrated MS. It tars up the completed MS and moves it into a new directory on almasciencedev.hq.eso.org. When the command line completes, it prints to the terminal the newly created URL where the data has been staged.

If there is more than one MOUS to calibrate, AOG can run up to two staging commands in parallel. If there are more than two MOUSs, the commands need to be issued sequentially.

AOG takes the resulting URL(s) and communicates them to the user via a standard email using the user's email on record with the EU ARC personnel in CC who handled on the request.

The EU ARC personnel then closes the helpdesk ticket.

Detailed Procedure

Check for Cycle 0 projects

MOUSs with project code 2011.x.xxxxx.x are not included in this scheme since in Cycle 0, the calibrated data was directly ingested in the archive. If a user requests calibrated data for a Cycle 0 project, the helpdesk ticket owner should remind the user that the calibrated data is already available from the archive.

AOG should not be contacted in case of Cycle 0 projects.

Create email to AOG after receipt of a request via helpdesk

For each requested MOUS, the assigned EU ARC personnel should send one email to AOG.

The email should have the following format:

To: aog@eso.org
CC: <EU ARC person>, dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Body: 
EU ARC contact <EU ARC person>
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X133d_X1244
project <project code>
for user <official user name from helpdesk profile>
email <official user email from profile in helpdesk>
helpdesk ticket <helpdesk ticket number>

In order to create the email, the EU ARC person logs in on the cluster and runs the script

  triggerCalMSCreation <proj code> "<official user name from helpdesk profile>" "<official user email from helpdesk profile>" <helpdesk ticket number> <MOUS UID>

Example:

triggerCalMSCreation.sh 2017.1.00727.S "Judit Szulagyi" judits@phys.ethz.ch 16140 uid://A001/X1296/X675

This script (developed by D. Petry) sends the above email to three addresses: the sending EU ARC person (cluster user), to AOG, and dpetry, and places a textfile with an equivalent of the email into a special directory on the cluster:

/opsw/work/datadelivery/stagecalmsTriggers

The file name is:

trigger-<MOUS UID>-YYYY-MM-DDThh-mm-ss.txt

Example:

trigger-uid___A001_X1296_X675-2019-09-02T14-00-27.txt

The content of the file looks like this:

EU ARC contact <EU ARC person email>
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X133d_X1244
project <project code>
for user <official user name from helpdesk profile>
email <official user email from profile in helpdesk>
helpdesk ticket <helpdesk ticket number>

Example:

EU ARC contact evkampe@eso.org
CC email dpetry@eso.org
ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X1296_X675

Dear AOG,
please create the calibrated MS for MOUS uid___A001_X1296_X675
project 2017.1.00727.S
for user Judit Szulagyi
email judits@phys.ethz.ch
helpdesk ticket 16140

AOG will install a cronjob on an AOG machine to look for new triggers once per hour.

Prevention of duplicate triggers

Once AOG has completely processed a trigger, AOG removes the corresponding trigger file from /opsw/work/datadelivery/stagecalmsTriggers . I.e. the trigger file is left in the directory until the staging is completed and the notification email to the ALMA user has gone out.

The script triggerCalMSCreation.sh checks existing "active" triggers and warns the user if a triggerfile for the given MOUS already exists in order to prevent unwanted duplicate triggers. The user can then abort the duplicate request.

Verification of entitlement and availability

After receiving the request email, AOG uses the contained user ID and the MOUS UID to determine whether the user is entitled at that moment to obtain this data. F. Stoehr has provided a draft query:

select asa_science.project_code,b.firstname PI_FIRSTNAME,b.lastname
PI_LASTNAME, b.email PI_EMAIL, b.account_id PI_ACCOUNTID, b.preferredarc
PI_ARC, a.firstname DELEGEE_FIRSTNAME,a.lastname
DELEGEE_LASTNAME,a.email DELEGEE_EMAIL, a.account_id DELEGEE_ACCOUNTID,
a.preferredarc DELEGEE_ARC from alma.dp_delegation join alma.account a
on delegee_rh_id=a.REQUEST_HANDLER_ID join alma.account b on
PI_RH_ID=b.REQUEST_HANDLER_ID join alma.asa_science on
asa_science.project_code = dp_delegation.project_code where
member_ouss_id = 'uid://A001/X134b/X41' group by
asa_science.project_code,b.firstname ,b.lastname , b.email,
b.account_id, b.preferredarc, a.firstname ,a.lastname,a.email,
a.account_id, a.preferredarc

where the MOUS ID uid://A001/X134b/X41 of course has to be replaced with the MOUS ID that one wants to check

If the user is entitled (i.e. is the PI or a delegee as identified by the user email), AOG also checks if the MOUS is in state delivered and if the MOUS is not in state QA3InProgress.

Reply to EU ARC personnel (helpdesk contact) in case the user is not entitled

If AOG determines that the user is not entitled at the moment to access the data, AOG replies to the request email and writes the text

NOT ENTITLED

above the quoted original email text.

This concludes AOG's involvement.

The EU ARC person (helpdesk contact) replies via helpdesk to the user using the following text:

You presently do not have access permission for the requested data.
Please wait until the data becomes public.

Reply to EU ARC person (helpdesk contact) in case the user or the MOUS does not exist

If AOG determines that the data is faulty, i.e. the MOUS UID is invalid (entitlement query results in zero rows), AOG replies to the request email and writes the text

INVALID MOUS UID

above the quoted original email text.

The EU ARC person can then communicate with the user and try to fix the information.

The corrected new request to AOG should be using a fresh template and no history of the previous emails!

Reply to EU ARC person (helpdesk contact) in case the MOUS is not in state delivered

If AOG determines that the MOUS UID is not in state delivered, AOG replies to the request email and writes the text

MOUS NOT DELIVERED

above the quoted original email text.

The EU ARC person (helpdesk contact) can then investigate further the status of the MOUS and tell the user what can be done.

Creation and staging of the MS(s)

With the information from the request email, AOG proceeds to run the MS creation on almasciencedev.hq.eso.org:

AOG logs in as arcdistribution@almasciencedev.hq.eso.org and types

stagecalms.sh PROJCODE MOUSUID HELPDESKTICKETNUM

This will trigger jobs on the EU ARC cluster.

Up to two stagecalms instances can be run at the same time.

The typical terminal output of the command looks like this

The staging directory will be accessible at http://almascience.eso.org/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a
Looking for cluster node with low load ...
DATAMACHINE chosen: arcp9
Working as arcproc@arcp9.hq.eso.org ...
(re-)creating directory /opsw/work/pipeproc/autopipeline/tmp/WORK11018
Now generating the calibrated MS(s) ...
Checking for pre-existing MSs for the following EBs ...
uid___A002_X7e4988_X1f2
MS for uid___A002_X7e4988_X1f2 was not found among the already staged MSs. Will not search further ...
None or not all of the needed MSs found: Will regenerate all the MSs ...
Copying over the generated MSs ...
receiving incremental file list
uid___A002_X7e4988_X1f2.ms.split.cal.tar

sent 42 bytes  received 15449500342 bytes  95073848.52 bytes/sec
total size is 15445729280  speedup is 1.00
Deleting redundant logs ...
Deleting temporary work directory ...

Staging of calibrated MS(s) for 2012.1.00357.S uid___A002_X788a57_X29 helpdesk ticket #15270 completed
at http://almascience.eso.org/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a
Data will be deleted 28 days from Wed Jul 17 17:09:18 UTC 2019unless you put a .donotremove into the directory.

There can be the additional information

Old project, using RHEL5 build of CASA.

which indicates that the required CASA version is so old that it needs to be run on a special RHEL5 machine. This message is just for information.

Re-use of existing calibrated MSs on almasciencedev

stagecalms has the special feature that it will minimise the computing effort for the data calibration by first searching for existing copies of the requested calibrated MS on almasciencedev.

If a copy of the MS already exists (because a different user requested it less than 28 days before the present request), then, instead of recreating the MS again, stagecalms will simply create a hard link to the existing MS in the new download directory.

In case this happens, stagecalms completes very quickly and there are messages like

Found pre-existing /home/arcdistribution/arcdistribution/preview/8b6db8a851c558b1669e7a0d987e6f87/uid___A002_Xd15514_X8bfa.ms.split.cal.tar .
   Will try to use this instead of recreating it (making hard link) ...

on the terminal.

Otherwise, there is a message

MS for uid___A002_X9aa6ef_X15a1 was not found among the already staged MSs. Will not search further ...
None or not all of the needed MSs found: Will regenerate all the MSs ...

The messages are just for information. AOG does not have to take special action either way.

What to do in case stagecalms completes with an error

In case there is an error, stagecalms will produce a terminal output similar to

Failed. Please investigate  /var/www/html/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a and possibly remove incomplete data belonging to MOUS uid://A001/X11f/X9c
The working directory /opsw/work/pipeproc/autopipeline/tmp/WORK23486 on the EU ARC cluster was not removed.

The entire terminal output including the command line which was entered by AOG should be copied and pasted into an email to dpetry@eso.org for investigation.

dpetry will then give further instructions.

Email to the PI after successful completion of stagecalms

Once stagecalms has completed (which can take hours), AOG should create an email and send it to the user with the EU ARC person, AOG, and dpetry@eso.org in BCC.

The format of the email should be the following:

To: <alma user email address obtained via ALMA user id>
BCC: <EU ARC person (helpdesk contact)>, dpetry@eso.org, aog@eso.org
Reply-to: <no_reply_eu_data_delivery@eso.org>
Subject: "Re: ALMA: REQUEST FOR CALIBRATED DATA MOUS uid___A001_X133d_X1244, helpdesk ticket <ticket number>"
Body: 
Your calibrated data is ready for download from
http://almascience.eso.org/arcdistribution/preview/cdb386950af8ee148b3a9bc633942c6a

It will be deleted 28 days from now.

Kind regards,

The ESO ALMA Team

where the download link is copied and pasted from the terminal output of stagecalms in the previous step.

This concludes AOG's work for the given MOUS.

The staged data will be automatically deleted after 28 days by a cronjob on almasciencedev.

EU ARC person (helpdesk contact) closes helpdesk ticket

Once the helpdesk person receives the email with the download link from AOG, he./she copies and pastes the email into a last comment on the ticket and closes the ticket.

-- Dirk Petry - 2019-08-05

Comments