Overview of the Phase 3 Process
The Phase 3 process consists of the following steps, each of them being described in detail below.
Available features of the Phase 3 system including optional actions are described thereafter in Phase 3 User Help.
- Registering your Phase 3 submission
- Preparing your data
- Uploading your data to ESO
- Verifying compliance with data format requirements
- Uploading the data release description
- Finalising your data submission
1. Registering your Phase 3 submission
To access the Phase 3 Release Manager – a dedicated web application for controlling the entire Phase 3 data submission process – navigate your browser to http://www.eso.org/rm.
When being prompted by the system, please log in to the ESO User Portal using your personal ESO User Portal credentials.
If your programme is not listed under Data Collections or if the display is empty, please register your Phase 3 submission by contacting Phase 3 operations support staff at https://support.eso.org/ quoting the subject “REQUEST FOR PHASE 3 PROGRAMME <PPP.C-NNNN>”, where <PPP.C-NNNN> denotes your ESO programme identifier.
Thereafter, usually within one working day, you will receive the confirmation by e-mail that the programme has been registered and it will show up in the Phase 3 Release Manager as a new Data Collection.
2. Preparing your data
Data should be prepared according to the data formats specified in the ESO Science Data Products Standard [latest version].
Please also take into account the Phase 3 FAQ section when preparing your data!
3. Uploading your data to ESO
Please transfer your data to the dedicated Phase 3 FTP server
ftp://phase3ftp.eso.org/<name>/batch_###
using the directory path displayed in the Phase 3 Release Manager. The path includes the <name> of your Phase 3 Data Collection and the Batch ID.
The supported software for the file transfer are LFTP or FileZilla, for example:
lftp -u username phase3ftp.eso.org
where username is your ESO User Portal credentials.
Possible problems concerning the file format need to be resolved first and the corrected files must be uploaded before proceeding. The list of Phase 3 checks carried out by the system at this stage is available here.
4. Verifying compliance with data format requirements
Once having completed the data transfer, please “CLOSE” the submission in order to proceed (by pressing the respective button in the Phase 3 Release Manager).
Your signal triggers the Phase 3 format and provenance verification process on the ESO server, which requires time between a couple of minutes and several hours to complete depending on the amount of data.
You will receive an e-mail notification upon completion of the process. If problems are reported at this stage, these need to be fixed before proceeding. In this case the directory in the Phase3ftp server is still write-mode enabled.
Note: “closing” the Phase 3 batch turns the respective FTP directory into read-only mode and you cannot upload further files, nor replace or delete previously uploaded files.
5. Uploading the data release description
Please upload the data release description document associated with your Phase 3 data (PDF format) to provide an account of the release content, the originating observations, the calibration and data reduction procedures, the data quality, the data format, and, possibly, the scientific context of the programme.
More about the preparation of the data release description
6. Finalising your data submission
Before the data are made publicly available to the community at large, please press “SUBMIT” to confirm the completeness and consistency of the actually uploaded data with respect to the data release description.
To this end the Phase 3 Release Manager application provides a detailed summary of the submitted data in terms of content, i.e. total number of files, data types, date of observation, sky coverage etc. (“Content Summary”).
Note: The final data submission must be done by the PI, this action cannot be delegated.
Then, the submitted data and documentation will undergo a final check by ESO, dubbed Phase 3 content validation, before being published through the ESO Science Archive Facility.
In case any problems are spotted at this stage, you will receive a detailed report including instructions of how to proceed.
Phase 3 User Help
This section describes the Phase 3 system in terms of available features including optional actions.
Phase 3 Software Documentation
Please find below a detailed documentation of the Phase 3 software system:
- Phase 3 checks carried out at upload time: summary of implemented checks.
- Upon closing the batch, provenance and catalogue unique identifier checks are executed.
- Phase 3 Software User Manual (in preparation)
Delegating Phase 3 permissions
As principal investigator you can delegate part of the Phase 3 tasks (e.g. uploading data, testing data compliance, verifying completeness of the release etc.) to other members within your team. To grant/revoke Phase 3 permissions please visit:
http://www.eso.org/ace/phase3Program/index
Permissions can be managed per Phase 3 Data Collection, that is permissions apply to all batches belonging to a given collection.
Any Phase 3 actions can be delegated except for the final confirmation that the actually uploaded data are complete and consistent with respect to the documentation.
Multiple delegates can, in principle, work on the same data release, but it is in the sole responsibility of the principal investigator to ensure the overall consistency.
Phase 3 survey manager role
The Phase 3 survey manager is a user role dedicated to ESO public survey programmes.
Before starting a Phase 3 submission, the P.I. of the survey programme must assign the survey manager role to her/himself or another user.
The survey manager has the overall responsibly for delivering Phase 3 data products according to the originally stipulated survey management plan.
The survey manager has the same permissions as the P.I. including release submission, except for re-assigning the survey manager itself.
Phase 3 FTP upload
Using any FTP client of your choice, please connect to
ftp://phase3ftp.eso.org
and login using your personnel ESO User Portal credentials.
Then, having changed to the directory
/<Data collection name>/batch_<ID>
according to the data collection name and batch ID displayed in the Phase 3 Release Manager, you can start transferring data.
The directory and the data deposited therein are visible and accessible only to the P.I. who owns the Phase 3 collection, the survey manager and appointed delegates. In the course of preparing the data release you can modify files on the FTP server as needed, including file replacements and removal.
FITS Checksums and MD5 Hash
FITS checksums are employed to safeguard the data transfer against transmission errors. Thus, it is essential that you update the respective checksum keywords beforehand.
For non-FITS format ancillary files there is a special mechanism in place to identify possible transmission errors by employing their MD5 hashes. The MD5 hash must be specified using the dedicated keyword ASSOMi in the science file defining the data set (cf. ESO SDP standard).
Updating previously released Phase 3 data
Previously released Phase 3 data can be updated by submitting a new Phase 3 batch. The following use cases are supported:
- consecutive updating data releases with increasing signal-to-noise ratio in the final combined data product may occur according to the progress of acquiring more and more observational data in the course of a programme aiming at very deep integrations on the same target. This happens typically when observations are scheduled over more than one period;
- reprocessing of previously released data motivated by improved data calibration procedures, improved quality control checks or enhanced data reduction algorithms, leading to a higher data quality than before.
Please feel free to contact Phase 3 operations support staff at https://support.eso.org/ quoting the subject “PHASE 3” if you are in doubt whether your use case is supported.
Identifying old with new files
Depending on the adopted naming convention for the new files, there are two different options to provide the required link between new (updating) and old (to be updated) files.
In case of data sets, consisting of one (main) science file and one or more ancillary files, the entire set of files need to be updated all at once. Ancillary files cannot be updated independently of the main science file.
a) CHANGES.USER. Each line in the CHANGES.USER special file must follow the pattern
<new_file> UPDATES <old_file>
with <new_file> referring to a file in this batch submission, and <old_file> referring to a previously submitted file within the same data collection/survey (both in terms of the originally submitted filename, also known as ORIGFILE). There should be no trailing/leading spaces to the filenames.
Only science data files (PRODCATG='SCIENCE*') are referenced in "CHANGES.USER". Ancillary files are not to be included.
b) Automatic tracking of data versions. If a new file is submitted with the same filename (ORIGFILE) as the earlier version, then these two files are automatically recognized by the system as different data versions without a line in CHANGES.USER being needed for this file.
Important: within one and the same Phase 3 batch CHANGES.USER and automatic tracking can be used both as long as the information provided in different ways are overall consistent, otherwise, in case of inconsistencies, an error will be raised at the time of data verification.
Updating procedure
Though the submission for updating Phase 3 batches basically follows the same procedure as for batches containing new data, the following points deserve particular attention.
- Improved versions of previously submitted files must be uploaded to a new Phase 3 batch within your existing data collection.
- Unless automatic version tracking is used (see above) the CHANGES.USER file need to be uploaded. The latter file can be uploaded via FTP before closing the batch. Differently, the user will be shown a pop up window by the Release Manager interface, with the request to upload the file, upon closing the batch.
- Data release description: please include a section documenting changes and improvements of the new data with respect to the old data.
- Important check for updates: when reviewing the completeness and consistency of the Phase 3 batch prior to submission, please check that the update status is correct (“UPDATING”) and the respective file counts are correct.
Result
Once the updating batch has been published the new version of the data becomes available, replacing the old obsolete version in any default archive query. Obsolete data are only accessible on demand.
Notes:
- NEW+UPDATING: updates can be combined with the submission of new data, i.e. data products which have not been submitted via Phase 3 yet, within the same Phase 3 batch.
- The automatic tracking of data versions implies that previous data cannot be referenced in terms of ORIGFILE.
Test Mode
To verify the compliance of a test data set with data format requirements without actually submitting those data for archiving and publication, i.e. without going through the entire Phase 3 data submission work flow, go to the RM and choose “TEST YOUR DATA” from the action menu of your Phase 3 batch.
In this way the Phase 3 format and provenance verification process will be executed, after which the batch directory will be reset to read/write mode, i.e. to the “OPEN” state.
You will receive a notification via e-mail upon completion of the validation process and the resulting test report will be accessible from the RM.
Reporting a Data Problem
Users of the ESO Science Archive Facility are generally encouraged to provide feedback regarding the data quality of the downloaded data products by contacting the ESO Archive Science Group using the online form https://support.eso.org/, topic: Phase 3.
As Phase 3 data provider, despite exercising due diligence during data preparation and submission, you may notice only after publication through the ESO science archive, that some previously provided Phase 3 data actually do not fulfill the criteria for being released as science data products and that these data should be flagged accordingly in the user interface to notify archive users when selecting data.
In such case please go to the RM and click “REPORT DATA PROBLEM” to specify the collection containing the data in question, the kind of problem, and the list of files being affected (referenced by their ORIGFILE name, not their ARCFILE name).
Please choose the best matching description for the kind of problem from the available list, or, if the problem type is not listed, enter your own description.
Default list of data problems:
· Data quality: data corrupted
· Data quality: high background
· Data quality: highly variable background
· Data quality: telescope tracking error
· Data quality: below quality cuts
· Data quality: astrom. calibration failure
· Data quality: flux calibration failure
· Data quality: wavelength calibration failure
· Data quality: other calibration failure
· Data content: technical/calibration data, not science
· Data content: does not belong to program
· Data content: wrong pointing
· Data content: spurious duplication
· Header keywords: RA/DEC incorrect
· Header keywords: non-compliant
Data problems are reported per “Phase 3 data set”, i.e. the filename of the science file (having PRODCATG=SCIENCE.*) must be listed and the ancillary (‘associated’) files, if any, will be automatically included.
The ESO Archive Science Group is responsible for handling your report. Having validated the input, the respective data sets will be marked as “deprecated” in the ESO archive, which entails that these data are not available by default but only on specific request.
Who takes the action of reporting Phase 3 data problems? The P.I., equivalently, the owner of the Phase 3 collection, or the Phase 3 survey manager, are entitled to report data problems.
If you provide a new corrected version of the problematic data set, please refer to the use case Updating previously released Phase 3 data.