Phase 2 preparation process with p2
On this page you will be guided trough the principles of the preparation of the ESO Phase 2 material with p2.
Users already familiar with the system should familiarize themself with the newest p2 features and functionalities.
Phase 2 preparation: start here
For a successful Service Mode run it is absolutely essential that all users read the following documentation.
ESO Phase 2 observing material
- The VLT Data Flow System (DFS) is a set of protocols, interfaces and tools developed by ESO for linking together the telescope and instruments with simple observation description tools, on-line data processing capabilities, and automated data archiving. The VLT DFS is implemented for all ESO Telescopes in Paranal and La Silla observatory. This includes the survey telescopes VISTA and VST. The basic unit for the observations, central to the VLT Data Flow System is the Observation Block (OB). Astronomers specify their programmes in terms of OBs, which contain all the information necessary to obtain a ”single” observation. These include the target position, the instrument and exposure setup parameters, special scheduling requirements, the time constraints, the finding charts, and possibly also ephemerides lists. Such a single observation can contain in principle one or multiple exposures, or even multiple instrument configurations with multiple exposures, although multiple instrument configurations are usually discouraged, as they may complicate the calibrations. The product of the execution of an OB is normally the smallest dataset consistent with the scientific and technical goals of a given observing programme.
- Science Observation Blocks (OBs) can be used to obtain scientific observations of an astronomical target, as well as reference data that require the observation of a specific target (such as photometric standards). As a minimum they include an acquisition and a science template and target information.
- Calibration Blocks (CBs) are used to acquire reference data such as lamp flat fields, biases, comparison lamps, etc. that do not require the observation of an astronomical target. Calibration blocks do not have acquisition template.
- an acquisition template, describing how the target acquisition is to be performed: it includes the information on the basic instrumental setup necessary for the acquisition; for example, which filter or slit is to be used in the (acquisition) observation, which exposure time the acquisition image should have, which position angle on the sky the slit should have, etc.
- One or more science templates, describing the instrument setup and the exposure parameters: for example, which mask should be loaded in a MOS observation, which grism should be used, which grating central wavelength should be set, which jitter pattern should be obtained, what integration time should each exposure have, etc.
- In some cases, the OB may end with an attached calibration template, such as for instance an arc lamp exposure for precise wavelength calibration to be obtained right after the last scientific exposure with the instrument in the same configuration. The set of acquisition, science, and possible attached calibration templates in a given OB compose the Observation Description.
In addition to the templates, a science OB contains other important, mostly instrument-independent information:
Target information, including coordinates, proper motion, and, for Solar System targets, differential motion
The Constraint Set, to be used in Service Mode only, specifying under which external conditions (airmass, seeing, transparency, lunar illumination...) the OB can be executed.
The Time Intervals information, also to be used in Service Mode only, specifying possible absolute time intervals (UT date and time) within which the OB need to be started - note that the OB may end after the end of its specified time interval.
The Sidereal Time Intervals are used for Service Mode to specify intervals in local sidereal time within which the OB need to be executed - this means that the Sidereal Time interval must encompass both start and end of an OB execution.
The Calibration Requirements information, a free text field where comments can be given to the Service Mode observer or where the Visitor Mode observer can include reminders on the calibrations needed.
In the Schedule tab of p2, it is possible to set User Priority for individual OBs (or for scheduling containers). User priority is a number ranging from 1 to 10, where 1 corresponds to the highest priority. It sets the relative priority for execution for OBs within the single observing run. Note however, that this priority can be overridden during Service Mode operation if for example OB with a higher priority cannot be executed (e.g. target not visible) while a lower priority one can (target observable within constraints and no other higher ranked run has observable OBs).
Some instruments (e.g. VIRCAM, FORS2, FLAMES, KMOS,...) require the insertion in the templates of parameter files (PAF) generated by instrument-specific observation preparation software (SADT, FIMS, FPOSS, KARMA,...).
Optionally, one or more Finding charts, especially for use in Service Mode (exception: VST and VISTA survey observations do not need finding charts).
Also optionally, an Ephemeris file for moving targets providing their coordinates at different dates.Obviously, none of these items are present in Calibration Blocks (CBs)
A group container is a simple collection of OBs the execution order of which may be influenced by the user through specification of group contribution. Once you place the OBs in a group and one of them is executed then the remaining OBs belonging to the group will be executed preferentially before any other part of the programme.
The use of group containers may be envisaged by the following examples:
- The observing programme contains several targets located very close to each other on the sky, and for each target more than one OB is needed. If possible, you want that all observations of a single target are completed before a new target observation is started. This can be achieved by creating a group containing all the OBs for a given target.
- The observing programme contains a target observed repeatedly through many different setups (i.e. filters/grisms). If possible, you want that all observations for a given setup are executed before starting a new one. This can be achieved by creating a group containing all the OBs for a given setup.
Groups have an overall priority, which is similar to an OB user priority in the case of independent OBs. In addition, to every OB belonging to a group is given a group contribution value, indicating how much the overall group score is increased by successfully executing that given OB. Once the observations of a given group have started, the group score is increased, thereby increasing the priority of all the OBs in that group, with respect to other independent OBs, or with respect to other scheduling containers that may have the same OB user priority or container user priority. The group priority, as that for independent OB, is a number ranging from 1 to 10, where 1 corresponds to the highest priority.
The time-link container defines a sequence of OBs that has to be executed in a given order, respecting the relative time intervals specified for each linked OBs.
There are two kinds of time-link container:
- OPEN time-link: only the lower limit - earliest after previous - of the separation interval is defined
- CLOSED time-link: both lower - earliest after previous - and upper limit - latest after previous - are defined
Because there is a number of rules applied to the OBs belonging to a time-link container we strongly encourage to consult the Service Mode Rules and Recommendations for OBs page section related to Time-linking of OBs before making use of this type of container.
Time-link containers have an overall priority. The priority is associated only to the container and not to the OBs within that given container. This implies that all the OBs of a given time-link share a unique priority although the sequence of their execution is given by the absolute and relative time intervals.
Please be also aware that different policies applies to the execution of time critical observations, depending on the fact that an absolute or a relative time-window for the observation is need. Please find here the details of the time-critical OB execution policy.
A concatenation container consists of two or more OBs that must be executed back-to-back without breaks. The sequence of the execution of OBs in a concatenation cannot be specified.
If the execution of one OB within a concatenation fails, then the whole concatenation fails too and it must be repeated, irrespective whether any other OBs of the concatenation has been successfully executed or not.
Concatenation containers have an overall priority. The priority is associated only to the container and not to the OBs within that given container. This implies that all the OBs of a given concatenation share a unique priority and there is no possibility to choose the sequence of execution of the OBs.
The typical use of concatenation may be envisaged in the case of observations of science targets which should be immediately followed by the observations of a calibrator source (i.e. NACO: science target and PSF calibrator; CRIRES: science target and telluric standard; HAWK-I: science target and photometric standard star, etc.).
Scheduling containers may not only contain OBs, but also other scheduling (nested) containers, thus realizing a hierarchical structure of containers and OBs. Currently supported nested containers are time links of concatenations and groups of concatenations. In this respect the nested concatenations within Group or Time-link could be seen as "super-OBs" that are in Group/Time-link and the rules described above apply. As for all Scheduling Containers, users can create and edit Nested Containers inside p2.
Time Links of Concatenations
The following rules apply to the OBs/nested concatenations in a time link:
- The first OB or nested concatenation can be executed at any suitable time
- After execution of the first OB or nested concatenation, an absolute earliest/latest after previous time constraint is imposed on the second item of the time link, and so forth ...
- If an OB or nested concatenation in a time link fails, execution continues
- Only the first item of a time link can have absolute time constraints. If it’s a nested concatenation, it means any OB in the concatenation may have an absolute time interval – if more than one does, then the absolute time constraints must be the same (checked during the Certification/Verification of the phase 2 material in p2).
Groups of Concatenations
As in the case of OBs, every nested concatenation belonging to a group is given a group contribution value, indicating how much the group’s overall group score is increased by successfully executing that OB/nested concatenation.
- Groups allow the astronomer to express that OBs/nested concatenations within a group should be executed before another group of OBs/concatentations defined within the same run starts being executed. This is a preference not a mandatory sequence for execution. For a more detailed description please see above Scheduling Containers: Groups.
- If an OB/nested concatenation in a group fails, execution continues
Filtering and ranking of OBs
Please note that in Service Mode all OBs, either loose or included in a Scheduling Container, once approved are added to the instrument's queue. The observing and short term scheduling tool includes a ranking algorithm that prioritises the OBs in terms of scientific priority, considering also the observing strategy and observing constraints likelihood of realisation. The top ranked observations can thus be chosen for execution at a given time of the night. Please check here for details related to the ranking algorithm.
In practical terms, when planning the observing strategy of a given run, users might consider the following to fine tune the prioritization within the run:
- The User Priority and Group Contribution available on the Scheduled tab of the p2UI should be used to regulate the internal priority of OBs within a given run.
- When an OB belonging to a Group Container is observed, all other OBs belonging to the same Group will have higher priority with respect to loose OBs and/or OBs in different groups. This is due to the fact that the algorithm privileges finishing a Group before starting a new one.
- In the presence of absolute or relative time constraints the algorithm applies a slight pull to OB, raising its priority, when the observations are close to the end of the assigned time window. Please note that use of the time constraints in OB preparation is allowed only when the science case justifies it. Different policies apply to time critical observations, depending if an absolute or a relative time-window for the observation is need. Please find here the details of the time-critical OB execution policy.
p2 for Phase 2 preparation
p2 is a user interface (UI) that allows ESO users to create, modify, or delete observation blocks (OBs), containers and an accompanying Readme file that define an observing run. Also, p2 allows easy creation of finding charts and the interactive preparation of OB parameters with the ObsPrep tool.
The submission of Phase 2 Waiver Requests and Programme Change Requests is fully handled within the p2 UI.
It is important to note that any new item created with p2 by the user (OB, scheduling container) is immediately recorded in the ESO operations database and is replicated to the observatory in near-real time.
In the following we describe the features of the Phase 2 process.
- Check, Certify and Revise (for OBs, scheduling containers, folders)
Users can at any time verify any OB/scheduling container to ensure that they comply with ESO general and instrument specific observing requirements and they are hence executable at the telescope. All the verifications are done via the External Verification Modules (EVMs), which perform consistency checks across the templates in an OB or Container. Moreover the EVM ensures the compliance of Service Mode OBs with rules that Service Mode users need to follow so that their observations can be scheduled and executed. The EVMs can also be used to verify the contents of OBs prepared for Visitor Mode observations.
Below follows a short description of the web interface's functions involved in this process. The user should get familiar with the meaning of the status icons and flags of OB/scheduling containers.
Users can run the OB verification anytime during the preparation of their OBs. To do it, simply highlight the OB or group of OBs, or scheduling container to be verified, and select the Check button on the UI. A verification report will then appear in a pop-up window. In addition to information messages on the checks performed on the OBs, the verification report also issues error and warning messages mainly related to Service Mode policies.
The use of the "Check" button leaves the OBs and scheduling containers in a fully editable status allowing the user to revise their content.
It is important to consider that the check of OBs is not a submission of Phase 2 material. The latter is in fact replaced by a notification. Please see explanations below.
The "Certify" button performs the same verification checks described in the case of the "Check" button, including the display of a verification report.
The use of the "Certify" button on OBs and/or scheduling containers changes their status from fully editable into partially-editable status. The OBs will appear as greyed out and a small check icon appears next to them. After certification the user will only be allowed to change the OB priority, or scheduling container priority and the specific contribution of individual OBs in a group container. These properties are accessible from the "Schedule" tab.
It is important to consider that the certification of OBs is not a submission of Phase 2 material. The latter is in fact replaced by a notification. Please see explanations below.
The user has the possibility at any time during the preparation of the Phase 2 observing material to revise any OBs or scheduling container that was certified to retrieve a fully editable status and apply any further change. This is done by selecting one or more OBs and/or scheduling containers and pressing "Revise".
The use of the "Revise" button on OBs and/or scheduling containers will reversed their status to fully-editable. After that, modification of the content of the OBs is allowed.
- A Notification concludes the Phase 2 preparation
Any OB and/or scheduling container created using p2 inside a given run is immediately recorded on the ESO database. Hence, it is not necessary anymore to submit the Phase 2 material. Once the Phase 2 observing material is certified (please see Certify above), a notification must be sent to inform the User Support Department (for Service Mode runs) or the staff at Paranal (for Visitor Mode runs) that there is Phase 2 material (OBs/containers and the Readme) that requires a review. This is simply done by selecting the run and clicking the "Notify ESO" button.
Hence the notification can be considered as the final step in the preparation of the Phase 2 material and triggers the following operations:
- all and only the OBs and/or scheduling containers which were certified become locked and read-only. Moreover, the Readme must be prepared at the time of the notification and will be included as part of the material ready for the review. Also the Readme will appear as read-only after that a notification is sent. In the case of a Service Mode run OBs are subject to the revision by the User Support Department. In case of a Visitor Mode run the prepared material will be fully editable. In both cases the user will be contacted with some feedback by the astronomer in charge;
- a message is displayed on the p2 interface to acknowledge the fact that ESO has been informed. The user is presented with a list of items that are ready for review;
- a notification email is sent to the user;
- a notification email is sent to the ESO User Support Department (Service Mode) or Paranal (Visitor Mode);
- Phase 2 revision for Service Mode runs
After the notification is sent (see previous section) a support astronomer (at the ESO User Support Department) starts the task of reviewing your Phase 2 material. The PI (and delegate) will be contacted by the support astronomer in charge of the run. In case that any problems or suggestions for improvement are identified the OBs will be released and the user will be asked to make modifications. Such OBs are identified within the run in the p2 UI by an exclamation mark ! icon.
The user can work on released OBs following the instructions of the support astronomer. The OB will have to be certified (see description above). Once all the issues are solved and the material is fully certified the user will inform ESO by using the "Notify ESO" action.
P2 tutorials are available for a number of instruments from the p2 Tutorial menu on the left hand side.
A dedicated p2 demo environment at http://www.eso.org/p2demo can be used to make test. Please note that template OBs with different observing modes and strategies are avaiable in dedicated Instrument Tutorial folders for each instrument.
Also available are some general general tutorials on the use of scheduling containers, on the attachment of finding charts, on filling out the README, and on finalizing the overall Phase 2 submission.
The creation and submission of Phase 2 Waiver Requests and Programme Change Requests is now fully supported in the p2 user inerface. Once logged in, the user should select the Run for which they intend to submit the Waiver Request or Change Request. By clicking on the Run, the user will be able to access the Change Request menu on the right top (see snapshot above).
By selecting the Change Request, the user will be guided though the submission process.
Please see video tutorial on the p2 tutorial pages.
- The Visitor Execution Sequence (VES)
p2 provides the possibility to create a Visitor Execution Sequence (VES). This allows any observer to create via the web interface, a sequence of OBs that should be executed at the telescope. The VES is instrument and user dependent. This means that a designated user will not see a VES created by the PI and vice versa. It is important to remember that only OBs that have been certified (please see text above) can be added to the VES. The same OB can be added multiple times to the VES, as the VM OBs can be executed multiple times.
- The Target Visibility
By selecting the Target Visibility tab of a given OB, the observability of the object is displayed as altitude vs time for a particular night. On the same plot, the trajectory of the object is shown together with the altitude of the moon on the given night while labels indicate the moon angular distance. Red points indicate violation of some observing constraints indicated in the OB.
For Visitor Mode runs the same feature is available for all the OBs that are included in the Visitor Execution Sequence.
- The OB/scheduling container status Overview
At any time during a given observing period users can check the status of the observing OBs of their run. The status of an OB describes the stage of the lifetime in which it is. Please consult here the meaning of the status and the corresponding icon. The overview of the statuses, together with other information, can be found in the Overview tab of the UI.
In the example above 3 OBs in a time-link container are fully approved and hence ready to be observed (status +; eye icon). As a consequence, the status of the scheduling container within which the OBs are collected is fully approved. The OBs are read-only (greyed out). The user is allowed to change the priority of the scheduling container.
A problem was found with OB-ID=5059977 OB-NAME=My OB_1. The OB is then marked with a status "-" and flagged by an exclamation mark. The user is asked to fix the OB. As such the OB is fully editable and must be certified by the user before being considered for review. The same applies for OBs in a scheduling container.
The ObsPrep tab of p2 allows interactive planning of the observations, e.g. the fine-tuning of the science field pointing, the selection of blind offset stars, the definition of observing offsets, and the selection of AO and/or VLT guide stars. The choices made are directly reflected by an update of the respective parameters in the OB acquisition template, the target tab and science templates. Note that the target tab must be filled in and an acquisition template in the Obs Description tab has to be chosen before ObsPrep can be used.
A short video tutorial is available here.