p2 API Python Tutorial

Introduction

The Phase 2 Application Programming Interface (API) uses the lightweight JSON standard as data format to exchange information between your application and the phase 2 server at ESO. In order to program more conveniently against the API, it is helpful to create small, language-specific bindings to the API. We provide a simple example binding for Python, i.e. p2api.py. In the following sections, we discuss basic OB programming using this binding with hands-on coding examples. Note that the data exchanged in the various API calls is often instrument-specific. In order to fully understand the data required and returned by the various API calls and their detailed behaviour, please consult the Phase 2 API Specification.

Getting started

Although the API binding should also work in Python 2.7, we use Python 3 for this tutorial. Please ensure that it is installed and the executable python3 is on your command line search path. Install or update the p2api Python binding and pretty printing support in your local account using command:

python3 -m pip install --upgrade --user p2api pprint

This will also install p2api's dependencies. The p2api package is hosted at the Python Package Index, where any new version will be published. For this tutorial we will use the interactive Python3 interpreter. Please start it and type the listed commands as we go through this tutorial. Fear not, you are safe to play in the demo environment.

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 
Type "help", "copyright", "credits" or "license" for more information. 
>>>

You can also download the example source code hello_world.py.

Pretty Printing Support

In order to print JSON data returned from the various API calls in a somewhat more readable way, we first add pretty printing support to our code. Type

import pprint
import p2api
p = pprint.PrettyPrinter(indent=4)

To get an overview of the available API calls, you can now type

help(p2api)

Log in

Now let's get access to the API and log in using the 'demo' environment. Other supported environments are the 'production' environment for Paranal observation preparation and the 'production_lasilla' environment for La Silla observation preparation. Run the following code, you should see not output.

api = p2api.ApiConnection('demo', '52052', 'tutorial')

Create a Folder

Let's create a dedicated Folder in which we work. A Folder has no impact on observation execution. It is just a means to structure your content. All kinds of containers in which Observing Blocks can be created have a containerId, i.e. Runs, Folders/Subfolders, TimeLinks, Concatenations, Groups. Point your browser to the Phase 2 Demo, you will be automatically logged in with the tutorial account. Click on the UVES observing run. Details of the run will be displayed, including a "ContainerID for obximport" which should be 1538878. Let's call this ID the runContainerId. Type and execute the following

runContainerId = 1538878
folder, folderVersion = api.createFolder(runContainerId, 'P2API Tutorial Folder')
folderId = folder['containerId']
p.pprint(folder)

The output should be similar to

{   'containerId': 1859476,
    'itemCount': 0,
    'itemType': 'Folder',
    'name': 'P2API Tutorial Folder',
    'parentContainerId': 1538878,
    'runId': 6092526}

Now go back to your web browser, expand the run by clicking on its plus icon. You should now see the created Folder.

Create an OB

Now let's create a new OB inside our Folder. Run the code below.

ob, obVersion = api.createOB(folderId, 'My First OB')
obId = ob['obId']
p.pprint(ob)

The output should be similar to

{   'constraints': {   
         'airmass': 2.8,
         'fli': 1.0,
         'moonDistance': 30,
         'name': 'No Name',
         'seeing': 2.0,
         'skyTransparency': 'Photometric',
         'twilight': 0},
    'executionTime': 0,
    'exposureTime': 0,
    'instrument': 'UVES',
    'ipVersion': 101.05,
    'itemType': 'OB',
    'migrate': False,
    'name': 'My First OB',
    'obId': 1859478,
    'obStatus': 'P',
    'obsDescription': {   
        'instrumentComments': '',
       'name': 'No name',
       'userComments': ''},
    'parentContainerId': 1859476,
    'runId': 6092526,
    'target': {   
        'dec': '00:00:00.000',
        'differentialDec': 0.0,
        'differentialRa': 0.0,
        'epoch': 2000.0,
        'equinox': 'J2000',
        'name': 'No name',
        'properMotionDec': 0.0,
        'properMotionRa': 0.0,
        'ra': '00:00:00.000'},
    'userPriority': 1}

Note that the second parameter returned from each API call, such as api.createOB(folderId, 'My First OB') is almost always a version of the created, modified or retrieved resource (OB, Template, Folder, Readme, ...). A version parameter is always required whenever such resource is modified with an API call, so that the server can protect itself against uncontrolled concurrent modifications. The detailed printout of the OB allows you to understand the properties and sub-properties an OB has, so that you can modify them.

Now go back to your browser, close and re-expand the Folder by clicking on its Folder icon. The new OB should now be visible. You can click on it to show its details.

Edit OB

The following code modifies some OB-level properties and then saves the OB. Note that depending on the instrument, observing constraints may or may not be available. Sending non-existent constraints results in an exception being thrown. When updating the OB, its current obVersion must be passed as parameter to the API call. Please check in the browser that the changes have arrived by refreshing the page showing OB details.

# edit OB
ob['userPriority'] = 9
ob['target']['name']                 = 'M 32 -- Interacting Galaxies'
ob['target']['ra']                   = '00:42:41.825'
ob['target']['dec']                  = '-60:51:54.610'
ob['target']['properMotionRa']       = 1
ob['target']['properMotionDec']      = 2
ob['constraints']['name']            = 'My hardest constraints ever'
ob['constraints']['airmass']         = 1.3
ob['constraints']['skyTransparency'] = 'Variable, thin cirrus'
ob['constraints']['fli']             = 0.1
ob['constraints']['seeing']          = 2.0
ob, obVersion = api.saveOB(ob, obVersion)

Attach Acquisition Template

Now attach an Acquisition Template, identified by its name.

acqTpl, acqTplVersion = api.createTemplate(obId, 'UVES_blue_acq_slit')
p.pprint(acqTpl)

With an output similar to

{   'parameters': [   {   'name': 'TEL.TARG.OFFSETALPHA',
                        'type': 'number',
                        'value': 0.0},
                    {   'name': 'TEL.TARG.OFFSETDELTA',
                        'type': 'number',
                        'value': 0.0},
                    {   'name': 'TEL.AG.GUIDESTAR',
                        'type': 'keyword',
                        'value': 'CATALOGUE'},
                    {   'name': 'TEL.GS1.ALPHA',
                        'type': 'coord',
                        'value': '00:00:00.000'},
                    {   'name': 'TEL.GS1.DELTA',
                        'type': 'coord',
                        'value': '00:00:00.000'},
                    {   'name': 'INS.DROT.MODE',
                        'type': 'keyword',
                        'value': 'ELEV'},
                    {   'name': 'INS.DROT.POSANG',
                        'type': 'number',
                        'value': 0.0},
                    {   'name': 'INS.FILT1.NAME',
                        'type': 'keyword',
                        'value': 'FREE'},
                    {   'name': 'INS.DPOL.MODE',
                        'type': 'keyword',
                        'value': 'OFF'},
                    {   'name': 'INS.ADC.MODE',
                        'type': 'keyword',
                        'value': 'OFF'}],
    'templateId': 1062980,
    'templateName': 'UVES_blue_acq_slit',
    'type': 'acquisition'}

Please refresh your OB in the browser to verify that you template has arrived.

Edit Acquisition Template Parameters

You can set any template parameter which are not of type file or paramfile as follows. Remember, we must provide the acqTplVersion when updating the existing template.

# edit Acquisition Template
acqTpl, acqTplVersion  = api.setTemplateParams(obId, acqTpl, {
    'TEL.GS1.ALPHA': '11:22:33.000',
    'INS.DROT.MODE': 'SKY',
    'INS.ADC.MODE': 'AUTO'
}, acqTplVersion)

Please refresh your OB again in the browser to verify that the values changed in the acquisition template.

Attach Science Template

Now attach a science template again identified by its name.

scTpl, scTplVersion = api.createTemplate(obId, 'UVES_blue_obs_exp')
p.pprint(scTpl)

With an output similar to

{   'parameters': [   {   'name': 'DET1.READ.SPEED',
                        'type': 'keyword',
                        'value': '225kHz,1x1,low'},
                    {   'name': 'DET1.WIN1.UIT1',
                        'type': 'number',
                        'value': 10.0},
                    {'name': 'SEQ.NEXPO', 'type': 'integer', 'value': 1},
                    {   'name': 'SEQ.SOURCE',
                        'type': 'keyword',
                        'value': 'POINT'},
                    {'name': 'SEQ.NOFF', 'type': 'integer', 'value': 1},
                    {   'name': 'TEL.TARG.OFFSETX',
                        'type': 'numlist',
                        'value': [0.0]},
                    {   'name': 'TEL.TARG.OFFSETY',
                        'type': 'numlist',
                        'value': [0.0]},
                    {   'name': 'INS.BLUEEXP.MODE',
                        'type': 'keyword',
                        'value': '346'},
                    {   'name': 'INS.SLIT2.WID',
                        'type': 'number',
                        'value': 0.6}],
    'templateId': 1062981,
    'templateName': 'UVES_blue_obs_exp',
    'type': 'science'}

Edit Science Template Parameters

Edit some parameters in the science template.

scTpl, scTplVersion = api.setTemplateParams(obId, scTpl, {
    'DET1.READ.SPEED' : '50kHz,2x2,high',
    'INS.SLIT2.WID'   : 0.15
}, scTplVersion)

Define Absolute Time Constraints

Now let's define a number of absolute time constraints on our OB. In order to provide a version atcVersion to the API call, we must first retrieve the current time constraints (which are empty)

absTCs, atcVersion = api.getAbsoluteTimeConstraints(obId)
absTCs, atcVersion = api.saveAbsoluteTimeConstraints(obId,[
    {
        'from': '2020-09-01T00:00',
        'to': '2020-09-30T23:59'
    },
    {
        'from': '2020-11-01T00:00',
        'to': '2020-11-30T23:59'
    }
], atcVersion)
p.pprint(absTCs)

With output

[   {'from': '2020-09-01T00:00', 'to': '2020-09-30T23:59'},
    {'from': '2020-11-01T00:00', 'to': '2020-11-30T23:59'}]

Define Sidereal Time Constraints

Sidereal time constraints work in exactly the same way as absolute ones, only their format is limited to HH:MM.

sidTCs, stcVersion = api.getSiderealTimeConstraints(obId)
sidTCs, stcVersion = api.saveSiderealTimeConstraints(obId,[
    {
        'from': '00:00',
        'to': '01:00'
    },
    {
        'from': '03:00',
        'to': '05:00'
    }
], stcVersion)
p.pprint(sidTCs)

Attach and delete an Ephemeris File

ESO-compliant ephemeris files for moving targets can be produced at this website. Assuming you have produced a valid file ephem.txt, attach this as always by first getting the current version and then saving the new file

# add Ephemeris text file
_, ephVersion = api.getEphemerisFile(obId, 'delete_me.txt')
_, ephVersion = api.saveEphemerisFile(obId, 'ephem.txt', ephVersion)

# delete Ephemeris text file again
api.deleteEphemerisFile(obId, ephVersion)

Attach, delete and download Finding Charts

You can attach up to 5 finding charts in JPEG format (each < 1 MByte) as follows, no version handling is required in this case. Let's attach two finding charts and then retrieve the list of all finding chart filenames attached (not the actual binary data). Finding charts of an OB are indexed with a 1-based index. Finally, we delete again the second one. Obviously you need to have some JPEGS in your local directory for this to work.

# add 2 finding charts
api.addFindingChart(obId, 'fc1.jpg')
api.addFindingChart(obId, 'fc2.jpg')
fcNames, _ = api.getFindingChartNames(obId)
p.pprint(fcNames)

# delete the second finding chart
api.deleteFindingChart(obId, 2)
fcNames, _ = api.getFindingChartNames(obId)
p.pprint(fcNames)

Refresh your OB in the browser and verify that you can see the finding charts. You can download a finding chart again by index. Note that it is considered insecure to let the server decide the filename. Instead, you have to specify it explicity. To download into file olieph.txt execute

api.getFindingChart(obId, 1, 'olieph.txt')

Verify OB to status (D) or (+)

When we are done editing the OB, we should verify it. Note that we can simply verify the OB without status change as a preliminary step, or have the OB change status from (P)artially Defined to (D)efined for service mode OBs or to (+)Executable for visitor mode respectively. This is controlled by the boolean "submit" flag. In order to submit an OB to ESO for observation, it must have the respective state.

# verify OB
response, _ = api.verifyOB(obId, True)
if response['observable']:
    print('*** Congratulations. Your OB' , obId, ob['name'], 'is observable!')
else:
    print('OB', obId, 'is >>not observable<<. See messages below.')
print(' ', '\n  '.join(response['messages']))

Which, if successful prints something similar to

OB VALIDATION: SUMMARY
+===================================+==========+========+========+
|Object name                        | Warnings | Errors | Status |
+===================================+==========+========+========+
|My First OB                        |        5 |      0 |     OK |
+-----------------------------------+----------+--------+--------+

OB VALIDATION: COMPLETE REPORT
--===
--=== Checks for OB 1859478 (My First OB) ===---
--===
   --> 5 WARNINGS: please check.
warning: This is not a ToO run, please check the target coordinates.
warning: The target seems to be missing from the target list in the proposal.
warning: If an absolute time window has been specified only to ensure compliance with the
    airmass constraint, then please remove the time window as it is unnecessary.
warning: UVES_blue_acq_slit (#0): When providing the coordinates of the telescope guide star,
the selection parameter must be set to SETUPFILE.
warning: UVES_blue_obs_exp (#1): undersampled resolution elements (readout mode= 50kHz,2x2,high and slit width= 0.15)

Now fetch the OB again to confirm its changed status

ob, obVersion = api.getOB(obId)
print('*** Status of verified OB', obId, 'is now', ob['obStatus'])

Duplicate OB

Now let's duplicate our OB so we have a copy in within the same Folder (you can also duplicate to a different Folder or Scheduling Container). Also, let us verify the copy to status (D) as well so that we have two observable OB. Have a look into your browser again!

ob2, ob2Version = api.duplicateOB(obId)
obId2 = ob2['obId']
response, _ = api.verifyOB(ob2['obId'], True)
if response['observable']:
    print('*** Congratulations. Your OB' , ob2['obId'], ob2['name'], 'is observable! ***')

# fetch second OB again to confirm its status change
ob2, ob2Version = api.getOB(obId2)
print('*** Status of verified OB', ob2['obId'], 'is now', ob2['obStatus'])

Populate the Visitor Execution Sequence (VES)

Note: The following only works if your run is a visitor mode run which are not available on the tutorial account in demo. So you cannot use the OBs that we prepared so far. Every user has a single dedicated Visitor Execution Sequence for each instrument. We can add observable visitor mode OBs in status (+) to the visitor execution sequence. If you have visitor OBs in status (+) available, this would work as follows

executionSequence, esVersion = api.getExecutionSequence('UVES')
executionSequence, esVersion = api.saveExecutionSequence('UVES',
    [
        { 'obId': obId  },
        { 'obId': obId2 }
    ],
esVersion)
print('*** OBs in UVES Execution Sequence', ', '.join(str(e['obId']) for e in executionSequence))

Creating Scheduling Containers

Assuming we are working with a service mode run, we can define Scheduling Containers, i.e. Concatenations, Groups and TimeLinks. Let's create one of each inside the Folder that we created earlier.

# create Scheduling Containers
grp, grpVersion = api.createGroup(folderId, 'My First Group')
print('*** Created Group with containerId', grp['containerId'])

con, conVersion = api.createConcatenation(folderId, 'My First Concatenation')
print('*** Created Concatenation with containerId', con['containerId'])

tl, tlVersion = api.createTimeLink(folderId, 'My First TimeLink')
print('*** Created TimeLink with containerId', tl['containerId'])

Refresh your browser to see those three empty Scheduling Containers in your Folder. Let us now populate those Scheduling Containers with some OBs. We could of course create new OBs as before, but for simplicity let us simply duplicate the OBs from our Folder into the three Scheduling Containers. Note that as opposed to the previous duplication, we duplicate OBs to a different destination container.

grpOb1, grpOb1Version = api.duplicateOB(obId, grp['containerId'])
grpOb2, grpOb2Version = api.duplicateOB(obId2, grp['containerId'])
conOb1, conOb1Version = api.duplicateOB(obId, con['containerId'])
conOb2, conOb2Version = api.duplicateOB(obId2, con['containerId'])
tlOb1, tlOb1Version   = api.duplicateOB(obId, tl['containerId'])
tlOb2, tlOb2Version   = api.duplicateOB(obId2, tl['containerId'])

No we have 3 Scheduling Containers each with 2 OBs inside. Note that the status of these duplicated OBs is back to (P)artially Defined, we need to verify them again to move them into status (D)efined.

Changing Group Contribution of a Group OB

Each OB in a Group has an individual Group Contribution. We can access such scheduling information as follows.

grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion = api.getOBSchedulingInfo(grpOb1['obId'])
p.pprint(grpOb1SchedulingInfo)

with output

{   'absoluteTimeConstraints': 2,
    'acquisitionTemplate': 'UVES_blue_acq_slit',
    'complete': True,
    'constraintSetName': 'My hardest constraints ever',
    'ephemerisFile': '',
    'executionTime': 0,
    'findingCharts': ['fc1.jpeg'],
    'groupContribution': 10,
    'itemType': 'OB',
    'name': 'My First OB_2',
    'obId': 1859492,
    'obStatus': 'P',
    'obsDescriptionName': 'No name',
    'targetName': 'No name',
    'userPriority': 1}

No let's change the group contribution and save.

grpOb1SchedulingInfo['groupContribution'] = 42
grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion = \
    api.saveOBSchedulingInfo(grpOb1['obId'], grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion)

In your web browser, you would have to navigate to the Schedule view to see this value.

Changing Earliest/Latest After Previous of a TimeLink OB

All but the first OB of a TimeLink have the two relative time constraints with respect to the previous OB referred to as earliest_after_previous and latest_after_previous.

tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion = api.getOBSchedulingInfo(tlOb2['obId'])
p.pprint(tlOb2SchedulingInfo)

with output

{   'absoluteTimeConstraints': 0,
    'acquisitionTemplate': 'UVES_blue_acq_slit',
    'afterPrevious': {'earliest': 'P0DT00H00M', 'latest': 'P0DT00H00M'},
    'complete': True,
    'constraintSetName': 'My hardest constraints ever',
    'ephemerisFile': '',
    'executionTime': 0,
    'findingCharts': ['fc1.jpeg'],
    'itemType': 'OB',
    'name': 'My First OB_3',
    'obId': 1859507,
    'obStatus': 'P',
    'obsDescriptionName': 'No name',
    'targetName': 'No name',
    'userPriority': 1}

Let us modify this interval to [10 - 42 days].

tlOb2SchedulingInfo['afterPrevious']['earliest'] = 'P10DT00H00M';
tlOb2SchedulingInfo['afterPrevious']['latest'] = 'P42DT00H00M';
tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion = \
    api.saveOBSchedulingInfo(tlOb2['obId'], tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion)

Finalizing for Submission

In order to finalize our work, let us verify all six Scheduling Container OBs to status (D)

response, _ = api.verifyOB(grpOb1['obId'], True) 
response, _ = api.verifyOB(grpOb2['obId'], True) 
response, _ = api.verifyOB(conOb1['obId'], True) 
response, _ = api.verifyOB(conOb2['obId'], True) 
response, _ = api.verifyOB(tlOb1['obId'], True) 
response, _ = api.verifyOB(tlOb2['obId'], True)

Note that once the OBs are in status (D), they can no longer be edited. The status (D) is supposed to help you in marking which OBs are completed and ready for submission to ESO. If you wish to go back to editing those OBs, you can do so by changing their status back to (P) using the API api.reviseOB(obId). Prior to submission to ESO, we also have to verify each Scheduling Container to status (D) to complete our work.

response, _ = api.verifyContainer(grp['containerId'], True)
print('Group', grp['containerId'], 'observable?', response['observable'])
response, _ = api.verifyContainer(con['containerId'], True)
print('Concatenation', con['containerId'], 'observable?', response['observable'])
response, _ = api.verifyContainer(tl['containerId'], True)
print('TimeLink', tl['containerId'], 'observable?', response['observable'])

If you refresh your browser you should now see that all Scheduling Containers have also changed status to (D)efined. Once the Scheduling Containers are in status (D), they can no longer be edited, hence you can no longer add new OBs, change the order of OBs, etc. If you wish to go back to editing those Scheduling Containers, you can do so by changing their status back to (P) using the API api.reviseContainer(containerId). Note that whenever you revise an OB in a Scheduling Container back to status (P), as a side effect, also the Scheduling Container status will change back to (P).

Submission to ESO

You can now submit your run to ESO. For this, we have to read the runId from the browser URL, which is the trailing number in https://www.eso.org/p2demo/home/run/6092526

runId = 6092526
response, _ = api.submitRun(runId)
p.pprint(response)

This call will return a summary of the submitted observations. Note that status of all submitted Scheduling Containers and OBs will change to (R)eview so that they become read-only.

[   {   'containerId': 1859193,
        'itemType': 'Folder',
        'name': 'move here',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'blah',
                            'obId': 1859218,
                            'obStatus': 'D'}]},
    {   'containerId': 1859207,
        'containerStatus': 'P',
        'itemType': 'Concatenation',
        'name': 'New Concatenation',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'hohoho_2',
                            'obId': 1859210,
                            'obStatus': 'D'},
                        {   'itemType': 'OB',
                            'name': 'hohoho',
                            'obId': 1858930,
                            'obStatus': 'D'}]},
    {   'containerId': 1859476,
        'itemType': 'Folder',
        'name': 'P2API Tutorial Folder',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'My First OB_2',
                            'obId': 1859483,
                            'obStatus': 'D'},
                        {   'itemType': 'OB',
                            'name': 'My First OB',
                            'obId': 1859478,
                            'obStatus': 'D'}]},
    {   'containerId': 1859486,
        'containerStatus': 'D',
        'itemType': 'Group',
        'name': 'My First Group',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'My First OB_2',
                            'obId': 1859492,
                            'obStatus': 'D'},
                        {   'itemType': 'OB',
                            'name': 'My First OB_3',
                            'obId': 1859495,
                            'obStatus': 'D'}]},
    {   'containerId': 1859488,
        'containerStatus': 'D',
        'itemType': 'Concatenation',
        'name': 'My First Concatenation',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'My First OB_2',
                            'obId': 1859498,
                            'obStatus': 'D'},
                        {   'itemType': 'OB',
                            'name': 'My First OB_3',
                            'obId': 1859501,
                            'obStatus': 'D'}]},
    {   'containerId': 1859490,
        'containerStatus': 'D',
        'itemType': 'TimeLink',
        'name': 'My First TimeLink',
        'obsBlocks': [   {   'itemType': 'OB',
                            'name': 'My First OB_2',
                            'obId': 1859504,
                            'obStatus': 'D'},
                        {   'itemType': 'OB',
                            'name': 'My First OB_3',
                            'obId': 1859507,
                            'obStatus': 'D'}]}]

Well done! You made it to the end of this tutorial. If you have any questions or suggestions for improvements, please don't hesitate to get in touch at usd_help@eso.org.