p2 API Python Tutorial
Introduction
The Phase 2 Application Programming Interface (API) uses the lightweight JSON standard as data format to exchange information between your application and the phase 2 server at ESO. In order to program more conveniently against the API, it is helpful to create small, language-specific bindings to the API. We provide a simple example binding for Python, i.e. p2api.py. In the following sections, we discuss basic OB programming using this binding with hands-on coding examples. Note that the data exchanged in the various API calls is often instrument-specific. In order to fully understand the data required and returned by the various API calls and their detailed behaviour, please consult the Phase 2 API Specification.
Although the API binding should also work in Python 2.7, we use Python 3 for this tutorial. Please ensure that it is installed and the executable python3 is on your command line search path.
The p2api is available for macOS, recent Fedora and CentOS-7 systems as MacPorts or RPM packages from the ESO Software Repositories. Please see the instructions here to enable the relevant repositories. The package names are py<NN>-eso-p2api (where NN is one of [27,34,35,36,37,38]) for MacPorts and python2-eso-p2api (and on systems which support python3 python3-eso-p2api).
If you do not have macOS or a supported Fedora or CentOS system, or for whatever reason you prefer to install/update from "source" then the p2api Python binding is hosted at the Python Package Index, where any new version will be published. We recommend also installing pretty printing support. The following command will install p2api and pretty printing in your account for python3.
python3 -m pip install --upgrade --user p2api
This will also install p2api's dependencies. For this tutorial we will use the interactive Python3 interpreter. Please start it and type the listed commands as we go through this tutorial. Fear not, you are safe to play in the demo environment.
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
You can also download the example source code hello_world.py.
Pretty Printing Support
In order to print JSON data returned from the various API calls in a somewhat more readable way, we first add pretty printing support to our code. Type
import pprint import p2api p = pprint.PrettyPrinter(indent=4)
To get an overview of the available API calls, you can now type
help(p2api.p2api)
Log in
Now let's get access to the API and log in using the 'demo' environment. Other supported environments are the 'production' environment for Paranal observation preparation and the 'production_lasilla' environment for La Silla observation preparation. Run the following code, you should see not output.
api = p2api.ApiConnection('demo', '52052', 'tutorial')
Create a Folder
Let's create a dedicated Folder in which we work. A Folder has no impact on observation execution. It is just a means to structure your content. All kinds of containers in which Observing Blocks can be created have a containerId, i.e. Runs, Folders/Subfolders, TimeLinks, Concatenations, Groups. Point your browser to the Phase 2 Demo, you will be automatically logged in with the tutorial account. Click on the "60.A-9252(G) · UVES" observing run. Details of the run will be displayed, including a "ContainerID for obximport" which should be 1538878. Let's call this ID the runContainerId. Type and execute the following
runContainerId = 1538878 folder, folderVersion = api.createFolder(runContainerId, 'P2API Tutorial Folder') folderId = folder['containerId'] p.pprint(folder)
The output should be similar to
{ 'containerId': 1859476,
'itemCount': 0,
'itemType': 'Folder',
'name': 'P2API Tutorial Folder',
'parentContainerId': 1538878,
'runId': 6092526}
Now go back to your web browser, expand the run by clicking on its plus icon. You should now see the created Folder.
Create an OB
Now let's create a new OB inside our Folder. Run the code below.
ob, obVersion = api.createOB(folderId, 'My First OB') obId = ob['obId'] p.pprint(ob)
The output should be similar to
{ 'constraints': {
'airmass': 2.8,
'fli': 1.0,
'moonDistance': 30,
'name': 'No Name',
'seeing': 2.0,
'skyTransparency': 'Photometric',
'twilight': 0},
'executionTime': 0,
'exposureTime': 0,
'instrument': 'UVES',
'ipVersion': 101.05,
'itemType': 'OB',
'migrate': False,
'name': 'My First OB',
'obId': 1859478,
'obStatus': 'P',
'obsDescription': {
'instrumentComments': '',
'name': 'No name',
'userComments': ''},
'parentContainerId': 1859476,
'runId': 6092526,
'target': {
'dec': '00:00:00.000',
'differentialDec': 0.0,
'differentialRa': 0.0,
'epoch': 2000.0,
'equinox': 'J2000',
'name': 'No name',
'properMotionDec': 0.0,
'properMotionRa': 0.0,
'ra': '00:00:00.000'},
'userPriority': 1}
Note that the second parameter returned from each API call, such as api.createOB(folderId, 'My First OB') is almost always a version of the created, modified or retrieved resource (OB, Template, Folder, Readme, ...). A version parameter is always required whenever such resource is modified with an API call, so that the server can protect itself against uncontrolled concurrent modifications. The detailed printout of the OB allows you to understand the properties and sub-properties an OB has, so that you can modify them.
Now go back to your browser, close and re-expand the Folder by clicking on its Folder icon. The new OB should now be visible. You can click on it to show its details.
Edit OB
The following code modifies some OB-level properties and then saves the OB. Note that depending on the instrument, observing constraints may or may not be available. Sending non-existent constraints results in an exception being thrown. When updating the OB, its current obVersion must be passed as parameter to the API call. Please check in the browser that the changes have arrived by refreshing the page showing OB details.
# edit OB ob['userPriority'] = 9 ob['target']['name'] = 'M 32 -- Interacting Galaxies' ob['target']['ra'] = '00:42:41.825' ob['target']['dec'] = '-60:51:54.610' ob['target']['properMotionRa'] = 1 ob['target']['properMotionDec'] = 2 ob['constraints']['name'] = 'My hardest constraints ever' ob['constraints']['airmass'] = 1.3 ob['constraints']['skyTransparency'] = 'Variable, thin cirrus' ob['constraints']['fli'] = 0.1 ob['constraints']['seeing'] = 2.0 ob, obVersion = api.saveOB(ob, obVersion)
Attach Acquisition Template
Now attach an Acquisition Template, identified by its name.
acqTpl, acqTplVersion = api.createTemplate(obId, 'UVES_blue_acq_slit') p.pprint(acqTpl)
With an output similar to
{ 'parameters': [ { 'name': 'TEL.TARG.OFFSETALPHA',
'type': 'number',
'value': 0.0},
{ 'name': 'TEL.TARG.OFFSETDELTA',
'type': 'number',
'value': 0.0},
{ 'name': 'TEL.AG.GUIDESTAR',
'type': 'keyword',
'value': 'CATALOGUE'},
{ 'name': 'TEL.GS1.ALPHA',
'type': 'coord',
'value': '00:00:00.000'},
{ 'name': 'TEL.GS1.DELTA',
'type': 'coord',
'value': '00:00:00.000'},
{ 'name': 'INS.DROT.MODE',
'type': 'keyword',
'value': 'ELEV'},
{ 'name': 'INS.DROT.POSANG',
'type': 'number',
'value': 0.0},
{ 'name': 'INS.FILT1.NAME',
'type': 'keyword',
'value': 'FREE'},
{ 'name': 'INS.DPOL.MODE',
'type': 'keyword',
'value': 'OFF'},
{ 'name': 'INS.ADC.MODE',
'type': 'keyword',
'value': 'OFF'}],
'templateId': 1062980,
'templateName': 'UVES_blue_acq_slit',
'type': 'acquisition'}
Please refresh your OB in the browser to verify that you template has arrived.
Edit Acquisition Template Parameters
You can set any template parameter which are not of type file or paramfile as follows. Remember, we must provide the acqTplVersion when updating the existing template.
# edit Acquisition Template
acqTpl, acqTplVersion = api.setTemplateParams(obId, acqTpl, {
'TEL.GS1.ALPHA': '11:22:33.000',
'INS.DROT.MODE': 'SKY',
'INS.ADC.MODE': 'AUTO'
}, acqTplVersion)
Please refresh your OB again in the browser to verify that the values changed in the acquisition template.
Attach Science Template
Now attach a science template again identified by its name.
scTpl, scTplVersion = api.createTemplate(obId, 'UVES_blue_obs_exp') p.pprint(scTpl)
With an output similar to
{ 'parameters': [ { 'name': 'DET1.READ.SPEED',
'type': 'keyword',
'value': '225kHz,1x1,low'},
{ 'name': 'DET1.WIN1.UIT1',
'type': 'number',
'value': 10.0},
{'name': 'SEQ.NEXPO', 'type': 'integer', 'value': 1},
{ 'name': 'SEQ.SOURCE',
'type': 'keyword',
'value': 'POINT'},
{'name': 'SEQ.NOFF', 'type': 'integer', 'value': 1},
{ 'name': 'TEL.TARG.OFFSETX',
'type': 'numlist',
'value': [0.0]},
{ 'name': 'TEL.TARG.OFFSETY',
'type': 'numlist',
'value': [0.0]},
{ 'name': 'INS.BLUEEXP.MODE',
'type': 'keyword',
'value': '346'},
{ 'name': 'INS.SLIT2.WID',
'type': 'number',
'value': 0.6}],
'templateId': 1062981,
'templateName': 'UVES_blue_obs_exp',
'type': 'science'}
Edit Science Template Parameters
Edit some parameters in the science template.
scTpl, scTplVersion = api.setTemplateParams(obId, scTpl, {
'DET1.READ.SPEED' : '50kHz,2x2,high',
'INS.SLIT2.WID' : 0.15
}, scTplVersion)
Define Absolute Time Constraints
Now let's define a number of absolute time constraints on our OB. In order to provide a version atcVersion to the API call, we must first retrieve the current time constraints (which are empty)
absTCs, atcVersion = api.getAbsoluteTimeConstraints(obId)
absTCs, atcVersion = api.saveAbsoluteTimeConstraints(obId,[
{
'from': '2020-09-01T00:00',
'to': '2020-09-30T23:59'
},
{
'from': '2020-11-01T00:00',
'to': '2020-11-30T23:59'
}
], atcVersion)
p.pprint(absTCs)
With output
[ {'from': '2020-09-01T00:00', 'to': '2020-09-30T23:59'},
{'from': '2020-11-01T00:00', 'to': '2020-11-30T23:59'}]
Define Sidereal Time Constraints
Sidereal time constraints work in exactly the same way as absolute ones, only their format is limited to HH:MM.
sidTCs, stcVersion = api.getSiderealTimeConstraints(obId)
sidTCs, stcVersion = api.saveSiderealTimeConstraints(obId,[
{
'from': '00:00',
'to': '01:00'
},
{
'from': '03:00',
'to': '05:00'
}
], stcVersion)
p.pprint(sidTCs)
Attach and delete an Ephemeris File
ESO-compliant ephemeris files for moving targets can be produced at this website. Assuming you have produced a valid file ephem.txt, attach this as always by first getting the current version and then saving the new file
# add Ephemeris text file _, ephVersion = api.getEphemerisFile(obId, 'delete_me.txt') _, ephVersion = api.saveEphemerisFile(obId, 'ephem.txt', ephVersion) # delete Ephemeris text file again api.deleteEphemerisFile(obId, ephVersion)
Attach, delete and download Finding Charts
You can attach up to 5 finding charts in JPEG format (each < 1 MByte) as follows, no version handling is required in this case. Let's attach two finding charts and then retrieve the list of all finding chart filenames attached (not the actual binary data). Finding charts of an OB are indexed with a 1-based index. Finally, we delete again the second one. Obviously you need to have some JPEGS in your local directory for this to work.
# add 2 finding charts api.addFindingChart(obId, 'fc1.jpg') api.addFindingChart(obId, 'fc2.jpg') fcNames, _ = api.getFindingChartNames(obId) p.pprint(fcNames) # delete the second finding chart api.deleteFindingChart(obId, 2) fcNames, _ = api.getFindingChartNames(obId) p.pprint(fcNames)
Refresh your OB in the browser and verify that you can see the finding charts. You can download a finding chart again by index. Note that it is considered insecure to let the server decide the filename. Instead, you have to specify it explicity. To download into file olieph.txt execute
api.getFindingChart(obId, 1, 'olieph.txt')
Verify OB to status (D) or (+)
When we are done editing the OB, we should verify it. Note that we can simply verify the OB without status change as a preliminary step, or have the OB change status from (P)artially Defined to (D)efined for service mode OBs or to (+)Executable for visitor mode respectively. This is controlled by the boolean "submit" flag. In order to submit an OB to ESO for observation, it must have the respective state.
# verify OB
response, _ = api.verifyOB(obId, True)
if response['observable']:
print('*** Congratulations. Your OB' , obId, ob['name'], 'is observable!')
else:
print('OB', obId, 'is >>not observable<<. See messages below.')
print(' ', '\n '.join(response['messages']))
Which, if successful prints something similar to
OB VALIDATION: SUMMARY
+===================================+==========+========+========+
|Object name | Warnings | Errors | Status |
+===================================+==========+========+========+
|My First OB | 5 | 0 | OK |
+-----------------------------------+----------+--------+--------+
OB VALIDATION: COMPLETE REPORT
--===
--=== Checks for OB 1859478 (My First OB) ===---
--===
--> 5 WARNINGS: please check.
warning: This is not a ToO run, please check the target coordinates.
warning: The target seems to be missing from the target list in the proposal.
warning: If an absolute time window has been specified only to ensure compliance with the
airmass constraint, then please remove the time window as it is unnecessary.
warning: UVES_blue_acq_slit (#0): When providing the coordinates of the telescope guide star,
the selection parameter must be set to SETUPFILE.
warning: UVES_blue_obs_exp (#1): undersampled resolution elements (readout mode= 50kHz,2x2,high and slit width= 0.15)
Now fetch the OB again to confirm its changed status
ob, obVersion = api.getOB(obId)
print('*** Status of verified OB', obId, 'is now', ob['obStatus'])
Duplicate OB
Now let's duplicate our OB so we have a copy in within the same Folder (you can also duplicate to a different Folder or Scheduling Container). Also, let us verify the copy to status (D) as well so that we have two observable OB. Have a look into your browser again!
ob2, ob2Version = api.duplicateOB(obId)
obId2 = ob2['obId']
response, _ = api.verifyOB(ob2['obId'], True)
if response['observable']:
print('*** Congratulations. Your OB' , ob2['obId'], ob2['name'], 'is observable! ***')
# fetch second OB again to confirm its status change
ob2, ob2Version = api.getOB(obId2)
print('*** Status of verified OB', ob2['obId'], 'is now', ob2['obStatus'])
Populate the Visitor Execution Sequence (VES)
Note: The following only works if your run is a visitor mode run which are not available on the tutorial account in demo. So you cannot use the OBs that we prepared so far. Every user has a single dedicated Visitor Execution Sequence for each instrument. We can add observable visitor mode OBs in status (+) to the visitor execution sequence. If you have visitor OBs in status (+) available, this would work as follows
executionSequence, esVersion = api.getExecutionSequence('UVES')
executionSequence, esVersion = api.saveExecutionSequence('UVES',
[
{ 'obId': obId },
{ 'obId': obId2 }
],
esVersion)
print('*** OBs in UVES Execution Sequence', ', '.join(str(e['obId']) for e in executionSequence))
Creating Scheduling Containers
Assuming we are working with a service mode run, we can define Scheduling Containers, i.e. Concatenations, Groups and TimeLinks. Let's create one of each inside the Folder that we created earlier.
# create Scheduling Containers
grp, grpVersion = api.createGroup(folderId, 'My First Group')
print('*** Created Group with containerId', grp['containerId'])
con, conVersion = api.createConcatenation(folderId, 'My First Concatenation')
print('*** Created Concatenation with containerId', con['containerId'])
tl, tlVersion = api.createTimeLink(folderId, 'My First TimeLink')
print('*** Created TimeLink with containerId', tl['containerId'])
Refresh your browser to see those three empty Scheduling Containers in your Folder. Let us now populate those Scheduling Containers with some OBs. We could of course create new OBs as before, but for simplicity let us simply duplicate the OBs from our Folder into the three Scheduling Containers. Note that as opposed to the previous duplication, we duplicate OBs to a different destination container.
grpOb1, grpOb1Version = api.duplicateOB(obId, grp['containerId']) grpOb2, grpOb2Version = api.duplicateOB(obId2, grp['containerId']) conOb1, conOb1Version = api.duplicateOB(obId, con['containerId']) conOb2, conOb2Version = api.duplicateOB(obId2, con['containerId']) tlOb1, tlOb1Version = api.duplicateOB(obId, tl['containerId']) tlOb2, tlOb2Version = api.duplicateOB(obId2, tl['containerId'])
No we have 3 Scheduling Containers each with 2 OBs inside. Note that the status of these duplicated OBs is back to (P)artially Defined, we need to verify them again to move them into status (D)efined.
Changing Group Contribution of a Group OB
Each OB in a Group has an individual Group Contribution. We can access such scheduling information as follows.
grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion = api.getOBSchedulingInfo(grpOb1['obId']) p.pprint(grpOb1SchedulingInfo)
with output
{ 'absoluteTimeConstraints': 2,
'acquisitionTemplate': 'UVES_blue_acq_slit',
'complete': True,
'constraintSetName': 'My hardest constraints ever',
'ephemerisFile': '',
'executionTime': 0,
'findingCharts': ['fc1.jpeg'],
'groupContribution': 10,
'itemType': 'OB',
'name': 'My First OB_2',
'obId': 1859492,
'obStatus': 'P',
'obsDescriptionName': 'No name',
'targetName': 'No name',
'userPriority': 1}
No let's change the group contribution and save.
grpOb1SchedulingInfo['groupContribution'] = 42
grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion = \
api.saveOBSchedulingInfo(grpOb1['obId'], grpOb1SchedulingInfo, grpOb1SchedulingInfoVersion)
In your web browser, you would have to navigate to the Schedule view to see this value.
Changing Earliest/Latest After Previous of a TimeLink OB
All but the first OB of a TimeLink have the two relative time constraints with respect to the previous OB referred to as earliest_after_previous and latest_after_previous.
tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion = api.getOBSchedulingInfo(tlOb2['obId']) p.pprint(tlOb2SchedulingInfo)
with output
{ 'absoluteTimeConstraints': 0,
'acquisitionTemplate': 'UVES_blue_acq_slit',
'afterPrevious': {'earliest': 'P0DT00H00M', 'latest': 'P0DT00H00M'},
'complete': True,
'constraintSetName': 'My hardest constraints ever',
'ephemerisFile': '',
'executionTime': 0,
'findingCharts': ['fc1.jpeg'],
'itemType': 'OB',
'name': 'My First OB_3',
'obId': 1859507,
'obStatus': 'P',
'obsDescriptionName': 'No name',
'targetName': 'No name',
'userPriority': 1}
Let us modify this interval to [10 - 42 days].
tlOb2SchedulingInfo['afterPrevious']['earliest'] = 'P10DT00H00M';
tlOb2SchedulingInfo['afterPrevious']['latest'] = 'P42DT00H00M';
tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion = \
api.saveOBSchedulingInfo(tlOb2['obId'], tlOb2SchedulingInfo, tlOb2SchedulingInfoVersion)
Finalizing for Submission
In order to finalize our work, let us verify all six Scheduling Container OBs to status (D)
response, _ = api.verifyOB(grpOb1['obId'], True) response, _ = api.verifyOB(grpOb2['obId'], True) response, _ = api.verifyOB(conOb1['obId'], True) response, _ = api.verifyOB(conOb2['obId'], True) response, _ = api.verifyOB(tlOb1['obId'], True) response, _ = api.verifyOB(tlOb2['obId'], True)
Note that once the OBs are in status (D), they can no longer be edited. The status (D) is supposed to help you in marking which OBs are completed and ready for submission to ESO. If you wish to go back to editing those OBs, you can do so by changing their status back to (P) using the API api.reviseOB(obId). Prior to submission to ESO, we also have to verify each Scheduling Container to status (D) to complete our work.
response, _ = api.verifyContainer(grp['containerId'], True)
print('Group', grp['containerId'], 'observable?', response['observable'])
response, _ = api.verifyContainer(con['containerId'], True)
print('Concatenation', con['containerId'], 'observable?', response['observable'])
response, _ = api.verifyContainer(tl['containerId'], True)
print('TimeLink', tl['containerId'], 'observable?', response['observable'])
If you refresh your browser you should now see that all Scheduling Containers have also changed status to (D)efined. Once the Scheduling Containers are in status (D), they can no longer be edited, hence you can no longer add new OBs, change the order of OBs, etc. If you wish to go back to editing those Scheduling Containers, you can do so by changing their status back to (P) using the API api.reviseContainer(containerId). Note that whenever you revise an OB in a Scheduling Container back to status (P), as a side effect, also the Scheduling Container status will change back to (P).
Submission to ESO
You can now submit your run to ESO. For this, we have to read the runId from the browser URL, which is the trailing number in https://www.eso.org/p2demo/home/run/6092526
runId = 6092526 response, _ = api.submitRun(runId) p.pprint(response)
This call will return a summary of the submitted observations. Note that status of all submitted Scheduling Containers and OBs will change to (R)eview so that they become read-only.
[ { 'containerId': 1859193,
'itemType': 'Folder',
'name': 'move here',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'blah',
'obId': 1859218,
'obStatus': 'D'}]},
{ 'containerId': 1859207,
'containerStatus': 'P',
'itemType': 'Concatenation',
'name': 'New Concatenation',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'hohoho_2',
'obId': 1859210,
'obStatus': 'D'},
{ 'itemType': 'OB',
'name': 'hohoho',
'obId': 1858930,
'obStatus': 'D'}]},
{ 'containerId': 1859476,
'itemType': 'Folder',
'name': 'P2API Tutorial Folder',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'My First OB_2',
'obId': 1859483,
'obStatus': 'D'},
{ 'itemType': 'OB',
'name': 'My First OB',
'obId': 1859478,
'obStatus': 'D'}]},
{ 'containerId': 1859486,
'containerStatus': 'D',
'itemType': 'Group',
'name': 'My First Group',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'My First OB_2',
'obId': 1859492,
'obStatus': 'D'},
{ 'itemType': 'OB',
'name': 'My First OB_3',
'obId': 1859495,
'obStatus': 'D'}]},
{ 'containerId': 1859488,
'containerStatus': 'D',
'itemType': 'Concatenation',
'name': 'My First Concatenation',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'My First OB_2',
'obId': 1859498,
'obStatus': 'D'},
{ 'itemType': 'OB',
'name': 'My First OB_3',
'obId': 1859501,
'obStatus': 'D'}]},
{ 'containerId': 1859490,
'containerStatus': 'D',
'itemType': 'TimeLink',
'name': 'My First TimeLink',
'obsBlocks': [ { 'itemType': 'OB',
'name': 'My First OB_2',
'obId': 1859504,
'obStatus': 'D'},
{ 'itemType': 'OB',
'name': 'My First OB_3',
'obId': 1859507,
'obStatus': 'D'}]}]
Well done! You made it to the end of this tutorial. If you have any questions or suggestions for improvements, please don't hesitate to get in touch with us via Helpdesk portal.