Once per year, now in the third quarter of the year, the User Support Department of ESO launches a Paranal Service Mode User Satisfaction Survey campaign.  This report details the findings of the September 2018 survey campaign, while previous such reports are found here.

We view these reports as an important way to

• close the loop with the ESO Community,
• gather information on issues that need to be addressed or reinforced,
• thank all respondents, and
• demonstrate clearly that such feedback is important to us! 

To this end, here we provide a summary of the responses received and trends in these responses over the last years, predominantly in the form of graphs. It should also be stressed that for those cases where respondents did identify themselves and did make specific free-text comments we have contacted them by e-mail to address their particular comments.

Methodology and General Results

The ESO Service Mode Questionnaire is always available on-line for users to fill in but the usual rate of return is less than 2 per month. However, experience shows that a targeted campaign focused on a single (in this case Phase 2 related) aspect results in many more survey completions.

In September 2018, we again took this approach, and asked Principal Investigators (PIs) of Service Mode runs scheduled for Paranal in Periods 101 and/or 102¹ (plus their then-active Phase 2 delegates) to complete the survey by a fixed deadline. We thus solicited a response from 512 PIs and their then-active Phase 2 delegates (178 individuals). Because of overlap this amounts to a total of 616 individuals who were contacted via e-mail. A deadline was set for two weeks from the date of contact.

A total of 185 responses were received by the deadline (some 32 of which were not fully complete), representing a 24.8% response rate (see the figure below). Here we note that as of 2017 we have slightly refined handling of incomplete responses. 

Interactive Figure Features

The figures below are all interactive.  By this we mean:

• Puttling the cursor over the plot will display the data values on the screen.
• Clicking on the menu icon in the upper right (the three short parallel horizontal lines) will open a menu of print/download options.
• For those figures with legends to the right of the plot clicking on any entry in the legend will toggle display of the corresponding data within the graph.
• Clicking on the Linux wedge of the final plot will show the breakdown by distribution.  

As a start in detailing the results from the survey, in the figure below we show the number of responses we received per instrument. In spite of the overall good response rate the large number of instruments offered in Service Mode means that on average this year we received about 14.5 responses per instrument (representing a decline from last year's figure of 15.5 responses per instrument).

In the following three stacked histogram plots we present a general overview of user satisfaction (in percentage of responses) with three general items:

• the help/advice provided by the User Support Department during the Phase 2 process
• the Phase 2 web documentation, and
• the overall support provided by the User Support Department.

Note that the sum of the responses to the question about one’s satisfaction with the help/advice provided is less than the total number of survey responses. However, there is no reason to expect, a priori, that these two numbers should be equal. This is because the responses to this question come from a subset of respondents (those that actually received help/advice at Phase 2), each of which may have received help/advice from multiple instruments.

The plots, designed to show the trend in user satisfaction expressed in survey results since March 2014, clearly show the consistently high satisfaction with these services offered by the User Support Department.

Seeking Help, Run Information, and Run Problem Resolution

Amongst the respondents 25.4% indicated that they had contacted ESO for non-Phase 2 related reasons within the previous 6 months (roughly the same percentage as in last year's survey). Of these 47 respondents, 44 contacted ESO via an e-mail to usd-help@eso.org (again consistent with September 2017), with the remainder distributed between other methods (e.g. clicking on ‘Ask for help’ within the ESO User Portal).  Below we show the degree to which these respondents were satisfied with various aspects of the resulting exchange with ESO, as above with the results of the previous 5 surveys for comparison.  In all aspects, the satisfaction level is consistently high.

Some 70% of the respondents checked on-line for information regarding the progress of their observational programmes (somewhat lower than last year but consistent with the percentage from 2016).  The survey asked those that did check for that information how much they agreed with four statements about that information. The outcome of those questions is shown below also displaying the results of previous year's surveys for comparison.  As in the past, with a few exceptonal cases the presentation of this information was deemed easily locatable, clearly presented, up-to-date, and complete.

Finally, we asked if the survey participant was alerted to any problems with their Service Mode observations during the period. For those that said that they had been contacted (51, essentially consistent with the last years) we then asked to what extent they agreed with two statements describing the contact and the problem resolution. The answers are shown below.  As in the past, with a few exceptonal cases the problem was clearly described and a fair solution achieved.

P2PP, p2, and Other Observation Preparation Tools

Below, we show details of the feedback received on different aspects of the now-replaced Phase 2 Proposal Preparation tool (P2PP), the replacement tool, p2, and other, instrument-specific, observation preparation tools.  For comparative purposes we show not only previous year's P2PP results, but also a direct side-by-side comparison of P2PP and p2 where applicable (p2 does not require installation, hence no comparison with that aspect of P2PP can be made).  In general the satisfaction levels with these tools are somewhat poorer than with the above mentioned services provided by the User Support Department.  In addition, the new p2 is overall well accepted (in fact, quite comparably with P2PP), though with still room for improvement, especially in its documentation.

ESO has released a Phase 2 Application Programming Interface (API) that can be used to create, modify, or delete observation blocks (OBs), containers and an accompanying ReadMe file that define an observing run.  We asked those users of p2 whether or not they had made use of this powerful facility.  The way respondents replied is shown below. In that plot we see that there is still room for improvement in terms of user uptake of the API.

Since the number of responses per observing preparation tool other than P2PP or p2 is rather limited (see the table below), any presentation of individual-tool responses on documentation, ease of use, or functionality would suffer from small number statistics. Thus, in the figures below answers for all tools are combined.  As with p2, when one considers the other tools as an emsemble we see room for improvement.

Related to the above tools is, of course, the suite of Exposure Time Calculators.  Thus, we asked survey participants the question, “How satisfied are you with the ETCs you have used?” The responses are shown below.  There are consistently few respondents that express dissatisfaction with the ETC.  The ~9% (average) of "No opinion" answers could be interpreted as that fraction of the respondents that did not use any ETC.

And lastly we asked survey participants, "Which operating system(s) do you use for ESO tools (e.g. for proposal/observation preparation, data reduction), excluding any browser-based tools?"  The breakdown of responses is shown in the interactive figure below.

Notes:

¹The total time allocated in Service Mode for Periods 101 and 102 was 12791.85 hours, while the corresponding number for Visitor Mode was 3376.4 hours. Thus, the September 2018 survey targetted PIs (and their then-active delegates) representing 79.1% of the total Paranal time allocation, including all public surveys.