Once per year, now in the third quarter of the year, the User Support Department of ESO launches a Paranal Service Mode User Satisfaction Survey campaign.  This report details the findings of the September 2019 survey campaign, while previous such reports are found here.

We view these reports as an important way to

• close the loop with the ESO Community,
• gather information on issues that need to be addressed or reinforced,
• thank all respondents, and
• demonstrate clearly that such feedback is important to us! 

To this end, here we provide a summary of the responses received and trends in these responses over the last years, predominantly in the form of graphs. It should also be stressed that for those cases where respondents did identify themselves and did make specific free-text comments we have contacted them by e-mail to address their particular comments.

Methodology and General Results

The ESO Service Mode Questionnaire is always available on-line for users to fill in but the usual rate of return is less than 2 per month. However, experience shows that a targeted campaign focused on a single (in this case Phase 2 related) aspect results in many more survey completions.

In September 2019, we again took this approach, and asked Principal Investigators (PIs) of Service Mode runs scheduled for Paranal in Periods 103 and/or 104¹ (plus their then-active Phase 2 delegates) to complete the survey by a fixed deadline. We thus solicited a response from 473 PIs and their then-active Phase 2 delegates (293 individuals). Because of overlap this amounts to a total of 683 individuals who were contacted via e-mail. A deadline was set for two weeks from the date of contact.

A total of 154 responses were received by the deadline (some 24 of which were not fully complete), yielding a 19% response rate (see the figure below).  This is 5.8% lower than in 2018, and that decrease is largely driven by the ~65% increase n the number of Phase 2 delegates.  The increased number of (Phase 2) delegates itself can be attributed to the increased number of large consortium Target of Opportunity runs which require very long lists of observation triggerers.

Interactive Figure Features

The figures below are all interactive.  By this we mean:

• Puttling the cursor over the plot will display the data values on the screen.
• Clicking on the menu icon in the upper right (the three short parallel horizontal lines) will open a menu of print/download options.
• For those figures with legends to the right of the plot clicking on any entry in the legend will toggle display of the corresponding data within the graph.
• Clicking on the Linux wedge of the final plot will show the breakdown by distribution.  

As a start in detailing the results from the survey, in the figure below we show the number of responses we received per instrument. In spite of the overall good response rate the large number of instruments offered in Service Mode means that on average this year we received 12 responses per instrument (representing a decline from last year's figure of 14.5 responses per instrument).

In the following three stacked histogram plots we present a general overview of user satisfaction (in percentage of responses) with three general items:

• the help/advice provided by the User Support Department during the Phase 2 process
• the Phase 2 web documentation, and
• the overall support provided by the User Support Department.

Note that the sum of the responses to the question about one’s satisfaction with the help/advice provided is less than the total number of survey responses. However, there is no reason to expect, a priori, that these two numbers should be equal. This is because the responses to this question come from a subset of respondents (those that actually received help/advice at Phase 2), each of which may have received help/advice from multiple instruments.

The plots, designed to show the trend in user satisfaction expressed in survey results since March 2015, clearly show the consistently high satisfaction with these services offered by the User Support Department.

Seeking Help, Run Information, and Run Problem Resolution

Amongst the respondents 20.1% indicated that they had contacted ESO for non-Phase 2 related reasons within the previous 6 months (somewhat lower than last year's survey). Of these 31 respondents, 27 contacted ESO via an e-mail to usd-help@eso.org, with the remainder distributed between other methods (e.g. clicking on ‘Ask for help’ within the ESO User Portal). Below we show the degree to which these respondents were satisfied with various aspects of the resulting exchange with ESO, as above with the results of the previous 5 surveys for comparison. In all aspects, the satisfaction level is consistently high.

Some 73% of the respondents checked on-line for information regarding the progress of their observational programmes (about the same as last year. The survey asked those that did check for that information how much they agreed with four statements about that information. The outcome of those questions is shown below also displaying the results of previous year's surveys for comparison. As in the past, with a few exceptional cases the presentation of this information was deemed easily locatable, clearly presented, up-to-date, and complete.

Finally, we asked if the survey participant was alerted to any problems with their Service Mode observations during the period. For those that said that they had been contacted (46, essentially consistent with the last years) we then asked to what extent they agreed with two statements describing the contact and the problem resolution. The answers are shown below. As in the past, with a few exceptional cases the problem was clearly described and a fair solution achieved.

p2, and Other Observation Preparation Tools

Unlike in years past, this year we asked no questions about the now-replaced Phase 2 Proposal Preparation tool (P2PP), restricting ourselves to questions about its replacement, p2. As in the past, we also asked about other, instrument-specific, observation preparation tools. For comparative purposes we include still the 2018 P2PP results for a direct side-by-side comparison of P2PP and p2. In general the satisfaction levels with these tools are somewhat poorer than with the above mentioned services provided by the User Support Department. In addition, the new p2 is overall well accepted (in fact, quite comparably with P2PP, with a suggestion of increasing satisfaction with its ease of use and functions provided), though with still room for improvement, especially in its documentation.

ESO has released a Phase 2 Application Programming Interface (API) that can be used to create, modify, or delete observation blocks (OBs), containers and an accompanying ReadMe file that define an observing run.  We asked those users of p2 whether or not they had made use of this powerful facility.  The way respondents replied is shown below. In that plot we see that there is still room for improvement in terms of user uptake of the API.

Since the number of responses per observing preparation tool other than p2 is rather limited (see the table below), any presentation of individual-tool responses on documentation, ease of use, or functionality would suffer from small number statistics. Thus, in the figures below answers for all tools are combined. As with p2, when one considers the other tools as an emsemble we see room for improvement.

Related to the above tools is, of course, the suite of Exposure Time Calculators.  Thus, we asked survey participants the question, “How satisfied are you with the ETCs you have used?” The responses are shown below. There are consistently few respondents that express dissatisfaction with the ETC. The over-the-years-steady ~9% (average) of "No opinion" answers could be interpreted as that fraction of the respondents that did not use any ETC.

For the preparation of P104 observations, a new feature was introduced in the p2 tool to help generate finding charts automatically.  We asked how satisfied users were with the new tool in terms of accompanying documentation and its usefulness (as compared to any other alternatives for producing finding charts).  The satisfaction levels are very high indeed, and very encouraging!

And lastly we asked survey participants, "Which operating system(s) do you use for ESO tools (e.g. for proposal/observation preparation, data reduction), excluding any browser-based tools?"  The breakdown of responses is shown in the interactive figure below.

Notes:

¹The total time allocated in Service Mode for Periods 103 and 104 was 13943.2 hours, while the corresponding number for Visitor Mode was 2669.4 hours. Thus, the September 2019 survey targetted PIs (and their then-active delegates) representing 83.9% of the total Paranal time allocation, including all public surveys.