Distributed peer review is an increasingly popular method used for science evaluations. By distributing the review load among many peers (as opposed to burdening a single panel or committee) reviewers have a lower workload (i.e. more time to spend per proposal) while investing significantly less time overall, hence allowing more opinions per proposal and a faster proposal evaluation process. ESO is now aiming to add "distributed peer review" as a new method for the future evaluation of proposals in certain categories, while retaining the Observing Programmes Committee (OPC) to oversee the whole process.
Volunteers Needed to Test a Distributed Peer Review System at ESO
Published: 29 Aug 2018
At this stage, ESO would like to conduct a first experiment in parallel with the regular Period 103 Call for Proposals. The goal of the exercise is to gain some experience from the technical side, as well as in terms of the evaluation performance and the user experience. For this experiment, we are looking for ~40 volunteer Principal Investigators (PIs). These PIs would review and grade eight (8) of each others' proposals. Subsequently, they would answer a short questionnaire about their experience.
PIs interested in helping ESO develop this mode should email the Observing Programmes Office at their earliest convenience, and in any case before the proposal submission deadline of 27 September. E-mails should be sent to opo@eso.org with the subject line: Distributed peer review.
Note: Distributed Peer Review is already in use or under consideration for select proposal categories at the Gemini and ALMA observatories. This particular evaluation does not replace the OPC process, nor will it have any influence on the OPC evaluation of any particular proposal. The volunteers' work will help ESO understand the pros and cons of distributed peer review as applied at ESO. It will also enable a comparison of outcomes with the scoring from the OPC panels.