Dual-Anonymous Guidelines for proposal preparation and submission

Introduction
Guidelines for applicants
Guidelines for reviewers
Types of anonymisation violations
Frequently Asked Questions on Dual-Anonymous Peer Review
An example of text anonymization
Acknowledgements
References, further reading and additional material

Introduction

Like the vast majority of scientific facilities and journals, ESO has so far used the single-anonymous peer-review (SA) in its time allocation process. In this paradigm, the principal investigator (and more in general the proposing team) does not know who the reviewer/s will be, while the reviewers are provided with extended information on the team. This usually consists of the applicants’ names, affiliations, publications relevant to the subject, previous usages of the given facility and so on. Implicitly and/or explicitly this implies that this information is used to judge the proposal quality and the ability of the team to carry out the proposed project. If on the one hand this may be useful to provide the reviewers with more background for evaluating the science case, on the other it may constitute a source of systematics, averting the focus of the review from the science case to the quality of the proposing team.

As a first countermeasure and following the practice recently deployed at other large ground-based and space-born facilities, as of Period 102 ESO has already:

removed the principal investigator’s (PI) name from the front page of the proposal distributed to the reviewers;
removed all the affiliations and e-mail addresses;
listed all co-investigators (CoIs) in alphabetical order on the last page of the proposal.

Despite of these actions, the reviewers are still able to identify the team, and are provided with the information regarding the publication record, previous work in the given field and previous usage of the facility. Once this information is there, it is to be expected that it is used, consciously or unconsciously, possibly diverting the discussion and the evaluation from the science case and introducing systematic effects which are not related to the effective scientific merit of the application. Gender bias is one of them (see Patat 2016 for a statistical study on time allocation at ESO).

This issue is addressed by the dual-anonymous peer review (DA). In this paradigm neither the applicants nor the reviewers know each other’s identity. DA has been already deployed by some organizations and is considered as the most effective form of peer-review (see for instance Mulligan, Hall & Raphael 2013). DA has been adopted by the Space Telescope Science Institute for allocating HST time as of Cycle 26 (Strolger & Reid 2019; Strolger & Natarajan 2019) and by ALMA as of Cycle 8. A recent study by Johnson & Kirk (2020) has shown that anonymization can nearly eliminate gender systematics from application processes. In particular, their analysis shows that when the identities of the team were known, about 50% of the discussions include some mention of the Principal Investigator or the team. And that gives an obvious opportunity for unconscious bias to creep back into the process. In addition, as shown by the outcome of HST Cycle 27, DA has also the potential of levelling the playing field between new and established researchers (Reid 2019, slides #9 and #10).

The problem of systematics affecting minorities can also be addressed implementing pro-active measures on the Organization's side, like 1) rising the awareness in the panels, and 2) forcing quotas. However, this creates additional issues. The first is that unconscious bias is so automatic in human beings that is very difficult to overcome even with intensive training. The second causes a backlash against minorities, because it creates the impression that what is achieved by them is not based on merit but just the result of the forced quotas. As argued by Johnson & Kirk (2020), DA addresses both problems, because it "eliminates the possibility for bias to occur, rather than trying to overcome it, and because it is difficult to argue that removing names from proposals is giving an unfair advantage to anyone".

In the DA approach, the applicants still provide all the information about the proposing team, but this is not displayed in the proposal version that is distributed to the reviewers. The information is stored by the organization that facilitates the process and disclosed to the reviewers only after the scientific evaluation and final ranking are completed.

Following the recommendations of the Time Allocation Working Group (Patat 2018), the information campaign and the P106 dry run, ESO has decided to move to the DA approach for allocating its telescope time as of Period 107 (April 1st - September 30th, 2021).

This page provides guidelines for assisting the applicants in the anonymization of their proposals.

Guidelines for applicants

This section provides the guidelines the applicants must follow for anonymizing their proposals, to help conceal the identity of the team and to ensure a fairer evaluation process. These apply to the entire proposal with a few, notable exceptions. The sections Background and Expertise and Data Product Delivery Plan (in the case of Large Programmes) are the only fields in which information on the proposing team can be disclosed. These fields – as well as the fields Investigators, Previous Usage and Applicants’ Publications – will not be included in the material distributed to the reviewers during the proposal review phase, and will only be accessible to them after the ranking phase is completed.

Here follow the main criteria the users must comply with:

Do not include applicant names or affiliations anywhere in the scientific rationale. This refers to all anonymous parts of the document, including diagrams, figures, captions, watermarks, …
The same applies to the proposal title, abstract, awarded and future time requests, time justification and the special remarks box.
When citing literature and providing references, especially in the case of self-referencing, third person neutral wording must be used. For instance, the sentence “as we have demonstrated in Galilei (1610)” must be rephrased as follows: “as shown by Galilei (1610)”.
Future (or current) time awarded at other facilities for the same science case to the proposing team must be cited anonymously, without reporting any name of programme ID. For example, the sentence: "We have obtained 30 orbits of HST time (programme HST001, PI: Galilei) for..." must be replaced by: "Thirty orbits of HST time were awarded to our team for...".
Acknowledgments must be avoided, and so must references to possible grant funding (the latter can be listed in the Background & Expertise section of the proposal).

Citing own work, data and models

One of the main difficulties encountered by the applicants while quoting in an anonymous way their previous work, data, data-reduction tools and/or modelling capabilities, is the fear that, in the lack of a proper and explicit citation, the reviewers would not trust them. Also, applicants may refrain from using the first person (though anonymous) fearing that this would violate to some extent violate the anonymisation guidelines.

In those cases in which the applicants wish to use the first person forms (e.g. "As we have shown (private communication)", "The data will be processed with our pipeline", ...), they should keep in mind that:

The reviewers are instructed not to try to guess the team's identity by explicit searches on the web. This constitutes a violation of the reviewer's guidelines. For the same reason, team identities resulting from active searches by the reviewers, will not be considered as anonymisation violations.
The reviewers are instructed to trust the statements made by the applicants even in the cases in which, because of the anonymisation, it is not possible for them to check directly in the literature.
In case of doubts, the reviewers can ask ESO to verify the correctness of one or more statements. This will be done by checking the content of the Background and Expertise section. It is therefore important that the applicants fill this section with all the details they reckon are necessary to allow ESO to verify the team's qualifications with respect to the proposed observations (expertise, data reduction, analysis, modelling).

Instructions for the specific cases are provided in the next sections.

Proprietary/unpublished work/data/codes

Here follows a summary on how to properly quote proprietary/unpublished work/data from the proposing team. For each case an example of anonymized reference is provided.

Published Proprietary models/codes (published/advertised, developed by the applicants but not publicly accessible)

"Calculations run with FANTASTIC code (private communication) predict that..."

Unpublished Proprietary models/codes (developed by the applicants, not published/advertised, not publicly accessible)

"Based on a model we have developed (in preparation) we predict that..."

Public data, unpublished results (data publicly accessible to everybody, no publication)

"After inspecting FORS2 archival data (programme ID 105.20AB), we have detected emission lines in..."

Proprietary data, yet to be taken (proposal accepted, data yet to be taken, at this or other facilities)

"The candidates will be selected from the data provided by our survey approved at JWST."

Proprietary data, unpublished results (data accessible only to the proposing team, no publication)

"From the analysis of an existing X-SHOOTER spectroscopy data set we have selected a sample of suitable candidates..."

Proprietary data, published results (data accessible only to the proposing team, published)

If the statement refers to results contained in a publication:

"As shown by Author et al (2019), NGC300 is a perfect laboratory for ..."

If the statement refers to new results, not contained in a publication:

"The analysis of existing VISIR data has revealed that..."

Paper submitted by the proposing team, manuscript not publicly available

This covers the cases in which the proposing team has submitted a paper to a journal, but has not made it publicly available (e.g. in arXiv), so that it cannot be cited with a reference.

"As shown in a paper submitted by our collaboration (priv. comm.)"

"The expansion velocity is 450 km/s [3]." - "[3] Our collaboration, submitted." or "[3] Submitted (priv. comm.)"

Paper submitted by the proposing team, manuscript publicly available

This covers the case in which the proposing team has submitted a paper to a journal and has made it publicly available (e.g. in arXiv) so that it can be referred to.

"This star is very peculiar [4]" - "[4] Brahe et al. 2023, https://arxiv.org/abs/999.99999"

"As recently shown by Brahe et al. (2023, https://arxiv.org/abs/999.99999)"

All these formulations are sufficiently ambiguous and can be safely used in the text. Of course, you can (and you should) be explicit in the non-anonymized sections (publications list, previous usage and Background & Expertise). This will give the reviewers the opportunity to check the correctness of the anonymized statements, once the review is completed, should they reckon this is critical for the specific case.

Continuation/follow-up proposals

In many cases a proposal is a continuation or a follow-up of a recently approved project, led by the same team. The data and the results from that proposal (especially in the case of a pilot programme) may play a crucial role for the new submission and hence need to be properly referred to. This is a particularly tricky case, one of those in which reaching 100% anonymity is hard. Depending on the status of the data (proprietary/public) and the results (unpublished/published), the considerations presented in the previous section do apply to this case as well.

The general guideline is that, whatever option the applicants decide to use, the indentity of the proposing team has to remain sufficiently ambiguos. For instance, a sentence like this:

"This is the continuation of an approved proposal. The analysis of the data obtained so far has shown that Jupiter has at least four satellites (Galilei 1609)."

violates the anonymization guidelines. Although the reference is quoted in an impersonal form, it does not leave any doubt on the identity of the applicant.

Understandably, the applicants may wish to signal to the reviewers that the current proposal is linked to a previous allocation. This is achieved indicating the instrument and the time already allocated to the project in the section AWARDED AND FUTURE TIME REQUESTS of the proposal. A comment can be added to provide more details, in a fully anonymous way (e.g. "Allocated in P106. Completed. Paper submitted." or "Allocated in P105. Incomplete data-set.", etc). Programme ID/s and/or literature references must not be listed in this section. More explicit details can be provided in the REPORT ON PREVIOUS USAGE OF ESO FACILITIES and Background and Expertise sections of the proposal.

For the purposes of signalling to the referees that the current submission is a continuation of an already approved programme this should be sufficient.

Here follow a few examples on how previous allocations and corresponding results should be referred to in the Scientific Rationale and Immediate Objective sections of the proposal.

Unpublished results

"An analysis of existing HAWK-I data (private communication) has shown that..."
"Based on the analysis of data obtained in previous semesters, we have now refined the sample..."
"Recently, a pilot study has revealed that ... (private communication). We propose follow-up observations to..."

Published results

(in the following examples, Explicit et al. are both the authors of the paper/s and the applicants)

"A study of a sample of 17 galaxies has shown that 90% them are the products of major mergers (Explicit et al. 2019) . With this proposal we plan to extend that study to a sample of 231 galaxies. This will enable..."
"Explicit et al. (2020) have shown that HeI6678A can be used to determine ... We propose to exploit this technique to..."

Caution in using the explicit reference/s has to be exerted depending on the sentences that preceed and follow the citation. If an obvious connection can be made between the publication and the time already allocated to the same project (if specified in the dedicated section), it is more appropriate to use the implicit version suggested for the unpublished results:

"A pilot study of a sample of 17 galaxies has shown that 90% them are the products of major mergers (private communication). With this proposal..."
"It has been recently demonstrated that HeI6678A can be used to determine ... (private communication). We propose to..."

Generic references to previous allocations

"Time was already allocated to this project for the RA range 0-12. This proposal extends the request to the RA range 12-24."
"Time was already allocated to this project. Only 50% of the observations were carried out. With this proposal we plan to complete the sample."

Complying with the above guidelines requires working on the grammar and structure of the scientific rationale. As a consequence, it will not be possible to re-cycle previous material without reviewing it in the light of the anonymization requirements. The proposing teams should take this into account when planning their submission/s, because text anonymization requires some extra effort.

Non anonymized sections

The following sections:

Previous usage of ESO Facilities
Applicant's publications
Data products delivery plan (for Large Programmes only)
Background & Expertise (see below)

must be filled as usual (i.e. in a non-anonymous way).

Team Background and Expertise

As part of the DA implementation, the applicants are now required to fill in the new Team Background and Expertise (B&E) section in the p1 form. This is meant to be a short description of the background, expertise and roles of the various team members in the context of the science case discussed in the proposal. This section is limited in space. For small teams the applicants may wish to provide a sentence for the qualifications of each member, while for larger teams (e.g. in Large Programmes), only the leading roles need to be specified. Here follows an example.

G. Galilei has expertise in VLT data reduction for X-Shooter and FORS2 which are requested in this proposal. G. F Sagredo is an expert in planet formation and F. Salviati is leading the field of planetesimal accretion theory. V. Viviani is a student of prof. G. Galilei and is working on his PhD thesis on the moons of Jupiter. The team has a strong background in ground-based observations in UV, optical and NIR, with specific competences in the field of planetary science, both experimental and theoretical. The team has published ten papers and a book on the subject in the last five years. The same team has obtained time on the Highest Spheres Telescope (HST#0123) for a complementary study. This project is supported by the GDT Grant #0017 from the Grand Duchy of Tuscany.

Important note

Users must not add comments with PDF editors to their rationales as, depending on the editing software, the author/s of the comments is/are visible when displaying the uploaded PDF, hence disclosing their identities.

Guidelines for reviewers

Starting with Period 107 anonymization is mandatory and the dual-anonymous procedure is also enforced on the reviewers’ side. The reviewers should follow these guidelines:

The review must aim at selecting the most promising proposals, not the best proposing teams.
The ranking is purely based on the scientific merit of the proposals: the pre-meeting review and the panel discussions must focus on science only.
The background and expertise of the applicants with ESO or other scientific facilities is not to be considered during the evalluations.
The reviewers should not try to guess the PI’s or the team’s identity. Actions taken in this sense by the reviewers will be considered as violations of these guidelines.
For the same reason, team identities resulting from active searches by the reviewers, will not be considered as anonymisation violations.
The chairs of the panels must refocus the discussion whenever this moves to the team identity, expertise or publication record.
The main purpose of the anonymization is to reduce the sources of "distraction", which may influence the objectiveness of the process.
The reviewers should flag to the Observing Programmes Office all the cases that they reckon do not comply with the anonymization rules spelled out in the Guidelines. This is best done using the specific text box in the Proposal Evaluation Interface.
In principle, only open violations will be considered for proposal rejection by the Director General. Although the specific cases will require a closer scrutiny, open violations are those which clearly indicate that the proposing team did not make any effort to obfuscate their identity.
In general, the applicants have done a good job when the identity of the team is reasonably ambiguous.
Cases in which the identity of the team cannot be derived directly from the proposal, but it can be determined by deliberate web searches by the reviewers (which are anyway highly discouraged), cannot be considered as a violation of the rules.
The reviewers must consider that it is impossible to reach 100% anonymity in all cases. There will always be certain cases in which it is impossible to conceal the identity of the teams. This is a fact that has to be accepted.
In their final comments to the applicants, the panels should not mention any possible anonymisation issue. Whenever applicable, this will be dealt by the Observatory in a separate note to the proposing team.
As a rule, the reviewers should trust the statements made by the applicants about their declared achievements, data reduction, analsysis and modelling capabilities, although these are kept anonymous. In case of doubt, the reviewers may ask ESO to verify the correctness of the statements made by the applicants. For this they should use the official contact channel.

Types of anonymisation violations

When scrutinizing the anonymisation violation reports submitted by the reviewers (both from Panels and DPR), ESO will categorize them in three classes:

Major violations - those revealing directly and unambiguously the identity of the proposing team (e.g. "As we have shown (Galilei et al. 1609)"). Major violations are flagged to the Director General and lead to the disqualification of the proposals.
Minor violations - those revealing unambiguosly but indirectly the team's identity. In these cases the identity cannot be deduced from the proposal's content, but may be derived by secondary sources (e.g. including project names which are quoted in cited publications). Minor violations do not lead to proposal disqualification, but will turn into a warning to the PI, asking them to fix the issue/s for future submissions.
No violations:
1. The team's identity remains ambiguous (e.g. there is more than one possibility);
2. The team has thoroughly followed the anonymisation guidelines and it is impossible to fully hide their identity (e.g. very well known teams working on very specific science cases);
3. Cases in which the identity is found by explicit searches by the reviewer.

Frequently Asked Questions on Dual-Anonymous Peer Review

Why is ESO considering Dual-Anonymous Peer Review (DA)?
Is it really possible to fully anonymize a proposal?
If the review is to be anonymous, why am I asked to provide information about my team?
Which parts of the proposal will be hidden to the reviewers during the review?
How can you demonstrate that DA is better than single-anonymous peer review?
Do you have evidence that DA reduces the systematics?
How will ESO make sure that the reviewers do not waste time in guessing the team’s identity?
What does DA mean for the applicants?
How will ESO make sure the time is allocated to teams capable of dealing with the data?
How will the review process change?
How does ESO make sure the accepted proposals are feasible if the Panels cannot access the team’s past experience?
I have obtained telescope time for the upcoming cycle at other world-class facilities for the same science case. How do I make this known to the reviewers?
We have unpublished and/or proprietary data. How can we refer to it in the proposal without disclosing our identity?
Our team has been obtaining time at this facility for many cycles. How do I make this known to the reviewers?
I am going su resubmit a proposal that was rejected in the past. How do I inform the reviewers that this is a resubmission without revealing my identity?
As a reviewer, how can I be sure that I allocate time to reliable teams?
Will this affect also Large Programmes?
What happens if a proposal does not comply with the anonymization rules?
My proposal is so unique that it is impossible to make it fully anonymous. Will it be disqualified?
I am going to submit a GTO proposal and the proposing team's identity is very easy to guess. What should I do?
How will ESO deal with conflicts if the team identity is not accessible to the reviewers?
My case is very special and I do not know how to properly apply the anonymization rules to it. What should I do?

1 Why did ESO implemented Dual-Anonymous Peer Review (DA)?

The goal is to minimize potential sources of systematics and put the reviewers in a better condition for focusing on the science case of a proposal. Conscious or unconscious biases can play an important role in creating systematic effects in the process. For instance, they are likely at least partially responsible for the observed difference in the success rates of proposal submitted to ESO by female and male PIs (see Patat 2016). While it is understood that this is a complex problem, there are clear indications that DA can reduce the systematics that affect gender, scientific seniority, affiliation, ethnicity and so on. In addition, the case of HST has shown that proposal anonymization helps a lot the panels to concentrate on the science. In this respect, it is also worth noting that HST TAC members reported that this type of review is actually a lot less stressful and more collegial, which is an important consideration given the increasing number of large-scale proposal reviews.

2 Is it really possible to fully anonymize a proposal?

Certainly not in all cases. However, with the exception of very special and unique instances, guessing the identity of a proposing team is more difficult than one would think. Statistical studies conducted in other fields show that even when the overall team identity is correctly guessed, the PI identity remains unknown in the majority of the cases. It is important to emphasize that the ultimate aim is NOT to make it impossible for anyone to guess who is on the proposal, but rather to change the tenor of the discussion so the focus is on the science, not the scientist. The main goal is to level the playing field for everyone, not address specific imbalances within particular sub-groups.

3 If the review is to be anonymous, why am I asked to provide information about my team?

Anonymous review does not mean that the proposals are anonymous, but that the team information is kept separated from the scientific evaluation process. The Observatory, which facilitates the process, has full access to all the information as the Director General is the ultimate responsible for the time allocation process. The information you provide on your team is initially concealed to the reviewers and is disclosed only at the end of the process. Therefore, indeed, you have to provide it when you prepare your proposal.

4 Which parts of the proposal will be hidden to the reviewers during the review?

The sections: Investigators, Previous Usage, Applicants’ publications and Background & Expertise will not be included in the proposal version distributed to the reviewers. Their content will be made available only once the rank lists have been compiled.

5 How can you demonstrate that DA is better than single-anonymous peer review?

The deployment of DA is one of the actions that are being taken in order to reduce systematics that are known to affect single-anonymous peer-review (SA). The purpose is to improve the equity and integrity of the review process, in the attempt of making the selection fairer. ESO will keep monitoring the success rates (by country, gender, professional seniority, …), and this will allow to quantify the effect and take further corrective measures in case of need. After the recent requirement for providing more detailed information about applicants and reviewers in the ESO User Portal (see this announcement), more accurate studies will soon be possible. And, of course, one can also ask the same question for SA.

6 Do you have evidence that DA reduces the systematics?

DA has already been deployed for more than ten years in different fields outside astronomy and the results have been documented in a number of publications. The conclusion is that the obfuscation of team information turns into a reduction of the systematics in the review process. The HST Working Group on the anonymization of proposals has compiled a list of relevant publications. See Johnson & Kirk (2020) for the specific case of HST.

7 How will ESO make sure that the reviewers do not waste time in guessing the team’s identity?

During the pre-meeting phase when the proposals are reviewed online by the single reviewers there is no way of doing that. But this applies to any of the review guidelines and it is not specific to the single- vs. dual-anonymous implementation. In the case of HST, the Space Telescope Science Institute has introduced the role of “levelers”, who are in charge of making sure the discussions focus only on science. The levelers are selected from the STScI staff members. In the current implementation at ESO this role is delegated to the panel chair for the online meeting discussions. Additionally, the Scientific Assistants are instructed to report possible deviations from the recommended behavior.

8 What does DA mean for the applicants?

As described in the Guidelines for applicants, they will have to comply with some simple rules when preparing the title, the abstract, the special remarks and especially the scientific rationale of their proposals. This requires a change of style (grammar and structure) in the way their previous work (both publications and usage of scientific facilities) is described and used in the text to support the proposed case. This implies that previous versions of the proposal will need to be reviewed and adapted to the new anonymization requirements set by the DA paradigm. The applicants are also requested to fill in the Background and Expertise field in the p1 web form. This is supposed to provide a concise summary of the applicants’ profiles and expertise relevant for the proposed science case. This field will not be included in the material distributed to the reviewers during the review and will only be accessible to them upon request and only after the ranking phase is completed.

9 How will ESO make sure the time is allocated to teams capable of dealing with the data?

During the review process the Panels are instructed to take note of the cases which require special competences or expertise levels. When the team information will be made available to them, they will be in the position of making a recommendation to the Observatory, expressing their possible concerns on the qualifications of the team with respect to the specific science case. However, they will not be in the position of changing the raking, which is purely based on the scientific merit of the proposal. The can only make a recommendation to the Director General for possible actions.

10 How will the review process change?

In terms of the various steps (asynchronous online grading, triage, face-to-face discussion and grading) nothing changes up to the final compilation of the merged ranking lists per telescope, which are produced by the Observing Programmes Office once the review is completed. At that point, the information on the proposing teams, which was concealed to the reviewers, is made accessible to the Panel Chairs. This is examined and the cases judged as critical during the review are discussed and flagged to the Observatory for possible disqualification. Reasons for disqualification include open violations of the anonymization rules, unethical behavior (e.g. deliberate misrepresentation of the team’s expertise and/or available resources) and justified concerns on the applicant’s competences for the specific science case. The disqualification flag is only a recommendation, and the final decision will be taken by the Director General, who will also evaluate the possibility of flagging obvious misbehaviors to the host institutes of the applicants.

11 How does ESO make sure the accepted proposals are feasible if the Panels cannot access the team’s past experience?

From the technical point of view, and as in the case of the current SAPR paradigm, the accepted proposals are subject to a feasibility assessment by the instrument specialists from the Observatory. From the scientific point of view, the reviewers have access to this information AFTER having expressed their opinion on the scientific merit of the proposal, which was formulated without being influenced by possible concerns on the team’s profile. At that point the panels can flag their concerns about the reliability/suitability of the team and make a recommendation to that effect.

12 I have obtained telescope time for the upcoming cycle at other world-class facilities for the same science case. How do I make this known to the reviewers?

This can be done in a fully non-anonymous way in the Background & Expertise field, which is made available to the panels only when the ranking is completed. In the anonymized proposal text this must be mentioned in an anonymous way, and referred to using the "(private communication)" expression.

13 We have unpublished and/or proprietary data. How can we refer to it in the proposal without disclosing our identity?

You can quote unpublished or proprietary data using an impersonal form. See the guidelines section for more details. Then, in the Previous Usage of ESO Facilities section of the proposal you can explicitly discuss this.

14 Our team has been obtaining time at this facility for many cycles. How do I make this known to the reviewers?

You can signal to the reviewers that this is the continuation of an accepted proposal in the Awarded and Future Time Requests section of the proposal. This has to be done in an anonyous way (i.e. not quoting any programme ID or giving any other information that would reveal the proposing team's identity). You can then discuss and present all the previous work in the scientific rationale in an anonymous way, hence making clear to the reviewers that this is an important topic, that has been systematically and successfully followed in the past. This will make the panel aware of the fact that a significant amount of time and effort was invested on the specific topic, properly setting the stage for the discussion as to whether the proposal makes a sufficiently compelling case for the need of new data. Finally, explicit statements can be made in the non-anonymized sections of the proposal (e.g. Background & Expertise, Previous Usage of ESO Facilities).

15 I am going to resumbit a proposal that was rejected in the past. How do I inform the reviewers that this is a resubmission without revealing my identity?

It is worth nothing that, in the current implementation, the users are not required to indicate resubmissions. However, if you feel that signalling the resubmission is useful for the process (for instance because you wish to show how you have addressed the points made by the previous review), you can do this in the text in an anonymous way, i.e. without providing the programme ID of the previous submission. Alternatively, you may wish to simply address the points made by the previous review without mentioning that this is actually a resubmission.

16 As a reviewer, how can I be sure that I allocate time to reliable teams?

Technically, the single reviewers and, more in general, the Panels do not allocate time: they only make a recommendation to the Director General in the form of a ranked list. The final allocation is then the result of a complex process which involves a number of stakeholders and constraints on which the Panels do not have any control (this includes technical activities, atmospheric conditions, instrument loads, right ascension distributions, carry-over from previous periods, ongoing Large Programmes, etc.). In any case, both in SA and DA the panels have the possibility of expressing their concerns about the teams. In the DA paradigm this happens after the rankings are compiled, but the possibility of flagging critical cases is still present, without the negative drawback of of influencing the scientific judgement. It is the Observatory which, based on the Panel recommendations, is responsible for allocating the time so that the scientific return is maximized. Finally, there is one further aspect. If the case is judged to be strong, it means that, scientifically speaking, the Panels reckon it is worth that ESO secures those data.

17 Will this affect also Large Programmes?

Yes. All proposals submitted for ESO telescopes will be subject to DA, and all applicants will have to comply with the anonymization rules. This will also apply to Director’s Discretionary Time (DDT) proposals. For Large Programmes, the Background & Expertise section may be longer.

18 What happens if a proposal does not comply with the anonymization rules?

Alleged violations to the anonymization rules will be flagged to the Director General and discussed internally. In case of major abuses, the corresponding proposals will be disqualified and excluded from any subsequent review step. Less serious infringements (e.g. usage of first person instead of neutral third person) will be flagged and discussed but will not be removed from the review in the panels. A final decision for those cases will be taken at the end of the process. Proper feedback on possible violations of the anonymization rules will be sent to the applicants.

19 My proposal is so unique that it is impossible to make it fully anonymous. Will it be disqualified?

No, if you have followed the anonymization rules. You should not obfuscate your science case and your models (and, by doing that, render your case weaker) to make you and/or your team anonymous. But you must use the neutral third person and cite your work as indicated in the anonymization guidelines (e.g. as private communication when required) so that your identity is not deliberately revealed to the reviewers. If yours is long-term, recurring project, already well known to the community by its name, using this name in the title and in the proposal is not considered as a violation of the anonymization rules, and it will be considered as one of those cases which is not possible to make 100% anonymous. The rationale is that, being the project so notorious, preventing the usage of the name would not solve the anonymization issue either. However, all other applicable anonymization rules must be applied to the proposal.

20 I am going to submit a GTO proposal and the proposing team's identity is very easy to guess. What should I do?

This is one of the cases in which 100% anonymity is probably impossible to reach. You must nevertheless follow the guidelines and this will be sufficient to guarantee your proposal is compliant and not subject to disqualification.

21 How will ESO deal with conflicts if the team identity is not accessible to the reviewers?

Institutional conflicts will be identified and handled based on the User Portal profiles of the applicants and the reviewers, as is the case in the current implementation. With DA, personal conflicts become, to some extent, simpler to manage. If a reviewer feels in conflict with a given proposal because s/he has a conflict with one of the applicants, the conflict itself (which would translate into the inability of expressing an objective judgement) is created by the fact that the names are disclosed. If they are not, the conflict simply does not exist. As for the scientific conflicts, these are generated by the science case and not by the team who wrote it. In this case, the reviewer is in the position of declaring that the science case is too close to her/his research field, and hence s/he may not be able to express an objective opinion (regardless of the fact that the proposal may have been written by a competing team). In this case there is no difference with SA, in that this is fully in the hands of the reviewer, as in both cases s/he may or may not declare the conflict.

22 My case is very special and I do not know how to properly apply the anonymization rules to it. What should I do?

In case of doubt, or if you feel that your specific case is not covered by these guidelines, please contact the Helpdesk via the ticketing system, using the wording: "Dual-Anonymous Peer Review" in the subject.

An example of text anonymization

Non anonymized text

Since 1609 our group has been accumulating evidence that earth rotates around the sun (Galilei 1610). In particular, our observations obtained at the Earliest Sidereal Observatory (under programmes ESO001, ESO002; Principal Investigator: Galilei) have demonstrated that Venus shows phases with a pattern that cannot be reconciled with the geocentric theory (Ptolemy 0150). In addition, we have discovered that at least four satellites orbit around Jupiter (Galilei 1610), hence demonstrating that not all celestial bodies rotate around the earth.

We have also provided further evidence of the incorrectness of the Ptolemaic model (Ptolemy 0150) with the observations of the sun we carried out with ESO telescopes (programmes ESO003 and ESO004; PI Galileo). As we discussed in Galilei (1613), we concluded that the surface of the sun is disturbed by topological defects which are not stationary and evolve with time (see also Scheiner 1612 for an alternative interpretation by a competing team). This is in evident conflict with the claims that all celestial bodies are incorruptible and composed by perfect matter (Aristoteles 0350 BC; Ptolemy 0150 and references therein). With this proposal we plan to finally demonstrate that our preliminary conclusion that earth orbits around the sun (Galileo, Salviati & Sagredo 1632) is correct, by measuring the parallax of a sub-set of the stars included in the HSC (Hipparchus 0135 BC) accessible from the southern hemisphere. With these data we will be able to settle once and forever the matter on the chief world systems (Copernicus 1543 and references therein). We have been granted 30 orbits at the Highest Spheres Telescope (programme HST#0123) to cover the part of the HSC sample with parallaxes below the limit that can be achieved from the ground. This project is supported by the GDT Grant #0017 from the Grand Duchy of Tuscany.

The same text after anonymization

Evidence that earth rotates around the sun has been accumulating since 1609 (Galilei 1610). In particular, it has been demonstrated that Venus shows phases (Galilei 1610) with a pattern that cannot be reconciled with the geocentric theory (Ptolemy 0150). In addition, it has been shown that at least four satellites orbit around Jupiter (Galilei 1610), hence demonstrating that not all celestial bodies rotate around the earth.

Further evidence of the incorrectness of the Ptolemaic model (Ptolemy 0150) was provided by the observations of the sun (Galilei 1613). These led to the conclusion that the surface of the sun is disturbed by topological defects which are not stationary and evolve with time (see also Scheiner 1612 for an alternative interpretation). This is in evident conflict with the claims that all celestial bodies are incorruptible and composed by perfect matter (Aristoteles 0350 BC; Ptolemy 0150 and references therein). With this proposal we plan to finally demonstrate that the preliminary conclusion that earth orbits around the sun (Galileo, Salviati & Sagredo 1632) is correct, by measuring the parallax of a sub-set of the stars included in the HSC (Hipparchus 0135 BC) accessible from the southern hemisphere. With these data we will be able to settle once and forever the matter on the chief world systems (Copernicus 1543 and references therein). Thirty orbits have been granted at the Highest Spheres Telescope (private communication) to cover the part of the HSC sample with parallaxes below the limit that can be achieved from the ground.

Acknowledgements

These guidelines were put together based on the experience gained at the Space Telescope Science Institute. The discussions with Neill Reid and Lou Strolger during the Dual Anonymous Proposal Review Workshop (Baltimore, September 25, 2019) and the help provided during the preparation of these guidelines are kindly acknowledged.

References

Johnson, S. K & Kirk, J. F., 2020, Dual-anonymization Yields Promising Results for Reducing Gender Bias: A Naturalistic Field Experiment of Applications for Hubble Space Telescope Time. Publications of the Astronomical Society of the Pacific, 132, 034503. Doi:10.1088/1538-3873/ab6ce0

Mulligan, A., Hall, L., & Raphael, E., 2013, Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, 64, 132-161. doi:10.1002/asi.22798

Patat, F., 2018, The Time Allocation Working Group Report, The Messenger, 173, 7-11. doi:10.18727/0722-6691/5091

Reid, N., 2019, HST TAC process and recent statistics, presentation given at the Dual Anonymous Proposal Review Workshop (Baltimore, September 25, 2019)

Strolger, L. and Natarajan, P., 2019, Doling out Hubble time with dual-anonymous evaluation, Physics Today. doi:10.1063/PT.6.3.20190301a

Strolger, N. and Reid, N., 2019, Adopting Dual-Anonymous Practices in the Reviews for Resource Allocation in Astronomy. Astro2020: Decadal Survey on Astronomy and Astrophysics, APC white papers, no. 272; Bulletin of the American Astronomical Society, Vol. 51, Issue 7, id. 272

Additional material

Last update: Mon Sep 25 11:47:15 CEST 2023

ESO — Reaching New Heights in Astronomy