Here we provide a description of the Proposal Evaluation Interface (PEI), by describing its main functionalities and how you will interact with it, in your role of Panel reviewer and/or Panel Chair. 

General remark

Although we are continuously making progress in adding new features to the PEI, the interface is still under development, therefore some of its functionalities may not be fully optimised yet. As of P112, we have basically completed the core functions to support your review work from the conflicts declaration phase to the finalisation and submission of the final feedback to the proposals PIs (for those proposals for which you are the Primary reviewer). In betweem PEI supports the preliminary and final grading (of normal proposals), the discussion (within the Panel) and the final voting (at the OPC meeting only) of Large Proposals (during the Spring meeting).  

Accessing PEI

The access to PEI is via your personal User Portal account. Once logged in, you will find a link to PEI on the left-side menu that will bring you to the Welcome page.

This Welcome screen summarises some key information about your current review (science category, your panel and your role as well as your chair). At the end of the row, there are three buttons that allow you:

to enter the proposals area (left);

to show more details about your panel (composition, progress in proposals grading) (middle)

to close the extra details about your panel (right).   

If you select Show panel details (middle button), then you will see the composition of the panel and, once the panel meetings start, also the progress of your panel proposals grading. Panel Chairs will also see a Close all panel activities button, which becomes active only when the Panel has completed all proposal reviews (at the Panel meetings). This is the last action that the Panel Chair shall perform, therefore at the very end of the last Panel meeting.

At the very top, you find a Help pull-down menu, that describes all main features of PEI (this page) and provides you with a quick table summarising the meaning of all icons used in the interface.

Preliminary grading  

When Review state is set to Proposals pre-grading, navigating PEI is quite straightforward.

By selecting the Go to proposals icon (left-most one of the three) in your Current review summary table, you will enter directly the tab (folder) My pre-grades, that lists all the proposals assigned to your Expert Panel and to which you have been assigned as Primary (P) or Secondary (S) reviewer.

At the top of the list of proposals on the left-side of the window, you may take advantage of some action button:'Export list' allows you to export in csv format all proposals of your Expert Panel (this is a temporary solution to help especially Panel Chairs with the organisation of their work across the panel meetings); 'Download all proposals/rationales' allows you to donwload all proposals or rationales in one go so that you can work offline; 'Expand/Collapse all' adds the information on the titles of each proposal.

By clickling on each proposal ID (new and old IDs are reported for your convenience) a new space will open on the right side, displaying some key information about the proposal (title and abstract) and allowing you to preform your preliminary evaluation.

Within this space, you can view the proposal rationale and/or the full PDF proposal, you can log any anonymization issue that you find while reading the proposal, you can specify your preliminary grade in the My pre-grade(s) window and you can log your annotations in the dedicated My scientific evaluation text-box.

As soon as you have entered a preliminary grade (per proposal or per run), the label Ready will appear on the left side, on the right of the proposal ID. This is meant to give you an idea of how many you have already pre-graded and how many you still have to pre-grade.  

The button at the top left Submit pre-grades will become active only once you have entered preliminary grades for all the proposals. Please be reminded that, once clicked, no un-do is possible. Once you have submitted your pre-grades, the label Ready will be replaced by the label Submitted.

In even-numbered Periods, the list will also include the Large Programmes (easy to spot as they are marked with the LP label). Expert Panels members will have to pre-grade all Large Programmes submitted to their science category, whereas OPC members usually pre-grade all Large Programmes, independently of their science category (unless the total number of Large Proposals received is overwhelmingly large - in this case, they will also pre-grade only the LPs in their own science category).  

Pre-meeting activities

As soon as the triage lists are ready to be released, PEI switches review state to Pre-meeting activities.

The row of tabs now lists:

Proposals in my panel / Conflicts and Impediments (for Panel Chairs only) / My pre-grades / Triaged proposals / Proposals grading / LPs panel discussion

Tab Proposals in my panel is a summary table of all proposals assigned to your panel and collects some basic information per proposal. It allows some level of sorting, by selectable pull-down menus or by basic text-searches (the Clear filters button allows you to cancel any previous selection and/or sorting choices) and, as your panel reviews progresses, it will record the final average grade of each proposal. This overview may also be useful to Panel Chairs, while organizing the work for the meeting slots (e.g., by selecting which proposals will be discussed on which date - for this you may find useful the Export to csv button) or to the panel, at the very end of the review process, to have an overall look at the final ranking of the panel reviews.

Tab Conflicts and Impediments is available only to Panel Chairs, offering an overview of all conflicts in their panel and where they can record any last-minute conflict and/or impediments (see next section for more details). It also offers an overview of the reviewer role (Primary or Secondary) for each panel member.  

Tab My pre-grades is what you have used during the pre-grading phase and it reports all your pre-grades, for reference.

Once a proposal is selected (by clicking on his ID), the information displayed on the right-side of the window includes also the average pre-grade and its associated standard deviation, to possibly idenitify those proposals that have received very discordant evaluations. These may deserve a closer look and discussion at the panel meeting. Once a triaged proposal is selected for the rescue, this action will automatically move it from this to the Proposal grading tab, and it will be marked with a circled R.

Tab Triaged proposals lists the proposals that have been triaged, i.e. those that ranked the lowest at the end of the pre-grading. These are also marked with a circled-T. Any non-conflicted panel member can propose to rescue a triaged proposal and the request has to be accepted. Once rescued, thal proposal has to be discussed and graded. Only Panel Chairs or the Panel Scientific Assistant (should the Panel Chair be conflicted) have the Rescue action button. Panel Chairs and/or Scientific Assistants should use the 'Expand all' feature on the left-side list of proposals in order to see who are the conflicted reviewers (thus avoiding that one of the conflicted panel members proposes a rescuing action).

Once a proposal is selected (by clicking on his ID), the information displayed on the right-side of the window includes also the average pre-grade and its associated standard deviation, to possibly idenitify those proposals that have received very discordant evaluations. These may deserve a closer look and discussion at the panel meeting. Once a triaged proposal is selected for the rescue, this action will automatically move it from this to the Proposal grading tab, and it will be marked with a circled R.

For completeness, proposals marked with a circled-RE are Triaged proposals that got automatically rescued by ESO (OPO), because their loose atmospheric conditions make them good filler candidate programmes. However, the automatic rescuing does NOT imply that they are already approved; rather that the panel should discuss and grade them as any other proposal (see plenary presentation from the Q&A session).  

Tab Proposals grading is the one you will be using the most during the Panel meetings, and where the final grading will be performed. During this Pre-meeting activities phase, you can already use it to continue the proposals evaluation or to add futher annotations.  

Tab LPs panel discussion includes all LP in your Panel science category (basically those that you have already pre-graded). This tab is intended for the panel discussion of the Large Proposals - no grading involved, this part has the goal (for the Panel Chair or OPC members part of your Panel) to collect all your views on these proposals.  

[New]  For OPC Members only:

As of OPC112, from this phase onwards, the Panel vs OPC review flows are split in two separate tabs, Panel and OPC, that give you access to a different set of tabs, depending on the review flow.

The Panel part is as described above.

The OPC part now includes the following tabs:    All Large proposals / Not Promoted LPs / LPs voting

Tab All Large proposals includes the complete list of all Large Proposals that were submitted for P112. It has similar funcionalities to the Proposals in my panel tab. 

Tab Not Promoted LPs collects those LPs that have ben preliminarly identified as Not Promoted (based on their pre-grades). Should you feel strong about any of them, you can propose to push it into the group of Promoted LPs by signalling your request to the OPC Chair (via email).

Tab LPs voting will be used during the OPC Meeting, for the discussion and final voting of the Large Proposals.

Panel meetings

Just before the start of the Panel meetings, the PEI interface changes status, from review state = Pre-meeting activities to review state = Proposals grading.

PEI offers you the same series of tabs as the previous review state, i.e.

Proposals in my panel / Conflicts and Impediments (for Panel Chairs only) / My pre-grades / Triaged proposals / Proposals grading / LPs panel discussion  

Please note that all action buttons (i.e., opening a proposal for grading, grading a proposal, marking last minute conflicts and/or impediments, etc) are enabled only at the start of the first meeting of your panel (because any action button has a direct effect on the real database). Until then, you can continue to familiarise with the PEI view(s) to be used during the panel meetings and update your comments on the individual proposals.

Proposals grading

The view in the Proposals grading tab is very similar to the one used to specify your pre-grades. You have the list of all proposals to be graded on the left and, as you select one proposal ID, this will open the right side part where you will specify your final grade and where you can take further annotations during the discussion (highly recommended for Primary reviewers, who will be responsible for finalising the feedback to the proposal PI).

The Panel Chair, with the support of the panel Scientific Assistant, should also check that all members conflicted with the proposal to-be-discussed-next leave the (virtual) room. Conflict information is available in the Conflicts & Impediments tab (as a full Panel overview) but also in the Proposals grading tab, by using the 'Expand all' feature, which will provide the extra information about who is the Primary and who is/are the conflicted reviewer(s), if any.

As soon as the Panel Chair selects the next proposal to be discussed this will automatically highlight (with a light-blue background) the same entry also for the other panel members. However, should reviewers find themselves reading or checking a different proposal, they can quickly go to the one currently under discussion by selecting the ligh-blue background entry in the list on the left or by selecting Go to grading at the top-left of the list.

Ideally, proposals should first be discussed within the panel before the Chair officially opens the grading (action button Grade, visible only to Panel Chairs - and to Scientific Assistant for those proposals with which the Panel Chair is conflicted). Should a proposal include more than just one run, then the panel needs to decide upfront if they want to grade that proposal as a whole or per run (selection can be made on the right side of the Grade opening button). The default choice is 'per proposal', because the vast majority of proposals has only one run.

All reviewers can then specify their chosen grade and submit it. Panel Chairs, on top of submitting their grade, follow also the grading progress of the panel, i.e. they have an overview table that lists all panel reviewers (grading or conflicted) and their grading status (Pending or Submitted). Panel Chairs can also Re-open the grading for a given reviewer, in case of, e.g., a typo. Once all grades are in, the Chairs must Close the grading of that proposal or put it On-hold. The latter option is envisioned for when a panel reviewer has participated to the proposals discussion but has suddenly dropped-out at the time of grading.  

The On-hold flag will be clearly visible in the proposals list on the left, so that the Chairs can re-open the grading of that proposal as soon as the drop-out reviewer is back in the meeting. Needless to say, the panel can continue with its review process, independently of the absence of the drop-out reviewer. In such cases, anyway expected to be very rare events, Panel Chairs will need to mark the missing reviewer as 'impeded' (using the Bulk impediments feature, under the Conflicts and impediments tab) and - if possible - postpone the discussion of the Primary proposals of the missing reviewer.  

For those proposals on which the Panel Chair is conflicted, the Panel Co-Chair will chair the discussion whereas the Scientific Assistant will open and close the grading.

During the grading process, the Grading state of a proposal will change from To be graded to Grading and finally Done (or On-hold).   

Closing the Panel

When a Panel has completed the evaluation and grading of all proposals (i.e. at the very end of your last Panel meeting), then Panel Chairs shall select Close all Panel activities (to be found under Show panel details, from the Current review window). This action will only freeze the grades of the proposals assigned to your panel, but you will still be able to revise/update your comments on any proposal.  

More details about Conflicts and impediments (for Panel Chairs only) 

Under the tab Conflicts and impediments, you can find an overview of all conflicts that were recorded before the start of the panel meetings. These can include both institutional conflicts detected automatically by the proposals distribution algorithm and scientific/personal/other types of conflicts as declared by the individual reviewer, although they are not distinguishable in order to keep anonymization.

Although a remote possibility, reviewers may recognize last-minute conflicts on specific proposals, which need to be properly recorded in PEI. Only you, as the Panel Chair, are allowed to log these conflicts via the Conflicts overview table, by simply selecting the conflict-icon corresponding to a specific pairing of reviewer-proposal and specifying the type of conflict (for statistical purposes) in the pop-up window.

As soon as a new conflict is recorded, all information and actions associated to that specific proposal will be disabled in the PEI views of the conflicted reviewer, who will need to leave the virtual room (the leaving and re-entering a panel meeting virtual room by a conflicted reviewer is taken care of by your panel scientific assistant).

Should a reviewer experience sudden connectivity issues or may be impeded to attend part of a panel meeting, you can easily record the absence (in this case, likely prolonged in time) of that reviewer, via the Bulk impediments feature, available below the Conflicts overview table. As it will be described below, this is a mandatory action in order to be able to close the grading of a proposal and move forward with the review process, should a reviewer lose their connection or face other sudden emergencies.

As a reminder, sudden drop-out reviewers, who have participated to the scientific presentation and discussion of a proposal, have the right and duty to express their grade, whereas if they have missed most - if not all - of the panel discussion they are not entitled to express their grade. In the former case, should the reviewer's drop-out happen just at the end of the scientific panel discussion, the grading of that proposal should then be put On-hold.

Moreover, the Bulk impediments feature allows you to mark one of your panel reviewers as impeded for a longer time (i.e., inability to attend one specific meeting slot) by selecting the proposals that will be discussed - if known in advance - during the reviewer's pre-announced absence.

Scientific evaluations and final feedback to the proposals PIs

As of OPC111, your scientific annotations on the individual proposals have been carried along the entire review process. In other words, comments specified during a given phase have been saved and kept for the next phase. Copy&paste actions are always possible, so that, especially towards the end of the review process, if one has revised the text too much and would like to retrieve the very first version of their notes, one can always go the corresponding PEI tab (e.g., My pre-grades) and select which parts to copy.

The most common scenario is likely the one where you have started taking notes during the pre-grading evaluation phase for your P+S proposals, you may then have logged some further annotations during the pre-meeting activities phase (especially for the Triaged proposals). During this phase, you will also have read-access to the anonymized comments from all members of your Panel.  

At the start of the panel meetings you will find the latest version of your annotations in the Proposal grading tab, where you can continue to update your comments. We recommend, especially for those proposals on which you are the Primary reviewer, to continue taking notes during the scientific discussion, because the final feedback to the PI shall include the panel view and evaluation on that proposal. 

As soon as the panel has completed the grading of all proposals and closes its activities, there remains only one last task for you to attend, i.e. the finalisation of the comments that will be sent to the proposal PIs. You are the ultimate responsible for writing the final feedback for those proposals to which you have been assigned the Primary reviewer role.

For the finalisation of the feedback, all Primary reviewers will have access not only to their personal notes, but also to all the notes written by the members of one's panel, now not anonymized anymore (this is to facilitate asking for clarifications to the author of a given comment, should doubts arise).

As soon as your Panel has declared to have completed all its activities, the PEI review state changes to Post-meeting activities. Primary reviewers can finalise their feedback under the tab, My Panel evaluations as Primary. This is the only tab that allows you to submit the final comments to the Approver (usually your Panel Chair, or co-Chair in case the Chair is conflicted - you will anyway see who the Approver is for each proposal). You can submit your final evaluations once you have completed them all (Submit all button) or one by one (Submit to approver button).   

For Approvers: as soon as PEI is in review state = Post-meeting activities, you will have your own My Panel evaluations as Primary tab where to finalise the feedback for your Primary proposals and also a new tab My Panel feedback approvals. Here, you will find all proposals for which you have been assigned the role of Approver. Panel Chairs are expected to be the Approvers of all their Panel proposals, except those with which they are conficted. For the conflicted proposals (usually very few), a different Approver will have been chosen (see below).  

In this tab, you will be able to easily identify the status of the final feedback of all proposals. At the start, all proposals will be marked with the label 'Not submitted yet'. As soon as a Primary reviewers has submitted one or all of their final evaluations, those proposals will be marked with the label 'Ready for approval'. You will then need to check the content and the wording of the final text submitted by the Primary (also based on your own personal notes from the discussion), revise it as you deem appropriate, and then approve it (via the Approve button). The approval has to be done per proposal. The approved proposal will then be labeled as 'Approved' and the Primary reviewer will be able to see if any change was made to their submitted text.

For the proposals assigned to the Panel Chair as Primary and/or for those proposal on which the chairs are conflicted, the Approver role will be assigned to the Panel co-chair. Should also the Panel co-chair be conflicted on the same proposal(s), another panel member will have to be identified as the Approver. This means that, in principle, any panel member may find the My Panel feedback approvals tab in their PEI, in case they have been identified as Approver of at least one proposal.

For OPC Chair and OPC Members (Large Programmes): the finalisation and approval of the feedback to Large Proposals PIs follows a workflow similar to the one just described, i.e. Primary reviewers will finalise their feedback, submit them to the OPC Chair, who will then have to review, eventually revise and approve them.

As of OPC112, this part of the workflow belongs to the OPC section (for OPC members only, not for the OPC Chair who has only one view).

Special situations

This section is far from being complete, but it aims at gathering tips and actions for special situations that may happen during the review process, especially during the on-line Panel meetings. Some of the information below has already been provided in the above sections.

Wrong grade specified: should a reviewer realize immediately after submitting their grade(s) to have made a mistake (e.g. a typo, for example they wanted to give 1.5, instead they typed 2.5 or viceversa), then the Panel Chair can allow only that reviewer to correct their final grade(s) by selecting the button RE-OPEN, which can be found in the panel (grading progress) table.

Wrong proposal selected for grading: should the wrong proposal be opened for grading, the Panel Chair shall Cancel the grading. This action cancels all grades already submitted.  

Sudden drop-out : should a reviewer suddenly experience issues with their connection, there are different ways to proceed depending on the actual situation:

a. If the reviewer has missed any part of the proposal discussion they shall not grade the proposal (even if they were to re-connect in time for the grading).

The Panel Chair needs first to mark the missing reviewer as “impeded” via the Bulk impediments feature in the Conflicts and impediments tab and then proceed with the grading (otherwise, without notifying the system that one reviewer is missing, they will not be able to Close the grading of that proposal). 

b. If the reviewer has taken part to the entire discussion, and only at time of grading they lose the connection, then they shall grade that proposal when they get back online.

Once the reviewers, still present and not conflicted, have all submitted their grades, the Panel Chair shall select the Hold button (instead of closing the grading), which allows them to close (temporarily) the grading of that proposal and move to the next one. No "Final grade" will be displayed in this case, because the grading of that proposal remains incomplete. Once the drop-out reviewer reconnects again, the Panel Chair can re-open the grading of that proposal (by selecting the Resume button), so that the previously missing reviewer (only) can submit their grade.