On April 24th at around 13 UTC, the day prior to the ALMA Cycle 11 CfP deadline, proposals requiring Bands 7 and higher started experiencing inconsistent observing time estimates. This was due to a temporary and intermittent connectivity issue between the online calibrator catalogue and the Observing Tool (OT), required to search for high frequency calibrators. When the calibrator catalogue could not be accessed, band-to-band calibration was assigned, resulting in an increase in the observing time estimates. In some cases, this prevented validation and submission if the observing time increased above the Large Program threshold. As announced on the Science Portal, initially the problem occurred intermittently and appeared resolved after a short time. Unfortunately, the issue recurred during the hours before the scheduled proposal submission deadline (April 25 at 15 UTC). A second issue related to user authentication also occurred close to the deadline, sometimes preventing users from logging in to submit their proposals. Both issues persisted until the deadline despite steps taken by ALMA to restart specific services. The issues have now been identified as resulting from a performance degradation of the servers at the JAO, the cause of which is still under active investigation.  

To assist users who were impacted by these technical issues, ALMA took the following steps.  

1. A second announcement was placed on the ALMA Science Portal instructing users to submit Helpdesk tickets prior to the ALMA deadline if they continued to be impacted by the issues. ALMA assisted all users who submitted Helpdesk tickets by the deadline and were unable to submit or resubmit their proposals due to these issues. Exceptionally, due to the issues occurring so close to the deadline, ALMA also helped users who submitted Helpdesk tickets immediately after the deadline. Observatory staff assisted users by submitting proposals (.aot files) to the ALMA Archive on behalf of these users. Only proposals affected by these specific issues were submitted in this manner. 
2. All cases in which the proposal was submitted by the PI but with incorrect observing time estimates due to the calibrator catalogue issue will be fixed by the Observatory before proposals are sent to review. Reviewers will also be instructed to ignore any discrepancies in observing time estimates for Bands 7 and above.  

It was decided to assist users in this way instead of extending the proposal submission deadline since the technical issues were ongoing and there was no estimate for when normal service could be restored.  

ALMA sincerely regrets any inconvenience or impact experienced by our users and is taking measures to identify the root cause of the incident to ensure this will not affect future ALMA CfPs. If you have any questions, or have comments or concerns, please contact your local ALMA Regional Centre via the ALMA Helpdesk at https://help.almascience.org/