ESO CASA and Pipeline Meeting - 15~17 May 2019

1. Travel to ESO

• Getting to ESO and hotels in Garching
• For a list of hotels in Munich please see Munich hotels. If you prefer to stay in Munich, we suggest to choose a hotel close to the U6 metro line.
• Visiting the ESO Supernova: programme

2. Participants

• External:
  • Kana Sugimoto (NRAO - NM)
  • Ryan Raba (NRAO - CV),
  • Dirk Muders (MPIFR - Bonn),
  • Stewart Williams (UKRI-STFC - Edinburgh)
• ESO:
  • Vinod Arumugam
  • Andy Biggs,
  • Sandra Castro,
  • Alex Chalevin
  • Maurizio Chavan,
  • Maria Diaz,
  • Enrique Garcia,
  • Justo Gonzalez,
  • Fabrizia Guglielmetti,
  • Liz Humphreys,
  • Alisdair Manning,
  • Luke Maud,
  • Stewart Mclay,
  • Federico Montesino,
  • Tony Mroczkowski,
  • Dirk Petry,
  • Suzanna Randall,
  • Erich Schmid
  • Felix Stoehr,
  • Martin Zwaan
• Remote participants via VC:
  • John Hibbard (NRAO - CV)
  • Remy Indebetouw (NRAO - CV)

3. Video Connection

To connect to a meeting please use one of the following connection methods

Video Conference, use the following address meet.room1@eso.org

Telephone +49 893200 69441

With Web RTC (Google Chrome, Safari, Microsoft Edge or Firefox) https://join.eso.org/invited.sf?secret=bNrpMvJSDcN8MImw_RU4xw&id=320069441

4. Dinner in Munich

Date:

5. Agenda

- Meeting room: Pavo, A.2.01 -

Wednesday, 15 May

• 9:30 - Welcome and Introductions (Sandra/Luke)
  • - ESO Code of Conduct
• 9:45 - Pipeline status today (Kana)
• 10:15 - Overview of the Pipeline and how it works (Stewart)
  • - Pipeline: weblogs, content, how, where, and what connections. Tips and Tricks (Stewart)
• 11:15 - Coffee break
• 11:30 - Pipeline: imaging status and inside view (Dirk M.)
• 12:15 - Lunch
• 13:30 - Pipeline development process and the Integration and Performance team (Ryan/Kana/Sandra)
• 14:00 - CASA and Pipeline: status of parallelisation and performance (Federico)
  • - Pipeline failures; MPI timeouts and what they mean (Federico)
  • - CASA memory usage and monitoring (Enrique)
• 15:15 - Coffee break
• 15:30 - Discussion topic: Group-level Processing, how to make best progress (Felix/all)
• 16:00 - Information topic: TelCal and testing for the upcoming Correlator Upgrade (Justo)

Thursday, 16 May

• 9:30 - CASA status today (Ryan)
  • - CASA forward plan for HPC testing, performance, benchmarking (Ryan)
  • - CASA 6 and CASA next generation infrastructure (Ryan)
• 10:30 - Discussion topic: Ways forward on Virtual-columns+lazy import: how to be able to deliver calibrated MS to all users without having to store them (Felix)
• 11:00 - Coffee break
• 11:15 - Discussion topic: Unique SPW IDs also across OUS: should we make a request to control? (Felix/all)
• 11:45 - Discussion topic: Use of EU cluster machines for CASA/PL development, best processes (Luke/all)
• 12:15 - Lunch
• 13:30 - Pipeline plans for Cycles 7, 8 and beyond (Kana)
• 14:00 - Status of EU PLWG (Luke)
• 14:30 - Summary of decisions from Pipeline F2F in March 2019 (Remy)
• 15:00 - Coffee break
• 15:15 - DARED and ADAPT (Maurizio)
• 16:00 - CLAI - Cluster/ADAPT Interface (Alex)
• 16:15 - Information topic: Flux Service (Alisdair)
• 19:30 Dinner at Paulaner Bräuhaus - See information and map above

Friday, 17 May

• 9:30 - QA2 at the EU ARC (Dirk P.) & Operations Pipeline at the EU ARC (Vinod)
• 10:30 - Discussion topic: Product ingestion problems: can the Pipeline tasks be used for manual data-reduction? (Felix/all)
• 11:00 - Coffee break
• 11:15 - Discussion of EU development possibilities for better user products/choices (Luke)
• 12:00 - Lunch
• 13:00 - AQUA (Alex and Suzanna)
• 13:30 - APA - AQUA Pipeline Agent and its Future (Fabrizia)
• 14:00 - Discussion topic: ESO QA changes for Cycle 7 and beyond with the introduction of ADAPT
• 14:30 - Information topic: Antenna positions service (Federico)
• 14:45 - Pipeline Scores (John H. via VC)
• 15:15 - Coffee break
• 15:30 - Open discussion (all)
• End of meeting

6. Summary of discussions

7. Action items

8. Input for Topics to Discuss

• Pipeline
  • Pipeline: status and plans for Cycles 7, 8 and beyond (Kana)
  • An overview of flagging in pipeline stages (TBD)
  • Overview of the Pipeline and how it works (Stewart W.)
  • Pipeline scores (TBD)
  • Pipeline: imaging (Dirk M.)
  • Pipeline: weblogs, content, how, where, and what connections. Tips and Tricks (Stewart W.)
  • Pipeline failures; MPI timeouts and what they mean (Federico)
  • Pipeline development process and the Integration and Performance team (Ryan, Sandra)
• CASA
  • CASA: status and plans (Ryan)
  • CASA 6, CASA next generation infrastructure (Ryan)
  • CASA forward plan for HPC testing, performance, benchmarking (Ryan)
  • CASA and Pipeline: status of parallelisation and performance (Federico/Sandra)
  • Ways forward on Virtual-columns+lazy import: how to be able to deliver calibrated MS to all users without having to store them
• ESO QA2 and ingestion
  • QA2 overview (Dirk P.)
  • QA2 manual processing
    • Python 3 migration plans
    • Parallelisation of manual runs
  • QA.... (Vinod)
  • QA - how the ARC nodes work (TBD)
  • ESO QA changes for Cycle 7 and beyond with the introduction of ADAPT
  • Product ingestion problems: Can the PL tasks be used for manual data-reduction? (Felix/all)
• Others
  • Summary of decisions from Pipeline F2F in March 2019
  • Status of EU PLWG (Luke)
  • Unique SPW IDs also across OUS: should we make a request to control? (Felix/all)
  • Group-level Processing, how to make best progress (Felix/all)
  • Discussion of EU development possibilities for better user products/choices
  • Use of EU cluster machines for CASA/PL development, best processes
• Software specific
  • AQUA (Alex and Suzanna)
  • APA (AQUA Pipeline Agent) (Fabrizia)
  • DARED and ADAPT (Maurizio)
  • Flux Service (Alisdair)
  • Antenna positions service (Federico)

-- Sandra Castro - 2019-03-22

• Interferometry-Imaging-Status-EU-ARC-May-2019.pdf: Interferometry Imaging Status

• Pipeline_Overview.pdf: Pipeline Overview

• eso_casa_pl_201905_parallelization_performance.pdf: CASA Parallelisation and Performance