Common DFOS tools:
Documentation
dfos = Data Flow Operations System, the common tool set for DFO
*make printable

Common DFO steps and DFOS tools: architecture

1. Daily workflow

DFOS pre_pipeline part part interacting with pipelines part interacting with QC procedures post_pipeline part
non-DFOS     pipeline infrastructure (esorex, CONDOR)
provided by DFS		 QC procedures
(instrument specific)
provided by the QC scientist		   
pipeline recipes (instrument specific)
provided by DFS

Details:

These steps are called per DATE, the middle part also per MODE (CALIB or SCIENCE).

  I. Pre processing
Ia. Automatic (within autoDaily), once per hour 		Ib. When new data available II. AB processing and QC checks      
workflow step create data report CALIB:
create ABs		 create jobs process ABs create QC reports certification
SCIENCE:
create ABs		        
DFOS tool createReport createAB createJob [JOBS]* [JOBS]* certifyProducts
calling ... fitsreport ABbuilder, filterRaw, fitsreport CALIB: processAB, ngasClient
 processQC; qc1Ingest; scoreQC
status flag report_Exist  cal_AB/ sci_AB cal_Queued/ sci_Queued cal_QC/ sci_QC cal_Certif/
sci_Certif
*: executable offline job file, launches ABs using processAB and processQC
  III. Post processing        
workflow
step		 CALIB: move to final
directories		 rename update ABs ingest to NGAS clean up
SCIENCE:   update ABs extract associations
DFOS tool moveProducts [JOBS]* finishNight
calling ...   renameProducts updateAB ingestProducts; dpIngest
SCIENCE: verifyAB		 extractStat; histoMonitor
status flag cal_Finished/
sci_Finished		 cal_Renamed/
sci_Renamed 		cal_Updated/
sci_Updated		 cal_Ingested/
sci_Verified		 finished/
removed
*: executable offline job file, provides ingestion and harvesting

Explanation
DFOS tool level1 (main) tool, called from the command line
calling ... level 2 (helper) tool, called from a level1 tool
status flag status flag to be set in log file
ALL process for all files
CALIB process for CALIB files
SCIENCE process for SCIENCE files

2. Packing workflow

This workflow was supported until October 2011. If you have historical interest, find its description here.

3. Workflow tools

process step calChecker (and telltracker*) auto-download, CALIB processing and QC pre-Imaging workflow PHOENIX workflow for science processing
DFOS tool calChecker autoDaily processPreImg* phoenix*
tools called HdrDownloader, createReport all tools from daily workflow part I up to II (except certification) createAB ... moveProducts, incl. processAB all tools from daily workflow, except for createAB, certifyProducts
status flag hdr_Exist, report_Exist      
*not part of standard process; only if this workflow is supported on the account

4. Monitors and GUIs

process step installation monitor calibration completeness daily workflow monitor scores overview AB product monitor (processing and QC reporting) productExplorer muc Monitor (condor queue on muc blades) historical record ("nautilus")
DFOS tool dfosExplorer calChecker dfoMonitor scoreHC -> scoreManager.html getStatusAB productExplorer mucMonitor histoMonitor/
phoenixMonitor*
calling ... dfosInstall HdrDownloader, createReport            

5. Trending and QC system

process step ingest QC1 parameters score product quality trending documentation of QC procedures maintenance of navigation bars browse plot hide python QC library
DFOS tool   scoreQC trendPlotter qcDocu webNavbar qc1Browser* qc1Plotter* qc1Hide* pyQC**
calling ... qc1Ingest   pet.py**; scoreHC            
* These tools are not installed on the dfo machines but on the archive server.
** set of python procedures

6. Other tools

process step download headers hide files or change file headers query products in NGAS replace product fits files with their headers tools for calSelector rules management
DFOS tool dataLoader hideFrame productExplorer cleanupProducts createCalibMap; calselManager
calling ... HdrDownloader        

DFOS tools: overview

Daily workflow:
pre-processing
  
Trending and Quality Control system
DFOS tool source   DFOS tool source
HdrDownloader DFS   trendPlotter QC
dpQuery DFS   pet.py QC
createReport QC   scoreHC QC
fitsreport DFS   scoreQC QC
filterRaw QC   webNavbar QC
Daily workflow: processing
  qcDocu QC
createAB QC  
qc1Ingest
 DFS
ABbuilder DFS  
qc1Browser
 DFS
createJob QC  
qc1Plotter
 DFS
processAB
 QC  
qc1Hide
 DFS
ngasClient
 QC   pyQC QC
ngamsCClient
 DFS      
hotfly
 DFS  
CalChecker workflow
processQC
 QC   calChecker QC
scoreQC
 QC   tellTracker QC
mucMonitor QC      
certifyProducts QC      
Daily workflow:
post-processing
      
moveProducts QC      
renameProducts
 QC      
ingestProducts
 QC      
dpIngest
 DFS      
updateAB QC      
verifyAB
 QC      
calSelector
 DFS      
calselManager QC      
createCalibMap QC      
finishNight QC      
extractStat
 QC      
cleanupProducts QC      

QC: tool developed and maintained by QC
DFS: tool developed and maintained by DFS

Last update: April 26, 2021 by rhanusch