Common DFOS tools:
|
dfos = Data Flow Operations System, the common tool set for DFO |
The automatic dfos system ensures that
Three core parts of dfos tools establish the automatic dfos system:
The following table gives an overview how these components work together (all tools work on latest date).
component |
|
|
|
||
main step | tool | ||||
reporting | hdr download | HdrDownloader |
|
|
|
data report | createReport+ |
|
|
|
|
calibration completeness | calChecker |
|
|
||
AB creation and processing | AB creation | createAB | |
||
processing | processAB/QC | |
|||
scoring | scoreQC | |
|||
HC monitor | affected plots | trendPlotter/ JOBS_HEALTH |
|
||
all plots | trendPlotter/ JOBS_TREND |
|
|||
update frequency:
|
|
|
|
||
multiple instances:
|
killer! | younger one killed*** | not an issue**** | ||
* daytime/nighttime |
The following table details the autoDaily tasks.
1. PREpare | ||
autoDaily: download headers | dataLoader | part of autoDaily |
ngasMonitor | ngasMonitor | detect new data |
no new data found: exit; new data found: | ||
create report | createReport -d 2020-11-15 | refresh data report |
create ABs: CALIB | createAB -m CALIB -d 2020-11-15 | |
filter data
|
filterRaw -d 2020-11-15 -m SCAN | [done within createAB ] |
Next, the PROCessing part of the workflow starts, for mode CALIB:
2. PROCessing (CALIB) | ||
create processing jobs | createJob -m CALIB -d 2020-11-15 | pipeline jobs (ABs) are written into a job file, for off-line processing |
execute the job file | [execAB call in JOBS_AUTO] | calls: |
execute QC jobs | [execQC call in JOBS_AUTO] |
included in job file |
update HC monitor | [execHC call in JOBS_AUTO] | included in job file |
Last update: April 26, 2021 by rhanusch |