make printable	new:	see also:
	v2.12: - dropping call of checkConsist; config key PRINTER dropped	processQC, moveProducts; scoreQC; rawdisp2reference getObInfo for OB related information
	v2.12.5: - calling rawdisp2reference at the end	databases DFO_db (raw_comments | prod_comments) (insert/update); qc1_score (read) dfos tools qc1Ingest to insert new entries into raw_comments and/or prod_comments; calls user-provided tool for displaying QC report; scoreHC to refresh score reports for HC monitor output certified or rejected products; uses scoreQC output upload/download up: comments about raw or product files
topics: description | auto-certification | jumping | workflow | incremental mode | comments | configuration | hints

certifyProducts

Description

This tool is called to certify pipeline products. It is, like processQC, a wrapper around an instrument-specific and user-provided procedure. The tool scans all directories $DFS_PRODUCT/<RAW_TYPE>/<DATE>, as configured per MODE. These have been created by createAB, and filled by processAB and processQC.

The tool assumes that the user has some procedure (defined in PROC_CALL) which is able to load the graphical information which has been created during the execution of processQC. It then loops over all RAW_TYPEs specified, and on all ABs for that RAW_TYPE. Thereby the tool also checks for unsuccessful AB executions.

Auto-certification. The tool offers auto-certification. This is a key concept towards combining objective and careful quality control with the efficiency requirement of mass production. This mechanism works with ABs being scored by scoreQC. The idea is to "delegate" certification to the scoring tool. This delegation requires a stable and complete configuration of scoring.

Per RAW_TYPE, the user can configure a certain certification mode:

certification mode	action
ALL	offer all ABs for certification
NONZERO	offer for review only ABs having non-zero scores, or no scores, or the RAW_TYPES specified in configuration key CERTIF_RAW
RANDOM	like NONZERO, plus offer N% ABs with zero scores, and at least one AB

There are two "extreme" situations:

- no scoring implemented, or scoring incomplete --> ALL (certification needs to be done by QC scientist). This can obviously be only a transient solution (for a new instrument or pipeline) or a niche solution (for rarely used modes without proper pipeline support).

- scoring fully implemented and complete --> NONZERO or RANDOM.

Strongly recommended is RANDOM: this is a choice which still offers the full QC report for a subset of ABs. This is a trade-off between the competing goals of review completeness and "review fatigue" caused by the need of reviewing many similar reports. Depending on the situation, the user may choose to have offered between 5% and 50% of all zero-score QC reports. Recommended value is 20% for instruments running in stable operations.

For new instruments ALL is the better choice.

[Technically, the system variable $RANDOM is used to derive the AB selection. Naturally, both the selected ABs, and their number, will differ from call to call. Combining RANDOM and jumping (see below), you may therefore select a certain 20% AB set, and immediately afterwards another set.]

All operational QC accounts should have a fully implemented scoring system. All operational QC accounts should run the RANDOM certification mode.

CERTIF_MODE is configurable. Exceptions from that standard choice can be configured for certain RAW_TYPEs. CERTIF_FRAC is applied for CERTIF_MODE=RANDOM only.

After certification, the box CERTIF in the AB monitor is filled as follows:

CERTIF
OK	certified
REJ	rejected
AUTO	auto-certified

If a product comment has been entered (see here), it will also be displayed in the CERTIF box.

In AUTO certification mode, the tool calls getStatusAB with option -e (quick edit) to update only the CERTIF column in the AB monitor.

Jumping. The tool offers the option to jump forth and back within the certification loops. Once the user has jumped, there is a security mechanism to avoid unintentional exit from the tool (since products may not yet been reviewed). The user needs to confirm the exit. The tool then either exits on confirmation, or re-starts the certification loop, with the option for the user to jump to any product not yet certified.

Auto-certification and jumping can be combined: in auto-certification mode, the user can jump to any RAW_TYPE, and any AB. From the selection on, the certification mode for the selected RAW_TYPE is evaluated again.

In NONZERO certification mode, a jump is offered at the very end of the certification workflow. If chosen, the tool then starts again, in forced mode ALL, to offer full choices for selection and jumping.

Workflow options. The tool has the following configurable options:

• enable detailed investigation (e.g. using rtd)
• ask for printing
• provide a warning, and (if configured in DISPLAY_MODE) display the score HTML file if a non-zero score is given (by scoreQC)

The tool pops up the text file with OB start times and comments (extracted by the getObInfo tool).

Unless it finds an AB selected for automatic certification, the tool asks for certification:

Certified (y/yc | n | j[ump]/e[xit]) (y)

Entering "y" (the default) results in certification, without further questions. "yc" brings you to a small menue where you can enter raw and/or product file comments (see here).

The provision of a product comment is enforced under the following conditions:

• the AB has scored red
• the AB has no score
• the AB failed
• the AB is rejected
• all of these conditions are evaluated only if no comment exists already.

The reason for enforcing a comment is that all these cases are considered exceptional and potentially caused by an issue. Therefore both the rejection or certification should be accompanied by some information provided as a comment.

When enforcing a comment, the tool checks for already existing comments (if one is found, the tool just continues). Comments could already exist because 'certifyProcust Light' has been run, or because the bulk mode for comment ingestion has been used.

Please keep in mind to always provide significant comments (not "AB scores red" or "parameter XY above threshold", but e.g. "flat-field lamp flux fixed the next day" or "RON ok, statistical outlier").

Typing "n" means REJECTION, and the tool will enter the rejection workflow:

• offer the option to enter raw (optional) and product (mandatory) file comments (about reason of rejection)
• display other ABs affected by this rejection (always)
• call configured $DELETE_CALL (optional)
• ask about removing the product(s) from the container of virtual calibs ($DFO_CAL_DIR/VCAL) (optional)
• hide the associated QC1 parameters in the QC1 database (table taken from the tlog file) (optional)
• find the configured local trending table (if any) and hide the corresponding entry (coupled to previous step)
• delete the product file(s) (always)
• call of hideFrame (actually it provides the list of raw files that you can then enter into the hideFrame mask; optional)
• call the BADQUAL workflow of calChecker (the helper script $DFO_GUI_DIR/edit_BADQUAL.esh is called; optional)
• same for CAL4CAL

Removing a product name from $DFO_CAL_DIR/VCAL means it is not available any longer for other associations. 'y' is the right answer if there is a problem with the raw file (e.g. no light found). 'n' may be the right answer if the raw file is ok but the pipeline recipe failed (because e.g. of inappropriate parameter choice, or a bad algorithm etc.). In such case you may want to deliver (pack) the raw file with a science file, and this is only possible if the virtual product name is found in $DFO_CAL_DIR/VCAL.

A special case is that the pipeline has failed and no products have been created. Then, the tool enters the same workflow as under rejection. At the end the tool asks:

Continue with next AB (y)/j[ump]/e[xit]) (y)

The tool writes into a trending log ('tlog') which is created, per AB, in $DFO_AB_DIR. This is a mechanism to store a certification flag. It is used by the TQS tools scoreQC and scoreHC.

There are also optional calls of user-provided procedures for the following cases:

• no product found ('FAILURE_CALL'); here you could e.g. check why no product was created
• product rejected ('DELETE_CALL'); here you could, in addition to the offered standard functions, investigate and probably delete the parent raw files

Preliminary certification of calibrations in incremental mode (certifyProducts light). If DATE = TODAY, CALIB data are processed by autoDaily incrementally. Since the flow of CALIB data is not yet finished, CALIB data from TODAY should normally not be certified and moved (instead, you wait until the next day). However, there might be cases when a provisional certification ("certifyProducts light") makes sense, in particular if comments are provided as feedback to Paranal operations. Then, you can use the blue button on the dfoMonitor which calls the workflow of the light mode of certifyProducts. Comments and certification flags are displayed on the AB monitor, which is also exported to the qcweb site. You can stop anywhere in the workflow, and jump into it again later. No flag is written into DFO_STATUS which means that officially you have not yet certified this date and need to visit this date again later. Depending on the status of your certification, you may then only need to enter and exit again, in order to proceed with moveProducts.

Note that for certifyProducts light, in case of rejection, the fits files are not deleted, contrary to the normal situation: if they were deleted, then autoDaily would create them again.

In unusual operational situations you may want to call preliminary certification for dates older than TODAY. There is no button foreseen on the dfoMonitor, but you can find the corresponding launcher scripts in $DFO_GUI_DIR until the corresponding date is finished. Look out for 'launch_certifyPLight_<date>_CALIB.esh' and call it on the command line.

File comments. After options 'n' and 'yc' you can enter comments, about the parent raw files and/or the products. In either case the workflow is as follows:

Entering and editing comments:

• You have two options to enter comments:
  - incrementally (AB by AB), during the workflow
  - in bulk mode, using the utility tool enterCommentsBulk
• the incremental mode is the normal case if you want to enter a comment only occasionally
• the bulk mode option makes sense when you have comments for many files and do not want to enter them one by one; all files of the current RAW_TYPE are found by the tool and get listed in the comment file (see below). You then edit the comment file as you wish and return to certification. Those entries which you left empty will be deleted or ignored.
• no matter how you entered the comments, they get stored in a text file $DFO_LST_DIR/RAWCOM/raw_comment_<date>.txt or $DFO_LST_DIR/PRODCOM/prod_comment_<date>.txt, resp.) but not stored in the database immediately
• thereby you have a chance to modify these text files anytime later as long as you still execute certifyProducts
• If there is no product, you can nevertheless enter a raw comment or a prod comment (which is then just a comment about the processing failure). The prod comment will be displayed on the AB monitor.
  
• you do not need to remember how to call enterCommentsBulk; the tool certifyProducts offers the proper call just in time and ready for copy and paste.
  
  

  enterCommentsBulk	certifyProducts
  	during execution: enter comments case by case	↓
  if called: enter comments in bulk mode
  	ingest comments
  	moveProducts

  Scheme describing comment edits in certifyProducts

Ingesting comments, updating score reports:

• at the end of the workflow, the content is ingested (after another confirmation) into the DFO databases raw_comments or prod_comments
• if there are product comments, the tlog files of the corresponding product files are scanned to detect associated HC reports. If any is found, their score reports need to be updated in order to include the new product comments. This is done by calling a job file with the corresponding 'scoreHC -r <report>' calls. The job file executes in the background (i.e. you can proceed with the next step), its output remains visible in a pop-up xterm. The xterm is closed after the job file is finished.

Display of comments:

• comments about raw files are displayed in the data report, this is why you are asked if createReport should be called (you may want to skip this for performance reasons, it is done ultimately upon finishNight)
• comments about products are displayed on the AB monitor in the CERTIF column

Product comments are also stored per AB in a text file called $DFO_LOG_DIR/<date>/<ab>.cmt. This text file is found by merge2Fits and included in the LOGS ancillary fits file.

Output

How to use

Type certifyProducts -h for on-line help, certifyProducts -v for the version number, and

certifyProducts -m CALIB -d 2024-07-12

to certify all pipeline products for mode CALIB on that date,

certifyProducts -m CALIB -d 2024-07-12 -L

for the light option (no flag entry in DFO_STATUS, not to be called from the command line).

Installation

Use dfosExplorer or dfosInstall.

The MIDAS procedure automode.prg is attached as an example only. It is *not* designed as a common tool. Your version is expected under $DFO_PROC_DIR.

Configuration file

Status information

The tool writes the DFO status cal_Certif.

Workflow description

certifyProducts -m CALIB -d <DATE>:

1. Read all RAW_TYPEs from config file for selected MODE

2. Loop over all $DFS_PRODUCT/<RAW_TYPE>/<DATE>; use AB_list as reference for expected products

2.1 pop up OB comments
2.2 check if subdir <RAW_TYPE> exists; if not: skip
2.3 check for configured CERTIF_MODE, CERTIF_FRAC and CERTIF_RAW entries; decide which ABs to review
2.4 check if product files exists

2.4.1 No: ask if raw comment is to be entered; enforced to enter "product comment" (actually a comment about pipeline failure)
2.4.2 call optional FAILURE_CALL;
2.4.3 offer to continue with next AB, jump, or exit

2.5 product files exist: call the configured procedure (PROC_CALL) to check the products
2.6 if configured: ask if detailed tool should be called (DETAILED_CALL)
2.7 certification: ask "QC passed for product (y/yc/n/j/e)"
2.8 certification=yc: ask if comments about raw or product files should be added; raw comments are always optional;
product comments are mandatory under the following conditions:

• red score
• product rejected

if yes, or if enforced: insert into file raw_comment or prod_comment; offer to call enterCommentsBulk; products certified; continue with next AB or next RAW_TYPE

2.9 if answer is 'y': products certified; continue with next product/AB or next RAW_TYPE
2.10 if answer is 'n': products rejected

2.10.1 ask if comment is to be inserted in raw_comments (optional) and prod_comments (mandatory)
2.10.2 display other ABs from the same day which use any of these products; you may decide to review or delete those products as well
2.10.3 if configured: call DELETE_CALL, a user-provided procedure to e.g. investigate or delete the parent raw files (mostly obsolete with XXLight!)
2.10.4 if products have been deleted: ask if product name should also be deleted from $DFO_CAL_DIR/VCAL (association pool of virtual products)
2.10.5 ask if QC1 parameters should be hidden; if yes, and if configured, delete entries in local $DFO_TREND_DIR/<TREND_TABLE>
2.10.6 delete all products having the same root name
2.10.7 provide input for hideFrame
2.10.8 ask to call BADQUAL workflow for calChecker, and provide info for it

2.11 if answer is 'j[ump]':

2.11.1 a list of available RAW_TYPEs is displayed, the current one is highlighted; select RAW_TYPE (or hit return to stay in current one)
2.11.2 a list of available ABs is displayed, with setup information; select AB (or hit return to select the top one)
2.11.3 after selection, the tool continues with step 2.2 (AB selection), until all products are reviewed, or another jump is chosen
2.11.4 once the user has jumped, there is a security mechanism to avoid unintentional exit from the tool (since products may not yet been reviewed)

2.12 if answer is 'e[xit]': exit after confirmation

3. set DFO status to 'cal_Certif' or 'sci_Certif' (unless option -L is set)

4. Offer one last chance to jump (if selected, tool is called again, and you can jump at the next certification offer; this may be useful in AUTO mode, or if you want to add another comment)

5. Call 'rawdisp2reference' and check for new raw_type/setup combinations not yet existing as references 

6. If comments exist (raw_comments or prod_comments), ingest them all in the DFO database (tables raw_comments or prod_comments).

7. If comments exist, check if HC reports are affected. If so, call 'scoreHC -r <report>' for each of them.

You can jump forward or backward. You can e.g. focus on all files of a specific RAW_TYPE, or all files of a selected setup. This becomes useful if you have repeatedly executed a set of ABs and want to focus on certain products. After jumping, the tool always continues with the configured AB selection from the selected AB on. By opting repeatedly for 'j', you can jump forward and backward.

Setup information is read from the AB_list and is always displayed together with the AB name.

You can browse or hide ingested comments here.

Hints.

• Nightlogs. Links to the nightlogs are available on the AB monitor and on the data report (under 'NLT'). An extraction with the OBs and comments for the $DFO_INSTRUMENT is available in the pop-up xterm.
• AUTO certification. You want to review a specific AB (or enter a comment) which has been skipped in AUTO mode: use 'j' at the next possible step (could be at the end) and jump to that AB.
• AB rejection. You want to enter a note about a bad raw file (e.g. "file was underexposed") or a product file (e.g. "rejected because recipe went wrong"): if you enter 'n' you will always be prompted for a comment, so you don't need to specify this.
• No product found. You need to enter a comment (like "recipe failed") which is displayed on the AB product monitor.
• Edit or hide a comment. For hiding, use the usual qc1Hide interface. Go to this page and find the link for hiding in the column "interface". For editing, jump back to the AB and go through the certification workflow again. Make sure to replace the old entry in the comment file by the new one, and re-submit. Note that the database entries are updated (as you can check under the same link). To update the product comment on the AB monitor, call 'scoreQC -a <ab>'. To update the raw comment on the data report, call 'createReport -d <date>'.
• Want to enter a file comment but left certifyProducts already? You just start certifyProducts again, enter the comment, choose option 'e' and ingest all comments.
• Please keep in mind to always provide significant comments (not "AB scores red" or "parameter XY above threshold", but e.g. "flat-field lamp flux fixed the next day" or "RON ok, statistical outlier".)

• The option -L (light) is for use from the dfoMonitor only, in incremental processing mode.
• In unusual operational situations you may want to call preliminary certification for dates older than TODAY. Find the corresponding launcher scripts in $DFO_GUI_DIR until the corresponding date is finished. Look out for 'launch_certifyPLight_<date>_CALIB.esh' and call it on the command line.