Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
- how to ingest general calibration products
  - v1.8: PAR scheme decommissioned (obsolete)
[ used databases ] databases none
[ used dfos tools ] dfos tools ingestProducts; cdbIngest, cdbDelete; fitsreport; called by moveProducts
[ output used by ] output executable file rn_files
[ upload/download ] upload/download none

renameProducts

Description

This tool renames FITS product files to a different naming scheme. Presently supported are:

CALIBNAME (mode: CALIB or GEN) | USERSCINAME (mode: SCIENCE)

CALIBNAME is the standard QC name for calibration products, to be inserted into the mcalib archive and to be delivered to the packages.

USERSCINAME is the naming scheme for science products. It is used within the phoenix environment.

This tool uses fitsreport to deliver the set of fits key values which is then further massaged to create the file names.

Plugins. The optional plugin $SPECIAL_PLUGIN can be configured and, if provided under $DFO_BIN_DIR, is called after final creation of rn_file but before execution. You could use this plugin e.g. to check for filename length or other things. The $MODE variable is exported by renameProducts and is available within the pgi.

The optional plugin $PRE_PLUGIN can also be configured. It is called after calling fitsreport and creating $TMP_DIR/results. You can use this plugin to modify the content of that file, before it is used by renameProducts to create the new file names. This may be useful if the key content is strange (e.g. DET.ID has two components in FORS1 separated by a blank).

GEN mode. This mode is very similar to CALIB but renames files under $GEN_CALDIR (as defined in config.createAB) and is used for the static calibration products which are not necessarily pipeline products.

Output

The output comes as a text file called rn_files, under the same directory as the FITS files (the renaming file). It is an executable shell script which can move the files from one scheme to the other scheme. Execution of the renaming file is under responsability of the user. The tool does not rename by itself.

How to use

Type renameProducts -h for on-line help, renameProducts -v for the version number.

renameProducts -m CALIB -d 2024-12-12

will produce a renaming file under $DFO_CAL_DIR/2024-12-12. For modes CALIB and SCIENCE, the tool scans the FITS files under $DFO_CAL_DIR/<date> or $DFO_SCI_DIR/<date>.

In mode GEN, all FITS files under $GEN_CALDIR are scanned:

renameProducts -m GEN

Installation

Use dfosExplorer, or type dfosInstall -t renameProducts .

Configuration file

config.renameProducts defines:

Section 1: general
CHECK_YN YES | NO check for multiple new file names in rn_file (these files will be overwritten upon execution, so you may want to be warned)

PRE_PLUGIN

pgi_pre_renameProducts optional plugin, called right after fitsreport and before creating the new names

SPECIAL_PLUGIN

pgi_renameProducts optional plugin, called after final creation of rn_file but before execution (you could use this plugin e.g. to check for filename length or other things; the $MODE variable is exported and available within the pgi)
Section 2.1: Specific parameters (defining the syntax of the new names)
More documentation can be found in the config file. The syntax is that of fitsreport for unformatted reports, with some more information behind the '#'.
SPECIFIC_ALL set of parameters for fitsreport; general part first part has general parameters (like Report mode etc.); leave as is
    second part : three parameters which are always needed; leave as is
SPECIFIC_CALIB set of parameters for fitsreport; CALIB part user-defined specifications; add SEQORD string and shell code pieces as examplified
SPECIFIC_GEN set of parameters for fitsreport; GEN part user-defined specifications; add SEQORD string and shell code pieces as examplified
SPECIFIC_SCIENCE set of parameters for fitsreport; SCIENCE part user-defined specifications; add SEQORD string and shell code pieces as examplified.
Section 2.2: 4 letter codes per pro.catg, for CALIB mode
CODE MBIA etc. define per pro.catg the new 4 letter code; these must be unique per instrument; more rules about 4 letter codes in the config file
Section 2.3: definition of the syntax of the new names
More documentation can be found in the config file.
CASE   define per RAW_TYPE a syntax case (which is just the name of a rule specified under 2.4)
Section 2.4
PARSTRING   definition of the rules defined in 2.3 (use shell code pieces)
Section 3
VERSION   map to assign version character (the first instance gets 'A' etc.)

NOTE: Since the tool uses fitsreport to create the key content table, the 'SPECIFIC' part of the config file mimics the syntax of fitsreport config files, with additional pieces describing the optional shell code to modify the key content. The advantage of that approach is to have the full functionality of fitsreport available, with e.g. multiple key definitions, and support and delimiter strings (helping to make key content unique).

NOTE: In the phoenix environment, and there only if MCALIBs are created, a non-standard config file can be used via the option -c <config-file>. This mechanism is used by the PHOENIX workflow tool to maintain versioning for the product names, to match the historical evolution of the product names. This is needed for MCALIB names only, since it might be desirable to keep the original naming shceme. The names of the various config files, and their versioning, are maintained in config.phoenix. This feature is not needed for the IDP production with phoenix, since the IDPs always have standardized product names.

Workflow description (for CALIB and GEN)

1. go to $DFO_CAL_DIR/<date> (CALIB); $GEN_CALDIR (e.g. $DFO_CAL_DIR/gen) (GEN)
2. for all product files (${DFO_FILE_NAME}*fits) create new name:

2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg; map into 4letter mcalib code in config file (mcal_code)
2.3 use ${DFO_OFFSET}, read mjd-obs; construct DATE (anything between [DATE]T12+offset … [DATEplus1]T12+offset is called DATE); NOTE: contrary to all other cases, DATE here has the format yymmdd!
2.4 CALIB: fill placeholder for version (e.g. "VERSION"); GEN: fill "A" for VERSION
2.5 read config file for setup key, support strings, SPECSTRING (pieces of shell code read from the config file), separators
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 if configured, execute $PRE_PLUGIN

2.8 read extension from original file
2.9 construct new name:
<ins_code>_<mcal_code>_<DATE>_VERSION_<setup_code>.<ext>
Examples:
GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits (CALIB)
GI_GCAT_021230A_line_catalog.fits (GEN)

2.10 check for filename length less than 68 chars; exit if longer name constructed: will lead to problems with database and tools

3. CALIB only: do versioning

3.1 read pro.catg; loop on pro.catg:
3.2 check per pro.catg in config file if versioning is required
3.3 if NO: loop on setup per pro.catg, write for all files 'A' instead of 'VERSION'
3.4 if YES: loop on setup per pro.catg:

3.4.1 sort by mjd-obs
3.4.2 give first one 'A', second one 'B' etc, up to 26 = Z
Example: GI_PFEX_040414B_Medusa1_L437.2nm_o2.fits (CALIB)
The tool has found two cases of files with GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits, this is the second one.

4. renaming

4.1 read CHECK_YN from config file
4.2
write per file "mv OLDNAME NEWNAME" into $DFO_CAL_DIR/<date>/rn_files
4.3 add line to set status flag to 'cal_Renamed'
4.4 if configured, execute $SPECIAL_PLUGIN
4.5 make rn_files executable but don't execute; if CHECK_YN=YES, warn the user if multiple OLDNAMEs end in a single NEWNAME
4.6 finish

Note: The procedure 4.5 is thought as a safety measure. Although in daily operations execute=yes will be the standard case, the authority given to the tool to 'implicitly delete' is potentially dangerous. By setting the additional parameter 'permission', the user keeps this under his control.

Workflow description for SCIENCE

1. go to $DFO_SCI_DIR/<date>

2. for all product files (${DFO_FILE_NAME}*fits) create new name:

2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg, map into sci_code
2.3 read obs.id
2.4 read PIPEFILE, use date_time part up to HH:MM:SS (datetime)
2.5 read config file for setup key, support strings, SPECSTRING
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 read extension from original file
2.8 construct new name: <ins_code>_<sci_code>_<obs_id>_<datetime>_<setup_code>.<ext>
Example: GI_SIFU_125427_2004-04-05T23:12:23 _Medusa1_L437.2nm_o2.fits

3. renaming

3.1 write per file "mv OLDNAME NEWNAME" into $DFO_SCI_DIR/<date>/rn_files
3.2 add line to set status flag to 'sci_Renamed'
3.3 if configured, execute $SPECIAL_PLUGIN
3.4 make rn_files executable but don't execute
3.5 finish

NOTE: contrary to CALIB, multiple files with the same name cannot occur in SCIENCE mode (unless the setup string is not properly defined).

Operational aspects


Last update: April 26, 2021 by rhanusch