Common DFOS tools:
|
dfos = Data Flow Operations System, the common tool set for DFO |
make printable | new: | see also: - how to ingest general calibration products |
||||||||
- v1.8: PAR scheme decommissioned (obsolete) |
|
This tool renames FITS product files to a different naming scheme. Presently supported are:
CALIBNAME (mode: CALIB or GEN) | USERSCINAME (mode: SCIENCE)
CALIBNAME is the standard QC name for calibration products, to be inserted into the mcalib archive and to be delivered to the packages.
USERSCINAME is the naming scheme for science products. It is used within the phoenix environment.
This tool uses fitsreport to deliver the set of fits key values which is then further massaged to create the file names.
Plugins. The optional plugin $SPECIAL_PLUGIN can be configured and, if provided under $DFO_BIN_DIR, is called after final creation of rn_file but before execution. You could use this plugin e.g. to check for filename length or other things. The $MODE variable is exported by renameProducts and is available within the pgi.
The optional plugin $PRE_PLUGIN can also be configured. It is called after calling fitsreport and creating $TMP_DIR/results. You can use this plugin to modify the content of that file, before it is used by renameProducts to create the new file names. This may be useful if the key content is strange (e.g. DET.ID has two components in FORS1 separated by a blank).
GEN mode. This mode is very similar to CALIB but renames files under $GEN_CALDIR (as defined in config.createAB) and is used for the static calibration products which are not necessarily pipeline products.
The output comes as a text file called rn_files, under
the same directory as the FITS files (the renaming file). It is an executable
shell script which can move the files from one scheme to the other scheme. Execution
of the renaming file is under responsability of the user. The tool does not
rename by itself.
Type renameProducts -h for on-line help, renameProducts -v for the version number.
renameProducts -m CALIB -d 2024-12-12
will produce a renaming file under $DFO_CAL_DIR/2024-12-12. For modes CALIB and SCIENCE, the tool scans the FITS files under $DFO_CAL_DIR/<date> or $DFO_SCI_DIR/<date>.
In mode GEN, all FITS files under $GEN_CALDIR are scanned:
renameProducts -m GEN
Use dfosExplorer, or type dfosInstall -t renameProducts .
config.renameProducts defines:
Section 1: general | ||
CHECK_YN | YES | NO | check for multiple new file names in rn_file (these files will be overwritten upon execution, so you may want to be warned) |
PRE_PLUGIN |
pgi_pre_renameProducts | optional plugin, called right after fitsreport and before creating the new names |
SPECIAL_PLUGIN |
pgi_renameProducts | optional plugin, called after final creation of rn_file but before execution (you could use this plugin e.g. to check for filename length or other things; the $MODE variable is exported and available within the pgi) |
Section 2.1: Specific parameters (defining the syntax of
the new names) More documentation can be found in the config file. The syntax is that of fitsreport for unformatted reports, with some more information behind the '#'. |
||
SPECIFIC_ALL | set of parameters for fitsreport; general part | first part has general parameters (like Report mode etc.); leave as is |
second part : three parameters which are always needed; leave as is | ||
SPECIFIC_CALIB | set of parameters for fitsreport; CALIB part | user-defined specifications; add SEQORD string and shell code pieces as examplified |
SPECIFIC_GEN | set of parameters for fitsreport; GEN part | user-defined specifications; add SEQORD string and shell code pieces as examplified |
SPECIFIC_SCIENCE | set of parameters for fitsreport; SCIENCE part | user-defined specifications; add SEQORD string and shell code pieces as examplified. |
Section 2.2: 4 letter codes per pro.catg, for CALIB mode | ||
CODE | MBIA etc. | define per pro.catg the new 4 letter code; these must be unique per instrument; more rules about 4 letter codes in the config file |
Section 2.3: definition of the syntax of the new names More documentation can be found in the config file. |
||
CASE | define per RAW_TYPE a syntax case (which is just the name of a rule specified under 2.4) | |
Section 2.4 | ||
PARSTRING | definition of the rules defined in 2.3 (use shell code pieces) | |
Section 3 | ||
VERSION | map to assign version character (the first instance gets 'A' etc.) |
NOTE: Since the tool uses fitsreport to create the key content table, the 'SPECIFIC' part of the config file mimics the syntax of fitsreport config files, with additional pieces describing the optional shell code to modify the key content. The advantage of that approach is to have the full functionality of fitsreport available, with e.g. multiple key definitions, and support and delimiter strings (helping to make key content unique).
NOTE: In the phoenix environment, and there only if MCALIBs are created, a non-standard config file can be used via the option -c <config-file>. This mechanism is used by the PHOENIX workflow tool to maintain versioning for the product names, to match the historical evolution of the product names. This is needed for MCALIB names only, since it might be desirable to keep the original naming shceme. The names of the various config files, and their versioning, are maintained in config.phoenix. This feature is not needed for the IDP production with phoenix, since the IDPs always have standardized product names.
1. go to $DFO_CAL_DIR/
2. for all product files (${DFO_FILE_NAME}*fits) create new name:
2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg; map into 4letter mcalib code in config file (mcal_code)
2.3 use ${DFO_OFFSET}, read mjd-obs; construct DATE (anything between [DATE]T12+offset … [DATEplus1]T12+offset is called DATE); NOTE: contrary to all other cases, DATE here has the format yymmdd!
2.4 CALIB: fill placeholder for version (e.g. "VERSION"); GEN: fill "A" for VERSION
2.5 read config file for setup key, support strings, SPECSTRING (pieces of shell code read from the config file), separators
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 if configured, execute $PRE_PLUGIN2.8 read extension from original file
2.9 construct new name:
<ins_code>_<mcal_code>_<DATE>_VERSION_<setup_code>.<ext>
Examples:
GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits (CALIB)
GI_GCAT_021230A_line_catalog.fits (GEN)2.10 check for filename length less than 68 chars; exit if longer name constructed: will lead to problems with database and tools
3. CALIB only: do versioning
3.1 read pro.catg; loop on pro.catg:
3.2 check per pro.catg in config file if versioning is required
3.3 if NO: loop on setup per pro.catg, write for all files 'A' instead of 'VERSION'
3.4 if YES: loop on setup per pro.catg:3.4.1 sort by mjd-obs
3.4.2 give first one 'A', second one 'B' etc, up to 26 = Z
Example: GI_PFEX_040414B_Medusa1_L437.2nm_o2.fits (CALIB)
The tool has found two cases of files with GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits, this is the second one.
4. renaming
4.1 read CHECK_YN from config file
4.2 write per file "mv OLDNAME NEWNAME" into $DFO_CAL_DIR/<date>
4.3 add line to set status flag to 'cal_Renamed'
4.4 if configured, execute $SPECIAL_PLUGIN
4.5 make rn_files executable but don't execute; if CHECK_YN=YES, warn the user
if multiple OLDNAMEs end in a single NEWNAME
4.6 finish
Note: The procedure 4.5 is thought as a safety measure. Although in daily operations
execute=yes will be the standard case, the authority given to the tool to 'implicitly
delete' is potentially dangerous. By setting the additional parameter 'permission',
the user keeps this under his control.
1. go to $DFO_SCI_DIR/<date>
2. for all product files (${DFO_FILE_NAME}*fits) create new name:
2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg, map into sci_code
2.3 read obs.id
2.4 read PIPEFILE, use date_time part up to HH:MM:SS (datetime)
2.5 read config file for setup key, support strings, SPECSTRING
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 read extension from original file
2.8 construct new name: <ins_code>_<sci_code>_<obs_id>_<datetime>_<setup_code>.<ext>
Example: GI_SIFU_125427_2004-04-05T23:12:23 _Medusa1_L437.2nm_o2.fits
3. renaming
3.1 write per file "mv OLDNAME NEWNAME" into $DFO_SCI_DIR/<date>/rn_files
3.2 add line to set status flag to 'sci_Renamed'
3.3 if configured, execute $SPECIAL_PLUGIN
3.4 make rn_files executable but don't execute
3.5 finish
NOTE: contrary to CALIB, multiple files with the same name cannot occur in SCIENCE mode (unless the setup string is not properly defined).
Last update: April 26, 2021 by rhanusch |