Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO

Note: The tool is now distributed through the utilities package (utilPack).

dfosBackup

Description

This tool is an infrastructure tool to support the regular backups of operational software and data. It is intended to be called once a week as a cronjob. It copies a set of defined data into two places, one on the raid5 data disk ($BACKUP_DIR1), the other under the home directory ($BACKUP_DIR2). The goal for the backups is to have copies of the operationally critical software, in cases of data losses and of unintentional data deletion.

The strategy for backups is driven by the following:

the raid5 disk array (typically 500 GB big) offers a relatively high degree of data security (redundancy if one disk fails, hot spare); it is currently not backed up (deleted data cannot be recovered)
the disk with the home directory (10 - 20GB) is backed up by on tape daily, with copies kept for one week, and one weekly copy for a total of four weeks.

Hence a weekly customized backup using dfosBackup, kept on two different places in the system, combines the security of the raid5 system with the available tape backups, and extends the backup depth to something like 10 weeks (configurable).

Because of backup size, the type of data to be backed up needs to be selected carefully. It contains:

all software tools ($DFO_BIN_DIR, $DFO_PROC_DIR, $DFO_TREND_DIR etc.)
all configuration information ($DFO_CONFIG_DIR)
all important log files (under $DFO_LOG_DIR: ABs, processing logs etc.)
all listings and reports
history and content of packages (under $DFO_SRV_DIR)
ops log files ($DFO_OPS_DIR)

The user can add files or directories to the configured set. A current backup on the GIRAFFE system with the standard configuration takes 90 MB.

There is the option to call a plugin script at the end. That script could have a function which transfers the backup to another machine, or a CD writer etc.

The total number of backups created can be configured. If that number e.g. is set to 10 and the tool is called weekly, backup no. 11 will overwrite backup no. 1 etc. Hence a backup depth of 10 weeks is offered. This seems to be a reasonable trade-off between security and backup data volume.

Check out here how to set up the tool as a cronjob.

How to use

dfosBackup -v|--version     gives the version number
dfosBackup -h|--help    gives a short help text

dfosBackup        calls the backup from the command line

Recommended call:
[00 08 * * 1 dfosCron -t dfosBackup ] if this line is defined by using crontab -e, the backup will be refreshed every Monday at 08:00.

Installation

The tool comes as part of the utilPack delivery.

Configuration file

Section 1: General parameters
BACKUP_DIR1	/data<nn>/data/backup	1st directory for backups (on raid5)
BACKUP_DIR2	/home/<account>/backup	2ndt directory for backups (on home account)
KEEP_NLATEST	10	keep N latest backups
PLUGIN_FINAL	pgi_dfosBackup	name of optional plugin file (expected under $DFO_BIN_DIR), called at the end
Section 2: Definition of backup items Note: this list should not be modified without good reasons; if you extend it, make sure not to include overly large items
DIRECTORY	$DFO_LST_DIR	path to files or directories to be backed up
FILE_DIR	e.g.: NR_* or monitor	string to identify file(s) or directory names under DIRECTORY; unix placeholder can be used (e.g. list*.txt)
BACKUP_TAR	e.g. LISTINGS, CONFIG etc.	root name of tar file; use only the reserved names: LISTINGS, CONFIG, LOGS, STATISTICS, TOOLS, SPECIAL; others won't get backed up. SPECIAL is reserved for any user-specific data.

Workflow description

1. find and create required directories; the backups go compressed to $BACKUP_DIR<n>/$DATE/LISTINGS_<date>.tar.Z etc.

2. loop over configured items

2.1 per item, call 'tar -rf ...'
2.2 compress the tar file
2.3 give some statistics

3. Remove outdated backup (in excess of $KEEP_NLATEST)

4. Call configured plugin.

Last update: April 26, 2021 by rhanusch