Pipeline 

for the FIASCO spectrograph - Version 1.1

by C. Guirao

1.- Name conventions

2.- Raw and reduced images

The pipeline is the process by which a 2-dim raw FITS image containing the spectrum of the target is processed. The result is a 1-dim FITS image containing the reduced spectrum. This process is also known in astronomical terms as data reduction.

Raw images are those generated by the CCD control software. These are 2-dim FITS image containing the spectrum of the observed target. Filenames are given the name of the target (see name convention) plus extension .fit.

Reduced images are the product of the FIASCO pipeline. These are 1-dim images containing the reduced spectrum of the target extracted from the 2-dim images.

The term data reduction, used frecuently in astronomical software, has here its literal meaning, i.e. a raw image from a 512x512 pixels CCD camera ocupies 512 Kbytes while its reduced spectrum uses less than 9 Kbytes.
 
 

3.- Calibration Data

Besides raw and reduced images, there is also the so called calibration data. This consists of a set of files used by the pipeline to convert the raw images into the reduced images. Tipically calibration data used by the pipeline are:

Calibration data is created by calibration procedures. This is normally a task executed before you start taking exposures from targets. Some of the calibration data like the master bias and dark currents can be taken from previous observation sessions as they are unlikely to change. Other calibration data are necessary to be taken during the observation session like the wavelength, flat fields and flux calibration data.

Detailed description regarding calibration data is available in another document called CALIBRATION.html
 
 

4.- Pre-pipeline

Some calibration data, like the wavelength calibration, is required to run the pipeline successfully.

You could execute the pipeline partially if some of the calibration data is missing but only in manual mode, however you need all the calibration data in order to run the pipeline in an automatic and unattended way.

The shell script pre_pipeline is used here to check that the current directory contains all the requested calibration data, otherwise it stops with an error message but also with instructions regarding the missing calibration data and how to create it.

To run the pre_pipeline check just move in the directory containing the calibration data and execute:

% cd working_dir

% $HOME/midwork/pre_pipeline

5.- Manual Pipeline

A manual pipeline is the one executed by the operator in a MIDAS session by executing the following command:

Midas > @@ pipeline input.fit
where input.fit is the name of a 2-dim FITS image containing the spectrum of a target (most likely a star). During the pipeline process, temporary files are generated with the original input filename plus an extension with the following meaning:
 
  input_ raw file with CCD corrections (hot columns)
  input_b as above + bias subtracted
  input_bd as above + dark current subtracted
  input_bd1 as above + spectrum extracted in 1dim image
  input_bd1f as above + flat field divided
  input_bd1fw as above + wavelength calibrated
  input_bd1fws as above + flux calibrated

The pipeline continues even when some calibration data is missing. In such cases the calibration step is skipped (if the master_bias.bdf file does not exist the bias subtraction is not performed) Only the wavelength calibration data is required in the pipeline- if it is missing the pipeline stops and exits with an error.

The procedure also logs to the standard output the results of the different operations performed on the input data. It keeps plotting in graphical windows the results after each calibration operation.

The pipeline finalizes successfully with the creation of two files:
 
  input.mt A 1dim FITS image containing the spectrum calibrated
  input.ps A PostScript file of the calibrated spectrum

Temporary files are also deleted at the end of the process if the pipeline completes succesfully.
 
 

6.- Automatic Pipeline

The automatic pipeline performs the same process as described in the manual pipeline, except that the MIDAS procedure @@ pipeline input.fit is initiated by a shell script called pipeline. This script polls the working directory for writable FITS files with the extension .fit. These are files supposely to be generated by the CCD control software and transfered or copied to the working directory.

The script that executes the automatic pipeline is $HOME/midwork/pipeline, and can be launched as:

% $HOME/midwork/pipeline working_dir
so it starts the pipeline in the directory indicated by working_dir, or
% $HOME/midwork/pipeline
then starts the pipeline in a directory using today's date format YYMMDD, with the particularity that this date remains unchanged until midday next day to avoid having an observation session with two different dates.

First thing pipeline does is executing another script $HOME/midwork/pre_pipeline to make it sure that all calibration data is available. After passing the pre_pipeline check, the script initiates a MIDAS session in background, and then it searches in the working directory for writable files with the extension .fit. Each file found it is then passed to the data reduction MIDAS procedure pipeline.prg in the form of the command:

Midas> @@ pipeline file.fit
Once processed by MIDAS, the protections of the original raw image file.fit are changed to read-only to avoid to be re-processed again. The scripts pipeline remains in an infinite loop until it receives an external signal like INT, e.g. by typing CTRL-C. Signals HUP, INT, QUIT and TERM are trapped by the pipeline script, it first instructs the MIDAS session in background to stop and finally the it exits.

7.- Documentation:
 
  INSTALLATION.html Installation procedure
  PROCEDURES.html Describes MIDAS procedures
  CALIBRATION.html Generation of calibration files
  PIPELINE.html This file