1. Introduction

Classification and Association of ISAAC raw frames occurs at two places in the VLT DataFlow:

• On Paranal, where raw data arrive in a sequence one-by-one and associated day-time calibrations might not be available for the on-line quick-look quasi real-time pipeline. Association occurs when the OBs for the day-time calibrations are scheduled, in order to garanty completenes of the required calibration set.
• At QC Garching, where all raw frames of a given night +- X nights and verified calibrations are available. The ISAAC pipeline in Garching runs with some delay with respect to the observation data. Generaly spoken, classifiaction is used to distribute raw data stacks to the right recipe, association is used to feed the recipe with the correct (and best available) calibration.
  • CLASSIFICATION: Raw files must be classified in order to know which recipe has be called and when it has to be called. In any case the FITS header keywords are used to cross check against a classification table. In the case of ISAAC this task is performed by the Data Organizer (DO) on Paranal where the pipeline is used as a quick-look tool. In Garching we use scripts to classify raw data. The most important key is the HIERARCH ESO TPL ID and the HIERARCH ESO DPR keywords. The end of template event occurs when HIERARCH ESO TPL EXPNO exceeds HIERARCH ESO TPL NEXP or when the HIERARCH ESO TPL ID has changed with respect to the previous file. Note that all raw frame of a night can be sorted according to the time stamp in ISO format (the ARCFILE key). As a consequence any classification system must have knowledge of the classification of the previous frame. After all files of a certain period, e.g. a complete night or many nights have passed a classification scheme the following known:
    • There is a number of file stacks, each stack is a set of files ready to be submitted to a recipe The last file of the stack triggered the end of template event.
    • Each file stack is associated to a reduction recipe
  • ASSOCIATION: Data association occurs at two placed within the Data Flow Operations. First when calling a reduction recipe and second when packing a Service mode data package. The tasks are very similar but not identical Data association requires rules and in this section we describe the association rules used with ISAAC data

  2. ISAAC Association Rules

  2.1 ISAAC Association Rules for SW Imaging

  • CALIBRATIONS
    • DARKS: to reduce a stack of dark frames, no association is required. Darks can be reduced without further calibration products. Our classification scheme garanties, that each input list of dark frames are homogeneous concerning the key HIERARCH ESO DET DIT, HIERARCH ESO OCS SELECT-ARM and HIERARCH ESO DET MODE NAME. The product tag is IS_SIDK (ISAAC Short wavelength Imaging DarK).
    • Twilight FLATS: to reduce a stack of twilight raw frames a dark calibration product can optionally be submitted. As for all other associations within the ISAAC DFO we assume that calibration products are available not only for the current night but also for all previous nights. This means it is possible to search for the closest in time calibration product. The association rule for the required dark is: Search for the closest in time IS_SIDK product which matches in HIERARCH ESO DET DIT, meaning we need a dark with the same discrete integration time.
    • Illumination Frames: to reduce a stack of raw illumination check frames the calibration products dark (SIDK) and twilight flat (SITF) can optionally be submitted. The association rule is : the matching parameter for SIDK is HIERARCH ESO DET DIT. The matching parameter for SITF is HIERARCH ESO INS FILT1 NAME. If HIERARCH ESO INS FILT1 NAME of SITF is open then match HIERARCH ESO INS FILT2 NAME. At this point we assume that there is only one instance of a filter in the filter wheel. If e.g. the filter is replaced by another filter with the same name one has to include a match of the HIERARCH ESO INS FILT1 ID key as well in order to distinguish the optical properties of different physical items in the instrument.
    • Zero points: raw zero-point frames taken with a technical template can be reduced without further associated calibrations, but a twilight flat calibration product can be associated. The matching parameter for SITF is HIERARCH ESO INS FILT1 NAME. If HIERARCH ESO INS FILT1 NAME of SITF is open then match HIERARCH ESO INS FILT2 NAME.

  • SCIENCE FRAMES
    • AutoJitter, AutoJitterOffset: The following calibration products can be associated with the jitter recipe:
      • twilight flat : SITF : matching in HIERARCH ESO INS FILT1 NAME or in HIERARCH ESO INS FILT2 NAME.
      • Bad Pixel Map: Bad pixel map frames as produced by the twilight flat recipe. If we have found a twilight flat calibration product IS_SITF_020410A_Js.fits there must be also a bad pixel map with the name IS_SITF_020410A_Js_badpixmap.fits. Therefore the association rule for the bad pixel map is the same as for the twilight flat itself
      • Darks: the SIDK product must match again in HIERARCH ESO DET DIT.
      • Ini: Note that the jitter recipe requires an initialization file to be generated with the -g option. This file contains more than 30 parameters to be adjusted. Among these there are parameters which control the flow of the recipe and the names of the associated calibration products. The jitter recipe is one of the few recipes, where the calibration products are no command line parameters to the recipe but entries in the ini file.

  2.2 ISAAC Association Rules for SW Spectroscopy

  • CALIBRATIONS
    • Darks (SSDK): Is for SW Imaging dark frames can be submitted to the dark recipe without further calibration product. See SIDK for details.
    • Flats (SSFL, SSFM) : raw SW spectroscopic flats can be reduced without further associated calibration product. The calibration template provides a sequence of frames with alternating LAMP-ON LAMP-OFF frames, meaning the darks are part of the input stack and classified as raw spectroscopic flat frames Basic properties of an flat product is the grating (either M or L), the filter, the wavelength and the slit.
    • Arcs (SSAL, SSAM) : raw SW arcs lamp frames can be reduced without further associated calibration product. Basic properties of an arc product is the grating (either M or L), the filter, the wavelength and the slit.
    • Startrace (SSST) : The startrace calibration template provides a series of 3 times 11 frames. The whole stack of 33 frames must be submitted to the SW startrace recipe. No further calibration products are required. The tag of the calibration product is SSST.
    • Standard Stars, telluric standards and flux standards or for what ever purpose are by definition calibrations. From the point of association rules, they must be handled as spectroscopic science targets. See there.

  • SCIENCE and Standard Stars
    Spectroscopic settings of standard stars and science targets are parameterized by the grating HIERARCH ESO INS GRAT NAME, the filter HIERARCH ESO INS FILT1 NAME or HIERARCH ESO INS FILT2 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT WLEN. Note that the grating is coded in the calibration products names as a M or as a L. The reduction is performed via the spjitter recipe. In a first step a initialization file must be generated via the -g option. In a second step the associated calibration product file name for the wavelength calibration must be inserted in the ini file. In a third step the other three calibration product files must be provided in a file which contains the list of calibration files. This list file must be submitted via the -c option to spjitter. An example calibration list file is:

               /diska/calibs/IS_SSFM_020405A_J_sl2_127.fits        MASTER_SP_FLAT
               /diska/calibs/IS_SSAM_020405A_J_sl2_127_Xe+Ar.tfits ARC_COEF
               /diska/calibs/IS_SSST_020202A_poly2d_MR.tfits       STAR_TRACE_COEF
              

    Note the mandatory classification tags in the file.
    • MasterSpFlat is the tag of the spectroscopic flat in the spjitter ini file. The association rule for the spectroscopic flat calibration product is: If HIERARCH ESO INS GRAT NAME is LR then search for the closest in time SSFL calibration product. If HIERARCH ESO INS GRAT NAME is MR then search for the closest in time SSFM calibration product. The following parameters must match : the the filter HIERARCH ESO INS FILT1 NAME or HIERARCH ESO INS FILT2 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT /WLEN. The tag MASTER_SP_FLAT must be used in the calibration file list.
    • StarTraceTable : No associaten rule is required for the SSST product. Among the many TFITS tables produced by the startrace recipe, the IS_SSST_{date}_poly2d_MR.tfits or the IS_SSST_{date}_poly2d_LR.tfits file must be taken, dependent on the used grating. The tag STAR_TRACE_COEF must be used in the calibration file list.
    • ArcTable is used for the correction of the optical image distortion. The association rule is the same as for the flat: If HIERARCH ESO INS GRAT NAME is LR then search for the closest in time SSAL calibration product. If HIERARCH ESO INS GRAT NAME is MR then search for the closest in time SSAM calibration product. The following parameters must match : the the filter HIERARCH ESO INS FILT1 NAME or HIERARCH ESO INS FILT2 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT WLEN. The tag ARC_COEF must be used in the calibration file list. Note that there might be also SSAL files based on the Xe lamp alone or the Argon lamp alone. These product file names end with '_Xe.ftits' or '_Ar.tfits'. The spjitter recipe accepts each of these calibration products. But there is one exception for the flux standards: If the HIERARCH ESO INS OPTI1 NAME contains a string beginning with mask then the HIERARCH ESO INS OPTI1 NAME of the arc needs not match.
    • The wavelength calibration can be performed in one of three ways : Arc table, imprinted sky lines, or physical model. The method must be selected by editing the control keys in the spjitter ini file. In case the wavelength calibration is performed with a arc table, the file name must be inserted to the WavecalArcFile tag in the ini file. The same association rules hold for the arc table used for the wavelength calibrations as for the arc table used to correct the optical distortion. Again there is some freedom to choose the product based on Xe arc lamp or Argon arc lamp or both lamps in one product if available.

  2.3 ISAAC Association rules for LW Imaging

  • CALIBRATIONS
    • DARKS: to reduce a stack of dark frames, no association is required. Darks can be reduced without further calibration products. Our classification scheme garanties, that each input list of dark frames are homogeneous concerning the key HIERARCH ESO DET DIT, HIERARCH ESO OCS SELECT-ARM and HIERARCH ESO DET MODE NAME. The product tag is LGDU (ISAAC Long wavelength Generic Dark Doublecorrelated) or LGDU (ISAAC Long wavelength Generic Dark Uncorrelated) depending on the HIERARCH ESO DET MODE NAME. LGDU and LGDD darks products occur only in the imaging reduction cascade. The LW spectroscopic pipeline does not require a dark calibration product, as the SW spectroscopic pipelimne does neither require a dark.
    • Twilight FLATS: (IS_LITF) to reduce a stack of twilight raw frames a dark calibration product can optionally be submitted. As for all other associations within the ISAAC DFO we assume that calibration products are available not only for the current night but also for all previous nights. This means it is possible to search for the closest in time calibration product. The association rule for the required dark is: Depending on the HIERARCH ESO DET MODE NAME, search for the closest in time IS_LGDU or LGDD product which matches in HIERARCH ESO DET DIT, meaning we need a dark with the same discrete integration time. Note that LITF is rarely used, since the illumination range in a twilight stack is smaller in the L-band and M-band with respect to the SW bands.
    • Sky FLATS: (IS_LISF) to reduce a stack of sky raw frames a dark calibration product can optionally be submitted. As for all other associations within the ISAAC DFO we assume that calibration products are available not only for the current night but also for all previous nights. This means it is possible to search for the closest in time calibration product. The association rule for the required dark is: Depending on the HIERARCH ESO DET MODE NAME, search for the closest in time IS_LGDU or LGDD product which matches in HIERARCH ESO DET DIT, meaning we need a dark with the same discrete integration time. Note that LISF takes exposures at quite different airmasses to overcome the problem of the small flux range given in the twiligth flat template. There no skyflat recipe yet.
    • Zero points: raw zero-point frames taken with a technical template can be reduced without further associated calibrations, but a twilight or sky flat calibration product can be associated.

  • SCIENCE FRAMES
    • AutoJitter, AutoJitterOffset (Non Chopping templates): science frame stacks are submitted to the jitter recipe, the same as used for the SW-arm. Hence similar association rules apply.
      • twilight flat or sky flat : (LITF or LISF) : matching in HIERARCH ESO INS FILT3 NAME or in HIERARCH ESO INS FILT3 NAME. and being observed with the same detector read-out mode HIERARCH ESO DET MODE NAME. Note that the fileter wheels in the LW-arm are called FILT3 and FILT4, while the SW-arm filter wheels are called FILT1 and FILT2.
      • Bad Pixel Map: Bad pixel map frames as produced by the twilight flat recipe. If we have found a twilight flat calibration product IS_LITF_020410A_M.fits there must be also a bad pixel map with the name IS_LITF_020410A_M_badpixmap.fits. Therefore the association rule for the bad pixel map is the same as for the twilight flat itself
      • Darks: the LGDU or the LGDD product must match again in HIERARCH ESO DET DIT and HIERARCH ESO DET MODE NAME.
      • Ini: Note that the jitter recipe requires an initialization file to be generated with the -g option. This file contains more than 30 parameters to be adjusted. Among these there are parameters which control the flow of the recipe and the names of the associated calibration products. The jitter recipe is one of the few recipes, where the calibration products are no command line parameters to the recipe but entries in the ini file.
    • img_obs_AutoChopNod, img_cal_StandardStar (Chopping templates): science raw frame stacks taken with the chopping templates must be submitted to the lw_jitter recipe. The association rules are the same as for the non-chopping templates, except no dark calibration product is associated. The HIERARCH ESO DET MODE NAME is different to the HIERARCH ESO DET MODE NAME of the non-chopping frames.

  2.2 ISAAC Association Rules for LW Spectroscopy

  • CALIBRATIONS
    • Flats (LSFL, LSFM) : raw LW spectroscopic flats can be reduced without further associated calibration product. The calibration template provides a sequence of frames with alternating LAMP-ON LAMP-OFF frames, meaning the darks are part of the input stack and classified as raw spectroscopic flat frames Basic properties of an flat product is the grating (either M or L), the filter, the wavelength and the slit.
    • Arcs (LSAL, LSAM) : raw SW arcs lamp frames can be reduced without further associated calibration product. Basic properties of an arc product is the grating (either M or L), the filter, the wavelength and the slit. Note that the wavelength can be outside the range given by the filter, meaning arcs can be taken in second or 3rd order.
    • Startrace (LSST) : The startrace calibration template provides a series of 3 times 11 frames. The whole stack of 33 frames must be submitted to the SW startrace recipe. No further calibration products are required. The tag of the calibration product is SSST.
    • Standard Stars, telluric standards and flux standards or for what ever purpose are by definition calibrations. From the point of association rules, they must be handled as spectroscopic science targets. See there.

  • SCIENCE and Standard Stars
    Spectroscopic settings of standard stars and science targets are parameterized by the grating HIERARCH ESO INS GRAT NAME, the filter HIERARCH ESO INS FILT3 NAME or HIERARCH ESO INS FILT4 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT WLEN. Note that the grating is coded in the calibration products names as a M or as a L. The reduction is performed via the spjitter recipe. In a first step a initialization file must be generated via the -g option. In a second step the associated calibration product file name for the wavelength calibration must be inserted in the ini file. In a third step the other three calibration product files must be provided in a file which contains the list of calibration files. This list file must be submitted via the -c option to spjitter. An example calibration list file is:

               /diska/calibs/IS_LSFM_020405A_SL_sl2_436.fits        LW_MASTER_SP_FLAT
               /diska/calibs/IS_LSAM_020405A_SH_sl2_436_Xe+Ar.tfits LW_ARC_COEF
               /diska/calibs/IS_LSST_020202A_poly2d_MR.tfits        LW_STAR_TRACE_COEF
              

    Note the mandatory classification tags in the file.
    • MasterSpFlat is the tag of the spectroscopic flat in the spjitter ini file. The association rule for the spectroscopic flat calibration product is: If HIERARCH ESO INS GRAT NAME is LR then search for the closest in time SSFL calibration product. If HIERARCH ESO INS GRAT NAME is MR then search for the closest in time SSFM calibration product. The following parameters must match : the the filter HIERARCH ESO INS FILT3 NAME or HIERARCH ESO INS FILT4 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT /WLEN. The tag MASTER_SP_FLAT must be used in the calibration file list.
    • StarTraceTable : No associaten rule is required for the SSST product. Among the many TFITS tables produced by the startrace recipe, the IS_SSST_{date}_poly2d_MR.tfits or the IS_SSST_{date}_poly2d_LR.tfits file must be taken, dependent on the used grating. The tag STAR_TRACE_COEF must be used in the calibration file list.
    • ArcTable is used for the correction of the optical image distortion. The association rule is the same as for the flat: If HIERARCH ESO INS GRAT NAME is LR then search for the closest in time SSAL calibration product. If HIERARCH ESO INS GRAT NAME is MR then search for the closest in time SSAM calibration product. The following parameters must match : the the filter HIERARCH ESO INS FILT3 NAME or HIERARCH ESO INS FILT4 NAME, the slit HIERARCH ESO INS OPTI1 NAME and the central wavelength HIERARCH ESO INS GRAT WLEN. The tag ARC_COEF must be used in the calibration file list. If HIERARCH ESO INS FILT3 NAME or HIERARCH ESO INS FILT4 NAME is M in the standard star or science frame then the ARC must match with filter SH, since arcs in the M-band are taken in second order with filter SH. This asociation rules applies for grating MR as well as for LR. For the LR-grating and L-band the filter rule is : link SL to SL. Since SL opservations can be associated with 1rst, second or 3rd order arcs, the filter association rule for the arcs is a little longer:
      • If HIERARCH ESO INS GRAT WLEN lower or equals 2.55 micron use the arc matching in slit, wavelength and the SH filter (second order) or the SL filter (first order).
      • If HIERARCH ESO INS GRAT WLEN greater then 2.55 micron use the arc matching in slit, wavelength and the J+Block filter (third order) or the SL filter (first order).
    • The wavelength calibration can be performed in one of three ways : Arc table, imprinted sky lines, or physical model. The method must be selected by editing the control keys in the spjitter ini file. In case the wavelength calibration is performed with a arc table, the file name must be inserted to the WavecalArcFile tag in the ini file. The same association rules hold for the arc table used for the wavelength calibrations as for the arc table used to correct the optical distortion. Again there is some freedom to choose the product based on Xe arc lamp or Argon arc lamp or both lamps in one product if available.

  2.5 ISAAC Association rules for SW Polarimetry

  The rules for polarimetric imaging in the SW-arm are identical to the rules for SW imaging

  2.6 ISAAC Association rules for LW FastJitter and Burst mode

  This mode is currently (2007-07) not pipeline supported.

  • CALIBRATIONS
    • DARKS: The additional keywords to distinguish a normal LW-arm dark from dark taken in burst or fastjitter mode are: HIERARCH ESO DET WIN NX, and HIERARCH ESO DET WIN NY that contain the size of thw window raed out in pixel. The values are smaller than 1024. The third key is HIERARCH ESO DET FRAM TYPE. The values can be INT for normal readmode, DIT for FastJitter (=HIT mode) or SAMPLE for the burst mode.
      • INT means a normal 2d frame with NDIT times DIT integration time
      • DIT means a cube of NDIT 2d frames. Each plane of the cube was exposed to DIT seconds, the whole cube was exposed to NDIT * DIT seconds.
      • SAMPLE means a cube of 2 * NDIT 2d frames. Each pair of planes in the data cube are the first and the second frame of the ResetReadRead sequence. Each pair of planes corresponds to an exposure of DIT, thw whole data cube corresponds to DIT * NDIT seconds. Subtracting the plane pairs from each other reduces the data cube to one of type DET.FRAM.TYPE==DIT.
    • Finally the the NAXIS shows if the frame is a normale (NAXIS==2) or is a cube (NAXIS==3). To classify hit/burst dark the NAXIS==3 condition is sufficient. The grouping rule (chopping the list of all draks taken in ine night in meaningful subsets) should result in groups of files that are common in DET WIN NX, DET WIN NY and DET FRAM TYPE.
    • Twilight are taken in window read out with NAXIS=2 and HIERARCH ESO DET FRAM TYPE == INT. These are normal frames, that are not resolved per NDIT, no cubes, only DET.WIN.TYPE == 1, while all other flats with full detector read out have DET.WIN.TYPE == 0. This means the grouping rule involves in addition to arm and filter the match of the DET.WIN.TYPE key.
      Flats require a set of raw dark frames or a master dark, that matches in arm, readmode, DIT and in addition in the window size DET WIN NX and DET WIN NY
      
  • SCIENCE
    • FastJitter and Burst mode frames are classified via the DPR.TYPE == JITTER,HIT and DPR.TYPE == JITTER,BURST tags respectively. The template usually generates frame with DET.FRAM.TYPE=DIT (in fastjitter mode) or SAMPLE (in burst mode) and a user specified number of planes in the cube followed by a frame with DET.FRAM.TYPE=INT, and NAXIS3=1; this is a cube with one plane.
      • For each frame we associate a sky flat matching in arm and filter (as in normal imaging) and in the window size DET.WIN.NX and DET.WIN.NY, but not in DET.FRAM.TYPE.
      • We associate a dark matching in window size DET.WIN.NX and DET.WIN.NY, and in read mode DET.FRAM.TYPE.