ISAAC | |||||
|
Programs performed in Service Mode receive a set of DVDs and in case the program is larger than 1.3 Gbyte (=2 CDs) otherwise CDs are delivered. Details on the data shipping can be found on the archive web page . A description on how to read optical media produced by the ESO/ST-ECF archive is available. There is also a general description of the basics ISAAC data management . There is also a list of FAQs. The ISAAC QC web pages contain beside this topic on service
mode packages also two other branches: the ISAAC
PIPELINE and ISAAC QUALITY CONTROL. Assume your VLT run ID is 67.H-0447(A) and you obtained data on 2001-04-25 and 2001-06-08. Assume furthermore that your VLT data were measured in a pipeline-supported mode. You will then find your data (as FITS files) organized in the following way:
If your data have not been pipeline processed,
the structure will be:
Remember this tree is the logical tree, i.e. this is the way the data have been organized before they have been put onto CDs/DVDs. Due to the 600MB capacity of the CDs, it may well be true that some of these directories have been chopped onto two CDs. It is a good idea to create this tree on your local directory and then copy all files from CDs into this tree.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CONTENT OF DIRECTORIES | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
calib
reduced
log
additional_files
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Raw data summary file: list_of_all_obs | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| This file gives a short description of each FITS file on this disk. This information has been extracted directly from the FITS header.
Note:
The .txt file
prints well with UNIX a2ps if tuned to landscape format, font size 5.0
. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Calibration data summary file: list_of_all_calibs | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| A similar file is provided for the raw calibration frames:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Reduced data summary file: list_of_all_red | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| A similar file is provided for the reduced science and reduced calibration frames
Science products and
master calibration data follow the ISAAC naming
convention. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
QC0 summary file: <RUN.ID>_qc0_report.txt | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| This file < run_id > contains a report of quality control parameters ('QC level 0' where level 0 stands for QC without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK). The list is intended to give a rough indication of whether or not the required constraints have been obeyed. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details. Note that the seeing
values reported here are DIMM seeing values, they are not measured on the frames
or on the jitter products. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
QC1 summary file <RUN.ID>_qc1_report.txt | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| DATA FILES | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Science Raw Data We deliver all the FITS files produced by Observation Blocks you created and that we were able to execute. Note that some of your OBs may have been executed more than once. In particular, if time permitted, we tried to re-execute OBs which produced data clearly out of the specified constraints.
Calibration Data We are also sending you all available calibration data appropriate for your program. These data may include:
ESO is further making an attempt
to provide master calibration frames which have been produced by the quality control
group. Some of these files may be included in your data set and are marked accordingly.
They are stored in the reduced/ subdirectories of each Not all programs will receive all classes of calibration data. SKYFLAT frames in different filters are included when available. STANDARD stars are included
for "clear/photometric" nights. Since 2003-08-06 we include an OddEvenColumnEffect ascii files: since of the spontaneous reappearance of the oddeven column effect in the SW arm array, an oddeven file is included for each SW-array twflat product. The ascii file is usually called IS_SITF_<yymmdd><v>_<filter>_oe.asc and contains for each of the raw input frames a row. Each row conatins from left to right: 1) mean flux in the upper left quadrant (UL_mean), 2) the flux of the odd columns alone of the upper left quadrant (UL_odd), in units of UL_mean (= the first column), 3) the flux of the even column alone in the upper left quadrant (UL_even) in units of UL_mea; 4) the mean flux of the upper right quadrant (UR_mean), followed by UR_odd and UR_even, 7, 8, 9) the same values of the lower left quadrant (LL_mean, LL_odd, LL_even), 10, 11, 12) and of the lower right quadrant (LR_mean, LR_odd, LR_even). From October 2003 (P72) on LW-arm zeropoints are
reduced using a twflat and three non-linearity correction coefficient files.ServiceMode
packages with LW-arm imaging contain linearity raw files for the LW-arm, products
(the detlin recipe was used) and corresponding raw files. SW-arm photometric zeropoints
are reduced using the corresponding twflat. From April 2004 (P73)
on Paranal night log files are no longer distributed. Instead equivalent database
extraction are provided in HTML format. Put you browser to GEN_INFO/ObservingReports/NIGHTLOG.html.
Furthermore we modified the calibration names of spectroscopic products. The
central wavelength is given by 4 digits (before it was 3 digits). A SW-MRspectroscopic
flat is now called 'IS_SSFM_040720B_SK_sl2_2225.fits' Check the ISAAC pipeline pages, specifically for calibration data. For more information about the quality control (QC) procedures and results of ISAAC calibration data, check the ISAAC QC pages. Reduced data Whenever possible we included reduced data, i.e. master calibration frames and/or combined jittered images/spectra produced by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations. Quality checks
for both the production of master calibrations and for science reduction are applied
Check the ISAAC pipeline pages, specifically for more details on science reduction. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PROBLEMS, ISSUES, HINTS | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Odd-even column effect P67: The odd-even column effect consists of variations of the signal between the columns of the array. This effect may be present in some frames. The effect is different in each quadrant and slightly more evident in the lower left one. Before the second week of May 2001 it was usually seen only on frames with signal > 10000 ADU. Starting the second week of May, the effect has been observed in frames with ~3000 ADUs. We are currently investigating the causes. The effect can be eliminated softwarewise by means of Fourier Transform. It is recommended to treat each quadrant individually. The relative difference between even and odd columns is a VERY sensitive function of the flux, but it does not scale linearly. Therefore after sky subtraction, some residuals may still be present. An eclipse recipe is being coded and tested right now. It will be made available to the community as soon as possible. As of 24/07/2001 the read-out speed of the SW array has been slow down by a factor of two. Since that night the odd-even column effect has vanished. P68: The odd-even column effect reappears for p68 (1 Oct 2001 - 14 Oct 2001) until the first intervention.While it is not present until Dec. 24, It reappears again on Dec 25 and exceeds the 1% level at the end of 2001. The odd-even column effect, as a non-linear detector response varies in time for each detector quadrant in a different manner . In Jan 2002 the odd even column effect vanished again. The residual gain difference, which is not flux dependent is less than 6%%. Response structure of the SW imaging Since 010408 the QE of the SW array decreased by about 4% for all pixels within x -gt 190 and 88 -lt y -lt 920. This effect can be eliminate by simply flatfielding IS_SIJC frames. This persistent structure faded during p67 and is no longer visible during p68. Startrace Problem In p67, medium resolution spectroscopic products like IS_SSCM and IS_SSSM suffer from an insufficient correction of the spectrum shape distortion. This effect becomes relevant for good seeing conditions ( -lt 0.6'' ) and slit_0.3_tilted. As a result the spectrum is not properly co-added. Startrace frames are taken now several times a period and co-adding should work properly. Dispersion solution for Z (0.9 micron) and SZ (1.06) micron For SW spectroscopy the sw_arc recipe fails for grating LR and filters Z and SZ, simply because the Xe, Ar arc line catalog used for the ISAAC pipeline does not contain sufficient lines for that wavelength range. The sw_arc products are nevertheless distributed, since they contain beside the invalid dispersion coefficients also the coefficients to correct the optical distortion in dispersion direction. The science and standard star spectra are reduced, using the sw_arc products to correct for the optical distortion, but they are not used for the wavelength calibration. The dispersion solution is either given by the built-in optical model or derived by using the sky lines if available. For any other wavelength range using filters J, SH and SK the dispersion solution is derived using the arc frames. More details on the SW_SSAL product page. Dispersion solution for SK (2.2 micron) The sw_arc recipe fails or converges to the wrong dispersion solution for grating MR filter SK and wavelength around 2.2 micron for the same reason. Since the arc recipe derives beside the dispersion solution the x-component of the optical distortion map, the product is used for later steps of the data reduction and is delivered with the ServiveMode packages, although the dispersion solution is not correct. Optical distortion solution for SK (1.98 micron) The sw-arc recipe fails also for grating MR filter SK and wavelength around 1.98 micron. For this setting all detected arcs lines are on one half of the detector and the algorithm exceeds the extrapolation limit for the optical correction function (WH 020517). Spectrum extraction For SW spectroscopy, spectrum extraction fails and no tfits file is provided in cases where the recipe cannot find the two negative spectra in the reduced science frames. Further details are given on the SW_SSCL product page. Consequences of the Intervention Oct 2001, SW Detector temperature variations Before the first intervention of P68 in October 2001 (52188 < MJD-OBS < 52196) , the SW detector temperature was stable at 60 +- 0.002 K (according to the HIERARCH ESO INS TEMP-DETSW keyword). When ISAAC came again in operation at MJD=52203, the SW detector temperature was still cooling down, meaning TEMP-DETSW was variable for 52203 < MJD-OBS < 52205. For 52206 <= MJD-OBS <= 52207 there was a stable plateau at TEMP-DETSW 65.3+-0.002 K . After MJD-OBS = 52210, TEMP-DETSW is again stable at 63+-0.002 K. As a consequence the dark level and the RON visible is highly variable in the period after bthe intervention. The zeropoints also show the same trend, meaning the detector efficiency as part of the whole instrument throughput changed. After MJD-OBS = 52210, when the SW detector was operating again at a stable temperature of TEMP-DETSW = 63+-0.002 K, qc parameters dark, RON, and zeropoint relaxed also, but now, as due to the new operation temperature at new levels. (WH, 011221). Pattern in spectroscopic flats In a sporadic manner spectroscopic master flats show a large scale pattern, related to a overfitting of the y-dependence of the flat flux level (perpendicular to the dispersion direction). From March 2002 on the -f 1 option is used for the sw_spflat recipe, meaning a polynom of order zero (just a constant) is used to normalized the flat and the y-dependence of the product flat is the one of the raw flat frames. (WH 020304) Fits header problems Between Nov 2001 and Jan 2001 not all acquisition images taken with the SW MoveToSlit and MoveToPixel template are stored, since the fits headers are missing. These files are not distributed with the package, since they cannot be classified. The fits headers will be reconstructed as far as possible and will be ingested into the ESO archive, from where they can be downloaded in case these files are relevant for the scientific evaluation. Due to some operational problems, some of the twilight flat stacks consist of frames where the TPL.START is not filled. Beside the twilight flat stacks also imaging darks (IS_SIDK) as well as jittered images (IS_SIJC) are affected. The missing TPL.START fortunately does not impact raw data classification, calibration association, data reduction and packing. These files are still part of the package; they can easily identified, from the attached GEN_INFO/<RUN.ID>_packsum.txt file where missing keyword entries are identified as 'oOo'. (WH 020201) We found missing HIERARCH ESO INS keyword trees in some of the LW imaging (LIJC) frames. Following nights are affected 020111, 020112 and 020113 (WH 020204). At least in Feb 2002, the temperature sensors are not properly logged and stored in the fits headers, meaning the HIERACH ESO INS TEMP-DETSW and others are given as 0.000 (WH 020301). Due to a faulty software installation on Paranal all frames of 020328 have corrupt fits headers. The following branches are missing: TPL, OBS, DPR and OCS. Furthermore some core keywords like the world coordinate system are missing as well. Information on the publication policy of these data is given elsewhere. (WH 020422). P69: From the beginning of P69 we include LW calibration frames in our QC process Dispersion solution for second order arcs in LW The lw_arc recipe cannot handle 2nd and 3rd order arcs as they appear in arcs using the SH filter for M-band and L-band spectroscopy and the J+Block filter for L-band spectroscopy. Arcs calibration products are nevertheless used to calibrate the x-component of the optical distortion. For this reason arc calibration products are part of the SM package, even the wavelength dispersion solution might be missing. As a consequence extracted spectra of science frames (the TFITS files) might be affected as well (WH 020527).Variations of the row y=512 and y=1 of Hawaii using DoubleCorr The rows y=512 and y=1 of the DoubleCorr readout mode used for imaging with the Hawaii array shows in a sporadic manner fluctuations. The dark current can be 5-12 counts higher when this effect occurs. In most cases the first frame in a sequence is affected. As a consequence, dark products, which are just the average of three raw dark frames might be affected. Also the first raw frame in a set of twilight frames can be subject of this feature. While twilight flat product frames before May 2002 have been produced with the -i darklist.asc frame to provide a dark product, This option and the dark product is omitted from May 2002 on. As a consequence the twflat recipe will no longer take the input dark as offset but will fit both the offset and the slope. The prodcuts generated in this way are robust against the y=512 raw fluctuations.Consequences of the Instrument Intervention 2002-06-23 2002-07-05 Contamination of the LW-arm: Since the intervention, a (light/heat) contamination is observed in all Darks of the LW-arm. The contamination is correlated with DIT. It occurs at the left hand side and in the top area of the array. As a consequence the statistical noise composed by read-noise and shot-noise is no longer a constant over the array. This only affects the Aladdin SW J spectroscopy, as this is the only science observation that is read-noise limited. The contamination is stable up to the end of July, but becomes variable thereafter. The SW-arm detector temperature control is not able to keep the detector temperature within 62+-0.5 K. Fluctuation of more than 1.5 K are observed after the intervention. On 2002-08-14 the operational temperature of the SW-array is modified from T= 62 K to T = 64 K. On 2002-08-26 the operational temperature of the SW-array is modified from T= 64 K to T = 65 K. As a consequence SW-arm Dark current levels for most of the DITs have changed as well. The odd-even-column effect of the LW-arm Aladdin array is below 0.005, but exceeded 0.01 on 2002-07-29 (to be continued). (WH 020819) On 2002-08-01 the collimator got stuck and was moved to a compromise position. Therefore ISAAC image quality is limited by the current fixed collimator position optimized for the broad band filters. The point spread function of SW-arm spectra of point sources can show up two peaks, meaning spectra are not longer fully focused. (WH 020917) Ongoing problems concerning the alignment of the LW-arm and SW-arm objective wheels the reduction scheme of LW chopping spectra is changed. Instead of the day-time calibration arc frames, the sky lines are used for wavelength calibration. In case the sky lines cannot be used, the physical model is used for for wavelength calibration. Note that LW chopping spectra are not subject of optical distortion correction using startrace frames and arc line curvature. LW chopping spectra products are also not flat fielded. (WH 020920) Tilted telluric absorption lines can occur in spectra of point sources. They can appear for both gratings LR and MR and both arms LW and SW. The tilt is about 0.187 pixels per row, while sky emission lines are not tilted, apart from the optical distortion which is several orders smaller. Since P70 we find more and more examples, where the tilt can become curved, meaning the dispersion is no longer a constant for all of the rows. The effect is not understood and the pipeline does not correct this feature. (WH 020925) Persistence can occur in dark frames of DIT greater than 100 sec. The amount is usually a few counts. They occur when due to operational reasons the last science frame of the night contains bright sources which saturate the array. The proceeding day-time calibrations are slightly contaminated by a persistence effect. (WH 021023) LW sky flats products (IS_LISF) are generated using the twflat recipe. The calibration template takes several exposures at three different airmasses, but the flux range is not large enough to allow for proper zero level offset determination. (WH 021105) After a detector intervention the arc persistence disappeared. On 28 Nov 2002, The read speed of the Hawaii changed from 6 to 12 to minimize the odd-even column effect. Min. DIT has increased from 1.77 to 3.55 s (WH 021202) There
is since Nov 2002 a big dust grain on the SW-detector. By chance it is on a row
where the SW-arm telluric star is jittered. As a consequence all SW-arm stars
taken with the SW-spec_cal_StandardStar template suffer from the obscuration
by the patch at pixel 585,630 The jitter recipe showed recently a high rate of frames which could not be correlated when applied to SW-arm imaging stacks. We found that the recent occurrence of the big bad pixel area appears in the list of suitable correlation objetcs and is used to correlate the image stack. To overcome these problems, we modified the jitter ini file. We use from now on (Dec 2002) 'AutoThreshold = 10.0;' as a peak detection sigma threshold instead of the default value of 2.0. So far we allow only for objets brighter than 20 sigma above the background. Another solution, would be to set 'ObjectSource = file ;' and write the x and y pixel of the desired correlation object into a file called 'objects.in'. Jitter will take the user-define position to search for the correlation object. (WH 030110) The big bad pixel area is gone. All science images taken after 2003-02-20 are pipeline processed with the default 'AutoThreshold = 2.0' parameter again. There occurred a misalignment in the SW-arm from 2003-02-14 to 2003-02-18 showing strong gradients in the SW-arm twilight flats in the x-direction. There occurred temperature fluctuations of the SW-arm array in the same time interval. A detailed analysis has shown, that the science observations are useful but suffer from the variable slope of the twflats and the unknown spatial distribution of the zeropoints over the array. The odd-even-column-effect
reappeared on 2003-08-06. Usually, the odd columns are usually 1.0002 times
higher and the even columns are 0.9998 times lower than the total flux of a twflat
image, at least for three quadrants. The upper right (UR) quadrant shows the inverse
relation. This behavior is independent on the flux and can be flatfielded. Since
2003-08-06 the odd/even columns are 1.02 times higher/lower and the relation is
strongly dependent on the flux, hence flatfielding becomes difficult. On 2003-08-08,
the detector read-spead was reduced b y a factor of 2 and as a consequence the
OECE reduced from 2% down to 1%. (As a further consequence of the lower read speed,
MINDIT, the smallest possible discrete integration time is now double as large
at 3.5451 sec). Tilted telluric features Since 2003-08-14 the shearing problem reoccurred, meaning absorption lines and telluric features imprinted in a spectrum appear tilted with respect to the dispersion direction. This wavelength shift across the PSF of the spectrum is not corrected by the pipeline. The collimator is no longer at a position for minimum tilt of the telluric features in the center of the spectrum. (WH 031010). The isaacp arc recipe was improved and put
in operation for P72. One setting (SW MR 1.71) cannot be handled, since there
are only two arc lines detected by the recipe to be used for the image distortion
part. The recipe stops without reaching the wavelength calibration part. Science
products with this setting are reduced without arc product (= without optical
distortion correction in x-direction) and the wavelength calibration is done using
the sky lines imprinted in the science raw frames. (WH 2004-03-03). 50Hz noise appears in the SW darks in DoubleCorr read mode for DIT=5 and lasts for about 2 weeks in Feb 2004. The RON increased from 2.5 up to 3.5. 50Hz noise occurs in a sporadic manner for DIT=5 and DIT=2 as well. (WH 2004-03-18). Persistence
can occur in long-DIT SW darks in NonDest readmode. In most cases these are
afterglow images of bright point sources taken during the night and amount to
a few counts. They cancel out during pipeline data reduction via the double subtraction.
(WH 2004-03-18). The grating calibration has been updated in December 2004. As a consequence the grating offsets (measured central wavelength minus nominal wavelength) is no longer +20 pixel but ~0 pixel. (WH 050104) The
sp_flat recipe flags all pixel of the master flat with less than 0.5 and more
than 2.0. The spjitter recipe flags the master calibration flat in the same manner
resulting in a too restrictive handling of flat fielding, in case when the master
flat is steep in wavelength direction. As a consequence science and std spectra
have been cut at both end of the spectra. These thresholds have been updated to
0.01 and 3.0. (WH 2005-01-01) The intervention in April 2005 and a further intervention in May 2005 resulted in slightly changed instrumental calibrations.For data reduction both interventions define 'breakpoint', meaning the complete cascade for a science OB should not cross the intervention periods. (WH 050512) ISAAC SW-arm science raw fits headers
taken between 2005-03-023 and 2005-05- 27T03:30 with ADA.POSANG != 0 contain wrong
word coordinate system (WCS) values. The world coordinates are expressed
by the CD matrix since 2005-03-23 and the correct values can be reconstructed
from other fits header key values. CDELT1=0.148/3600; CDELT2=CDELT1; (old WCS
: CROTA1=CROTA2=ADA.POSANG); 2005-07-01 ... 2005-08-01 is a period of strong 50Hz
noise in SW-arm DoubleCorr readmode darks. Darks with DIT=2, 3.5 and
10 sec were impact. 2005-10-19 A temperature increase by about 100K in the vessel caused an increase of the Aladdin dark current and a decrease of the Hawaii dark current by a few 10 counts. After two days the regular dark currents were retained again. start 2005-10-19T22:00 ; peak at 2005-10-20T11:00. No further side effects have been detected. Known IRAF Problems Filename Length Problem: To display or manipulate the FITS files with older versions of IRAF (before 2.11), you can:
Header Interpretation Problem: ESO FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines. Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format. ST-EFC Science Archive Facility provides a stand-alone hierarch28 fits tool which does the FITS header conversion.
|