Quality Control: Twilight Flats

ISAAC Quality Control:
Twilight Flats

QUALITY CONTROL HOME
CAL \| HC \| refs \| QC
ISAAC QC
ISAAC QC HOME
TOC
Trending & QC1
general info
trending & QC1
Dark
Phot.Zeropoints
Slit/Grating
CheckFlat
ImagingFlat
DetectorLinearity
Illumination
ArcLamp
SpecFlat
StarTrace
Telluric Std
Flux Std
Science Imaging
Science spectroscopy

ISAAC Health Checks
Pipeline
general info
science data
science recipe
calib data
calib recipes
assoc map
Data Packages
general info
directories
list files
problems, issues
product names
Data Management
general info
QC links:
QC home
Data Products
calChecker
HealthChecks
Reference Frames

QC1 database
Paranal autrep database
(ESO internal)
[best viewing with style sheets and javascript enabled]

fixed pattern noise | bad pixels | operations | odd even column effect


QC PLOTS
	CURRENT	HISTORY
fixed pattern noise
differential fixpat. noise
bad pixels
operations
QC1 database (advanced users): browse \| plot (twilight)
odd even column effect
QC1 database (advanced users): browse \| plot (odd-even)
Click on CURRENT to see the current trending (Health Check). Click on HISTORY to see the historical evolution of the trending.

Twilight flats are acquired as a series of jittered sky frames during twilight scanning a high flux range. In the LW bands (L, M_NB and some narrow band filters) a different template is used, that jitters on three different zenith distances. The main purpose of the twilight flat is to correct the pixel to pixel gain variations and detect bad pixels to be masked. Furthermore the odd-even column effect is monitored.

top fixed pattern noise

QC1 parameters

parameter	QC1 database: table, name	procedure
fixed pattern noise	isaac_twflat, std_UL, std_UR, std_LL, std_LR	- Fixed pattern noise (FPN) is derived via a post-pipeline DFO script in a similar manner from the twflat/skyflat product frame as for the dark frames. The stdev of the central 100 x 100 pixel large window is taken from each quadrant of the product frame. This parameter is rather insensitive to measure the oddeven column effect, since each pixel value of the flat product is the slope derived from a linear regression algorithms fit to the flux relation of the raw input frames. The oddeven column effect as non-linear effect below the percent level is averaged out in this parameter. Variations in the number of debris features, and the number of hot pixels are measured by this parameter.
large scale gradients	isaac_twflat, tot_UL, tot_UR, tot_LL, tot_LR	- A arge scale gradient is derived via a post-pipeline DFO script in a similar manner from the twflat/skyflat product frame as for the dark frames. The median of the central 100 x 100 pixel large window is taken from each quadrant of the product frame.
differential fixed pattern noise	isaac_twflat, diff_fpn_UL, diff_fpn_UR, diff_fpn_LL, diff_fpn_LR	- Differential Fixed pattern noise (DIFF FPN) is derived via a post-pipeline DFO script. The master twilight flat is divided by a reference flat to get rid of the dominating structural noise due to the detector (the pixel-to-pixel gain variation pattern) that is of the order of 10%. The histogram of this ratio frame is fit by a Gaussian to retrieve a robust estimate of the variance. The reference master twilight flat is updated from time to time. Mean and stdev QC parameters derived from the ratio frame are very sensitive to changes in the fringe pattern and the pixel-to-pixel pattern due to the detector.

Trending

To demonstrate the meaning of monitoring the robust scatter of the differential twilight flats we show in the upper row examples of the current master twilight flat divided by the reference twilight flat. It shows long-term variations. The reference product is usually updated after an intervention. In the lower row we show examples of the current master twilight flat divided by the last (most recent) master twilight flat. These ratio frames show the short term variations. The pixel-to pixel gain variations are mostly canceled out in the ratio frames. The remaining components are large scale structures due to minor changes in the optical path and different illumination .Furthermore nonlinearities between the odd and even columns become visible.


Hawaii ratio (reference flat)	Aladdin SW-band ratio (reference flat)	Aladdin LW-band ratio (reference flat)

Hawaii ratio (with consecutive flat)	Aladdin SW-band ratio (with consecutive flat)	Aladdin LW-band ratio (with consecutive flat)

The histogram of these frames and in particular the width of the distribution is a measure of small changes beyond the large (scatter/flux ~ 10%) pixel-to-pixel variations. These include changes in the photon noise (see operational related QC parameters), changes in the non-linearity between odd and even columns, and changes in the fixed -pattern noise. Single pixels with large deviations (e.g. a new debris) would dominate a QC parameter based on the sample standard deviation, but in the histogram it would appear as a small a small peak far beyond the distribution center, and would not affect the Gaussian fit. . The procedure is applied for each quadrant individually. Results of the fitting routine are shown for two different setups.

Hawaii array
Aladdin array

Figure caption: The red line is histogram of the master flat divided by the reference flat. The green line is the initial guess based histogram, and the blue curve is the final least square fit.

top bad pixels

QC1 parameters

parameter	QC1 database: table, name	procedure
number of bad pixel	isaac_twflat, badpix	- For each pixel a robust fit is performed to retrieve the response of the pixel at different flux levels. Those pixels that deviate from the mean linear relation by a certain threshold are declared as bad pixel. They show either a highly non-linear relation or are dead, due to debris.

top odd-even column effect

The Hawaii array showed in some epochs a weird behavior: the odd columns showed show a non-linear behavior that is different with respect to the even columns. Due to the non-linearity of this effect flat-fielding does not correct for the oddeven column pattern. OddEvenColumnEffect is measured by two parameters per detector quadrant. For each raw frame of the twflat stack the following calculations are done: The pipeline retrieves for each quadrant of the array the flux averaged over all odd columns alone (flux_odd) and over all even columns alone (flux_even) and the flux of the whole quadrant (flux_tot) . For each raw frame the quantity (flux_odd - flux_tot)/flux_tot and (flux_even - flux_tot)/flux_tot is built which are symmetric with respect to zero, since flux_odd + flux_even == flox_tot. Usually both quantities are rather independent on flux_tot on levels of +-1e-4. Such a constant ratio between odd/even columns and the total flux demonstrates a high degree of linearity and can be canceled out by flat field dividing.

Fig 1: oddevencolumn effect not present. The odd and the even columns behave linear with flux, the difference between odd and even columns is below 0.02 per cent.

On some occasions, the ISAAC Hawaii array spontaneously shows a sudden increase of these values and in connection with a strong dependence on flux_tot, giving rise to problems concerning data reduction and eliminating the odd-even column effect. In addition this behavior is rather variable with time:

Fig. 2: odd even column effect present (red lines). The non-linear behavior of the last Js twflat (blue) and of the one taken before (black) is shown in addition to monitor time variability of the odd even column effect.

The odd even column effect is monitored in the following manner. All (flux_odd - flux_tot) / flux_tot (and for reasons of consistency check the (flux_even - flux_tot ) / flux_tot) values of a given twflat raw frame stack are considered as two different samples. For the odd sample and the even sample we determine the mean and the stddev These four parameters are monitored for each quadrant. The mean is a measure of the amplitude of the odd even column effect and the stddev is a measure of the non-linearity with respect to the flux range of the raw frames. The trending plots show for each quadrant of the chip two panels: the mean panel and the stdev panel. The oddeven column effect does occur for the SW-arm (Hawaii array), but is monitored for the LW-arm (Aladdin array) as well.

QC1 parameters

parameter	QC1 database: table, name	procedure
gain ratio between odd and even columns	isaac_twflat_oe, oc_UL_mean, oc_UR_mean, oc_LL_mean, oc_LR_mean	- Extract the mean flux of the odd columns alone, of the even columns alone, and of all columns from each quadrant of a raw twlight frame of the stack. If there are n raw frames, build a sample of n values of (flux(odd) - flux(tot) ) / flux(tot). oc_UL_mean is sample mean of the upper left quadrant. It is a measure on the gain differences between odd and even columns. Up the 2006-01-01 the complementary ec_UL_mean values have been derived and ingested, but thereafter no longer as being obsolete.
non-linearity of odd versus even columns	isaac_twflat_oe, oc_UL_std, oc_UR_std, oc_LL_std, oc_LR_std	- oc_UL_std is sample standard deviation of the upper left quadrant. It is a measure of the non-linearity of the odd columns alone with respect to the linear behavior, taking the odd and the even columns together. Up the 2006-01-01 the complementary ec_UL_mean values have been derived and ingested, but thereafter no longer as being obsolete.

top operational related parameters

Twilight flats are acquired in a relatively short time window of twilight. The registered flux range, the number of raw frames (<=25) and the mean flux level of the calibration OB vary from day to day. Operational related QC parameters measure these parameters.

QC1 parameters

parameter	QC1 database: table, name	procedure
minimum flux	isaac_twflat, qc_twflat_medmin	- Calculate the median flux in each twilight raw frame and calculate the minimum of all n raw flux medians.
maximum flux	isaac_twflat, qc_twflat_medmax	The maximum of all n raw flux medians.
average flux	isaac_twflat, qc_twflat_medavg	The average of all n raw flux medians.
flux standard deviation	isaac_twflat, qc_twflat_medstdev	The standard deviation of all n raw flux medians.
number of raw frames	isaac_twflat, pro_datancom	The number of raw frames is fixed (25) by the template, but can be lower due to manual intervention.