• Q&A of General Interest
  1. fitsverify on a Mac?
  2. Java version for Phase 3 validator
  3. PROV keywords
  4. More on PROV keywords
  5. Missing keyword is not reported
  6. Displaying big files
  7. Visualising 1D spectra
• Q&A: Catalogues
  1. Unique source identifier in VISTA survey catalogues
  2. Multi-band aperture-matched source lists and band-merged catalogue-like products
  3. Band-merged catalogues for the deep survey fields
  4. Light curves
  5. Migrating catalogues from ASCII to FITS format
     
• Q&A: Imaging
  1. Specification of a mask image
  2. VST tile keywords
  3. Imaging with inaccurate calibration
  4. Uncalibrated images
• Q&A: Spectroscopy
  1. Definition of keyword TOT_FLUX (total flux)
  2. Flux scale error keyword definition
  3. Statistical error in spectral coordinates
  4. Systematic error in spectral coordinates
  5. Positioning of the TFIELDS keyword
  6. Spectral Resolving Power (not resolution)
  7. Specification of a 2D spectrum
  8. Variable length arrays are not permitted
  9. Flux error units: % is not permitted
  10. Computing EXPTIME and MJD-END of SOFI spectra
  11. FILTER keyword in spectroscopic data

Q: Is the compiled version of fitsverify available from the Phase 3 web pages running on Mac OS X?

A: the version of fitsverify is compiled for the Linux operating system. One may download the source code from:

http://heasarc.gsfc.nasa.gov/docs/software/ftools/fitsverify/

and compile it on Mac OS X. We cannot assure though that everything would work, as it was not tested.

Q: Which environment does validator.jar need to run? Which Java version?

A: the validator has been developed and tested under Scientific Linux 5.3 (based on Red Hat Enterprise 5). It requires Java version jdk1.6.0_18 or higher. The validator is expected to run under Linux; other platforms are not supported at the moment.

Q: In the example header provided in the user documention the PROV refer to the archive file names, e.g. PROV1 = 'VCAM.2010-09-29T05:55:14.101.fits' while ours is PROV0001 = 'v20100928_00514.fit[1]'. I understand that it makes more sense for you the first one, but can you confirm this point for me please? I.e. we have to change PROV0001 to PROV1 and also look up the name of the ARCFILE in our provenance file.

A: Regarding the PROV* keywords, it is correct that the ESO/EDP standard differs from the convention used at CASU. Specifically, the ESO/EDP data standard mandates that

• the index i (i=1,..,N) of the indexed keyword PROVi is not zero-padded;
• the values of PROVi are pointers to files rather than pointers to FITS extensions, i.e. no trailing extension number in square brackets;
• the PROVi keywords reside in the primary HDU of FITS file.

Generally, the PROV pointers should point to files being archived at ESO. In the case of ESO public imaging surveys this can be either the original raw files using the ESO archive identifier like 'VCAM.2010-09-29T05:55:14.101.fits' or other product files being submitted to ESO for archival and publication.

Examples

VHS is expected to deliver tiles as primary survey products. If the PI/Co-I's choose not to submit the intermediate products, let's say pawprints, from which the tiles were created, then the PROVi keywords written in the tiles should point to the original raw files like 'VCAM.2010-09-29T05:55:14.101.fits'. The header example for the VISTA tile given in Sect. 3.2.1 of the ESO/EDP Standard (Issue 3) corresponds to this case.

Otherwise, if, based on an evaluation of the extra scientific merit to be expected, intermediate products (pawprints) are planned to be delivered as well, then the PROVi keywords written in the tiles should point to the pawprints and, in turn, the PROVi keys in the pawprints should point to the original raw files.

In the case of single-band source catalogues extracted per tile, called "source lists" in our terminology, the PROV1 key has to point to the originating tile unconditionally (cf. ESO/EDP Standard, Issue 3, Sect. 3.2.4, p.36).

For the definition of processing provenance in the context of Phase 3, including an extended set of examples, please refer to the ESO/EDP Standard, Issue 3, Sect. 2.4.

Q: I note that the PROV keywords do not seem to be mandatory; if that is the case we do not need to provide them, right?

A: PROV keys were recommended for image data products (tile, pawprint etc.) submitted in 2011. The upgrade of the Phase 3 Infrastructure that took place in May 2012 entails the requirement saying that processing provenance is compulsory metadata for Phase 3 data products delivered thereafter.

Q: In checking the results of the Phase 3 validator (validation.jar file) I noted that the presence of the ELLIPTIC keyword is not checked for. I noticed this in one file which did not have this keyword defined and the validation did not show any error?

A: Regarding the ELLIPTIC keyword, we know that the validator does not catch the lack of this keyword (and other keywords) in all cases even though it is mandatory according to the EDP format standard. We are working on this issue, but, unfortunately, it is not trivial to fix it quickly.

The updated version of the Phase 3 validator (v1.1.1, May 2012) generally improves the checks on FITS keywords and their content to identify problems as early as possible in the validation process. The existence of the ELLIPTIC keyword is now verified for VISTA tiles and pawprints (but still not for single band source lists).

Q: How can I display the 9Gb fits files of the stacked UltraVISTA images?

A: You must have a 64 bit OS to display the images. It will work right away in ds9 if your machine has at least 9GByte of RAM. If not, you can use imcopy in IRAF or the stand-alone utility fitscopy that comes with CFITSIO. Compiling fitscopy is usually as simple as:
./configure
make
make fitscopy
You can then copy out a subsection e.g. as
fitscopy 'filename.fits[6317:42736,7063:37182]' filename.trim.fits
This particular subsection would be about 4 Gbyte (and requires about 4 Gbyte RAM to make); you can of course try something smaller. The format of the subsection is x1:x2,y1:y2 (just like in IRAF).

Q: How can I visualise the 1D spectra stored in binary table format as defined in the ESO science data products standard?

A: Please follow the instructions here.

Sources in the band merged catalogues with duplication/deblending problems should still have a unique identifier.

Q: Please can you clarify if the column is to be called anything in particular and confirm it'll be OK for it to be a LONG INTEGER. Also will the validator.jar need to be updated?

A: There is no Phase 3 requirement for a particular column name. The names must comply with the restrictions specified in ESO SDP Standard, Issue 5, Sect. 5.1.3, keyword TTYPE*. The unified content descriptor (UCD) meta.id;meta.main shall be set to identify the catalogue column that represents the source identifier. Long integer type is ok.

Here is an example:

TTYPE1 = 'SOURCEID'
TFORM1 = 'K '
TCOMM1 = 'Unique source identifier'
TUCD1  = 'meta.id;meta.main'

Note: TNULL1 should not be defined here as the identifier is not nullable by definition. The validator does not need to be updated to accept these data.

Q: What are the differences between multi-band aperture-matched source lists and band-merged catalogue-like products.

A: Here are some useful information on multi-band aperture-matched source lists and band-merged catalogues.

Multi-band aperture-matched source lists (FITS tables) are produced from the single band source lists and are associated with the tiles observed in different bands for one pointing sky position. The astrometric and photometric calibrations for the ra/dec and magnitudes of the sources in these lists are based on night-calibrations. E.g. these are not global calibrations. The multi-band aperture matched source lists cannot be queried for content from the ESO archive; they can be downloaded as fits tables and will refer to the ra/dec sky position of tiles they were produced from.

Multi-band (or band merged) catalogue are released at the milestones for the survey releases. The first Phase 3 submission period for catalogue data resulting from VISTA public surveys occurs between 25 May and 30 June 2012. The calibration of the astrometry and photometry is global, on the whole region covered by the release. The catalogues will be querable for content, differently from the multi-band aperture-matched source lists , i.e. the individual sources in the catalogues will be accessed via the ESO query interface.

Sources in the catalogues may be detected based on the χ² image or other criteria, driven by the scientific goal of the project. They will not come necessarily from matching single band source lists from the CASU data reduction pipeline.

Furthermore catalogues delivered at the survey releases may contain additional data, that do not come from the VISTA observations: for example photo-z or matched information with X-ray or other multi wavelength observations, weak lensing information etc.

Q: What are multi-band aperture-matched source lists and band merged catalogues for the deep survey fields?

A: It deals with the specific properties of deep surveys with respect to the wide area surveys. As illustrated before multi-band aperture matched source lists are different data products from band-merged catalogues.

Multi-band aperture matched source lists may refer to shallower data products than the deep stacked tiles that are obtained from the data collected over the whole period.

In case of the deep fields, the survey teams may decide to release 1 hr or xx hrs deep tiles with associated multi-band aperture-matched source lists, which will be very valuable data-products for multi-epoch observations, for example monitoring AGN variability or SN searches. In this case the information contained in the multi-band aperture matched source lists is not equivalent to the catalogue extracted from the deep stacks, as the variability information is lost in this final catalogue.

About the multi-band aperture-matched source lists: for deep surveys, the distinction between these source lists and the catalogues expected for the survey releases is more difficult to draw. While for wide surveys the scientific quality difference is clear, it is less so for deep surveys.

Example specific for a deep field

A deep field is observed in Z, J, and Ks in a period  with 28 tile-OBs in Z, 21 tile-OBs in J and 21 tile-OBs in Ks successfully executed. The related data products would be 28 tiles in Z, 21 tiles in J and 21 tiles in Ks + weight maps + single band source lists as data products for these observations.

One would then have 21 multi-band aperture matched source lists in Z, J, and Ks computed from the single-band source lists closest in time.

Q: Are the data standards specified for light curves data products? When is the submission date for these products?

A: The ESO External Data Products Standard, Issue 3, Date 22.05.2012, now also covers multi-epoch photometric catalogues (Sect. 4.2.2) to support the Phase 3 delivery of light curves during the May/June 2012 submission period depending on survey progress.

Q: How to convert an ascii file (table) to a fits file?

A: You can perform the following steps:

1. Create a comma separated ascii table. For example if the table (cat1.txt) has n columns separated by white spaces:
   awk '{print $1,","$2,","$3,","$...,","$n-1,","$n}' cat1.txt > cat2.txt
2. Convert the new table (cat2.txt) into a fits file using the stilts tool:
   stilts tcopy cat2.txt cat.fits ifmt=csv ofmt=fits-basic
   Usage:
   stilts <stilts-flags> tcopy ifmt=<in-format> ofmt=<out-format> [in=]<table> [out=]<out-table>
3.  Add the missing header keywords according to the ESO External Data Products Standard.
   

The STIL Tool Set can be downloaded from:

For more information please check the following documentation:

Q: May you please explain how to specify a mask image as an ancillary product?

A: The mask image describes the data quality of an image array by flagging bad pixels using integer numbers >0. For example: bright stars (=1), readout spikes (=2) etc. Mask value =0 indicates 'good' data. The detailed definition of mask values >1 shall be documented in the Phase 3 release description.
The mask image is a FITS file with the same dimension and number of extensions (if any) as the FITS file that contains the image data. The mask has integer data type, e.g. BITPIX = 8 or 16.
The indexed keyword ASSOCi is to be set to the following value:
ASSOCi = 'ANCILLARY.MASK'
In the case of OmegaCAM data,  i shall be greater than 1 given that  ASSOC1 is reserved for ANCILLARY.WEIGHTMAP.

Q: How to fill in the values of the TL_RA, TL_DEC, and TL_OFFAN for VST data?

A: In what follows, the assumption is that the information provided by the VST survey teams in the OB does indeed refer to the nominal survey tile position.

• The position of the tile in terms of TL_RA and TL_DEC is shown in the HIERARCH ESO TEL TARG ALPHA and HIERARCH ESO TEL TARG DELTA keywords. That is, for an OB pointing to a certain coordinate for the acquisition template, the HIERARCH ESO TEL TARG ALPHA/DELTA keywords do not change for each exposure even if these are acquired at relative offsets with respect to the above reference coordinates. The exact coordinates for each exposure are logged in the RA / DEC keywords. The RA/DEC keywords in the image headers contain the precise position on the sky of each individual exposure.
• The unit and format of "HIERARCH ESO TEL TARG ALPHA" and "HIERARCH ESO TEL TARG DELTA" are HHMMSS.TTT and DDMMSS.TTT, respectively.
• The tile orientation is recorded in HIERARCH ESO ADA POSANG. POSANG = 0 means the usual N/S orientation, with North and East being aligned with positive y and x. Within the fixed x-y coordinate system, the North axis rotates clockwise with increasing POSANG towards East, thus following the same sign convention as the position angle on the sky. Hence: TL_OFFAN = -POSANG.

Finally, note that by definition, the TL_* keywords are identical for all data belonging to the same tile.

Q: How to submit image data without accurate astrometric/photometric calibration via Phase 3?

A: Imaging data need to be characterized in terms of their astrometric registration (WCS), photometric scale (PHOTZP) and dynamic range (ABMAGLIM, ABMAGSAT) in order to qualify as Science Data Product according to the ESO/SDP standard.

Taking into account that the accuracy of these calibrations varies depending on the scientific goals for which the data was obtained, the ESO/SDP standard does not specify a fixed level of accuracy for calibrations but allows to encode the quality using specific keywords, namely CRDERi, CSYERi, and PHOTZPER.

It means in practical terms that the image WCS may be defined based on the information given in the raw data if it is infeasible to register the image with respect to an astrometric reference catalogue. Similarly, the photometric scale and depth of the image data may be estimated based on the nominal instrumental characteristics (default instrumental zero points, exposure time calculator) if photometric standards have not been obtained as part of the observation. The larger uncertainty associated with the estimated photometric properties needs to be expressed by setting PHOTZPER to an appropriate value.

Q: How to submit an image without photometric and astrometric calibration?

A: An image without photometric and astrometric calibration can be submitted as an associated file of a main SCIENCE product (e.g. SCIENCE.SPECTRUM for PESSTO). The uncalibrated image is identified in the header of the SCIENCE file with the ASSOCi keyword as follows: ASSOCi = 'ANCILLARY.IMAGE'.

Q: When shall I set the TOT_FLUX keyword to true for spectroscopic observations?

A: The TOT_FLUX keyword shall be set to T (true) if measurements were taken with a slit wide enough to collect all flux from the source or when correction for slit loss were applied to the measurements.

Q: May you please clarify the definition of the FLUXERR keyword for spectroscopic observations?

A: The FLUXERR is a mandatory quantity that provides the uncertainty on the flux scale. It applies to spectroscopic data with FLUXCAL set to 'ABSOLUTE'. In this case, the FLUXERR value is expressed as a percentage with values between 0 and 100, unless its value cannot be determined, in which case the special value -2. shall be used.

If the spectroscopic data are of type FLUXCAL = 'UNCALIBRATED', the FLUXERR keyword shall be set to -1.

Q: May you please clarify the relation between the LAMRMS and the SPEC_ERR keywords for spectroscopic observations?

A: The keyword LAMRMS refer to a well defined measurement on the calibration data whereas the keyword SPEC_ERR refer to an estimate of the error in the science data that might come from LAMRMS or other information.

More precisely, LAMRMS is the root-mean-square of the residuals, defined as:

LAMRMS = sqrt( sum_i(R_i²)/N )

where R_i= residual of wavelength solution for i-th arc line, N = numbers of arc lines = LAMNLIN.

It is an estimator for the uncertainty in each residual under the assumption that all uncertainties are equal and that the model that was subtracted is the correct/appropriate model for the data (so that the errors are random and not systematic).

Identifying a best estimate for SPEC_ERR requires the PI's judgement. In most cases, including UVES and XSHOOTER, the formula SPEC_ERR = LAMRMS / sqrt(LAMNLIN) is a good approximation if the scatter is mainly due to random measurements errors and those errors are similar for all lines.

Note that the value of LAMRMS / sqrt(LAMNLIN) is a lower limit to SPEC_ERR.

Q: May you please clarify the definition of the SPEC_SYE keyword for spectroscopic observations?

A: The keyword SPEC_SYE aims at capturing the systematic error in spectral coordinate and is expressed in nanometers.  Possible sources of systematic error include any residual offset of the wavelength calibration and any error related to broadening of the lines due to the observation setup and the observation conditions.

If systematic uncertainties are known to represent an insignificant contribution to the overall error budget on the wavelength axis, or all the systematic errors have been accounted for in the submitted data product spectrum, then SPEC_SYE can be set to zero.

Q: What is the correct position of the TFIELDS keyword in the binary table extension of a spectroscopic observation?

A: As per the FITS standard, the TFIELDS keyword that specifies the number of fields in each row of the binary table must be the 8^th keyword in the FITS extension containing the binary table.

Please note that the current version of the Science Data Product standard (v5) will require an update to comply to the FITS standard.

Q: Should I provide the resolution or the resolving power for a 1D spectrum?

A: The SPEC_RES FITS keyword must represent the spectral resolving power (defined as Lambda / Delta_Lambda), and not the spectral resolution. SPEC_RES is therefore a dimensionless (floating point) number.

Please note that the definition of the SPEC_RES keyword in paragraph 2.12 of the Science Data Product standard, Issue 5, is to be amended. 

Q: How to submit wavelength calibrated and distortion corrected 2D spectra?

A: A 2D spectrum denotes the 2D array with one axis oriented along the dispersion direction (calibrated in wavelength), the other axis being the spatial dimension, normally oriented along the slit. Wavelength calibrated and distortion corrected 2D spectra can be submitted as associated files of the SCIENCE.SPECTRUM products. They are identified in the headers of the SCIENCE.SPECTRUM files with the ASSOCi keyword as follows: ASSOCi = 'ANCILLARY.2DSPECTRUM'.

Q: Are variable length arrays supported in the FITS binary table of a 1D spectrum?

No, variable length arrays are not supported by Phase3.

The 1D spectrum is serialised in a binary table as a single record, with as many cells as needed (e.g. one for the wavelength, one for the flux, etc.), each cell containing one array of data points, and with arrays across all cells of equal size. Despite being supported by FITS, the usage of variable length arrays is not allowed in Phase3, as such format is not well supported by the existing spectral tools.

Q: Can I express the errors on the flux of a 1d spectrum as a percentage (%)?

No, percentage is not allowed. The values of the error array associated to the flux of a 1d spectrum must be provided in the same units of the values in the flux array.

Q: How to compute the EXPTIME and MJD-END of a SOFI spectrum?

EXPTIME

The EXPTIME value of a Phase 3 product must be set to the total integration time on target. In the case of a single SOFI observation, the correct value can be computed using the following formula:

EXPTIME = DIT * NDIT

where DIT and NDIT are to be found in the raw header, e.g.:

HIERARCH ESO DET DIT = 90.0000000 / Integration Time
HIERARCH ESO DET NDIT = 3 / # of Sub-Integrations

If instead the Phase 3 product is the result of a co-addition of multiple exposures, then obviously the EXPTIME will be the sum of the individual DIT * NDIT values.

MJD-END

The end time of a Phase 3 SOFI 1d spectrum product (MJD-END) must be computed using the following formula:

MJD-END = MJD-OBS of the last raw observation + NDIT * ( DIT + 1.8) / 86400

where the 1.8 seconds accounts for the necessary overheads, and 86400 scales back from seconds to days.

Q: May you please clarify the content of the FILTER keyword in 1D spectra products and in the ancillary 2D spectra?

The FILTER keyword must be dropped from the headers of the ESO 1d spectroscopic data products, or it can be renamed as follows:

Example:

Original data contains:

FILTER  = 'GBF     '            / Filter name

Please rename as follows:

OFILTER = 'GBF     '            / Filter name

The same applies to any ancillary 2d spectra.