• Q&A of General Interest
  1. PROV keywords
  2. More on PROV keywords
  3. Encoding PROV in catalogue data 05.2016 updated
  4. Displaying big files
  5. Visualising 1D spectra
  6. CONTINUE FITS convention not supported 03.2014
  7. Additional OBSTECH values 03.2014
  8. Clarification of the WCS and CDELT requirements 04.2014
  9. NCOMBINE is now mandatory for images and 1d spectra 04.2014
  10. TEXPTIME is now mandatory for images, 1d spectra, and single band source lists 04.2014
  11. Definition of mandatory keyword 03.2014
  12. Programmatic access to archive and original filenames 03.2014
  13. Clarification of the ancillary data content 03.2014
  14. EXPTIME cannot be negative 04.2014
  15. Positioning of ORIGFILE 04.2014
  16. Including data from other facilities (not ESO) 05.2015
  17. Calibration data taken under SCIENCE category 07.2017
  18. Providing a DOI as bibliographic reference 06.2017
  19. Renaming downloaded Phase 3 products to their original file names 04.2018
      
• Q&A: Catalogues
  1. Unique source identifier
     
  2. Multi-band aperture-matched source lists and band-merged catalogue-like products
  3. Band-merged catalogues for the deep survey fields
  4. Light curves
  5. Migrating catalogues from ASCII to FITS format
     
  6. TCOMMi and TUCDi keywords 09.2014
  7. Processing provenance per catalogue record (spectroscopic programs) 02.2015
  8. Mandatory keywords for photometric catalogues using multi-file format 08.2016
• Q&A: Imaging
  1. Specification of a mask image
  2. VST tile keywords
  3. Imaging with inaccurate calibration
  4. Uncalibrated images
  5. NDITHER has become not mandatory for the OmegaCAM products 03.2014
  6. What is the correct EFFRON for median-combined SOFI images? 09.2014
  7. Missing PSF_FWHM image quality parameter 05.2015
  8. Definition of EPS_REG keyword for VISTA Public Surveys 04.2016
• Q&A: Spectroscopy
  1. Converting a spectrum to the Phase 3 required binary table format 08.11.2017
     
  2. Definition of keyword TOT_FLUX (total flux)
  3. Flux scale error keyword definition
  4. Statistical error in spectral coordinates
  5. Systematic error in spectral coordinates
  6. Positioning of the TFIELDS keyword
  7. Spectral Resolving Power (not resolution)
  8. Specification of a 2D spectrum
  9. Variable length arrays are not permitted in spectra 04.2014
  10. Flux error units: "%" is not permitted
  11. Computing EXPTIME and MJD-END for SOFI spectra
  12. BUNIT keyword in spectroscopic data 04.2014
  13. Extended object keyword definition 03.2014
  14. DISPELEM, OBSTECH values 08.2016
  15. Encoding quality information 03.2014
  16. Monotonicity of the wavelength array 03.2014
  17. Off source raw exposures 03.2014
  18. Definition of WAVELMIN and WAVELMAX 03.2014
  19. Clarification about CONTNORM 02.2015
  20. Preparing APEX 1-D spectra compliant to the Phase3 standard 03.2017
  21. Air wavelength vs. vacuum wavelength (1D spectrum data format) 07.2017
• Q&A: Data Cubes (Integral Field Spectroscopy)
  1. Associating Imaging Data to Data Cubes 13.06.2018
  2. How are exposure maps defined in the SDP standard? 17.09.2018 new
  3. How is the variation of the number of contributing exposures defined in the SDP standard? 17.09.2018 new
  4. Clarification of requirements for the OBJECT keyword 14.11.2018 new
• Typos
  1. See the list of typos in the SDP standard, v.5 03.2014

Q: In the example header provided in the user documention the PROV refer to the archive file names, e.g. PROV1 = 'VCAM.2010-09-29T05:55:14.101.fits' while ours is PROV0001 = 'v20100928_00514.fit[1]'. I understand that it makes more sense for you the first one, but can you confirm this point for me please? I.e. we have to change PROV0001 to PROV1 and also look up the name of the ARCFILE in our provenance file.

A: Regarding the PROV* keywords, it is correct that the ESO/EDP standard differs from the convention used at CASU. Specifically, the ESO/EDP data standard mandates that

• the index i (i=1,..,N) of the indexed keyword PROVi is not zero-padded;
• the values of PROVi are pointers to files rather than pointers to FITS extensions, i.e. no trailing extension number in square brackets;
• the PROVi keywords reside in the primary HDU of FITS file.

Generally, the PROV pointers should point to files being archived at ESO. In the case of ESO public imaging surveys this can be either the original raw files using the ESO archive identifier like 'VCAM.2010-09-29T05:55:14.101.fits' or other product files being submitted to ESO for archival and publication.

Examples

VHS is expected to deliver tiles as primary survey products. If the PI/Co-I's choose not to submit the intermediate products, let's say pawprints, from which the tiles were created, then the PROVi keywords written in the tiles should point to the original raw files like 'VCAM.2010-09-29T05:55:14.101.fits'. The header example for the VISTA tile given in Sect. 3.2.1 of the ESO/EDP Standard (Issue 3) corresponds to this case.

Otherwise, if, based on an evaluation of the extra scientific merit to be expected, intermediate products (pawprints) are planned to be delivered as well, then the PROVi keywords written in the tiles should point to the pawprints and, in turn, the PROVi keys in the pawprints should point to the original raw files.

In the case of single-band source catalogues extracted per tile, called "source lists" in our terminology, the PROV1 key has to point to the originating tile unconditionally (cf. ESO/EDP Standard, Issue 3, Sect. 3.2.4, p.36).

For the definition of processing provenance in the context of Phase 3, including an extended set of examples, please refer to the ESO/EDP Standard, Issue 3, Sect. 2.4.

Q: I note that the PROV keywords do not seem to be mandatory; if that is the case we do not need to provide them, right?

A: PROV keys were recommended for image data products (tile, pawprint etc.) submitted in 2011. The upgrade of the Phase 3 Infrastructure that took place in May 2012 entails the requirement saying that processing provenance is compulsory metadata for Phase 3 data products delivered thereafter.

Q: How to encode processing provenance (PROV) of catalogue data being submitted at the same time as the original images?

A: The processing provenance shall be specified within the catalogue files in terms of original filenames of the imaging data (a.k.a. ORIGFILE), as usual, for example:

PROV1 = 'vmc_er1_05h59-066d20_tile_j_deepimage_1403891.fits.fz'

PROV2 = 'vmc_er1_05h59-066d20_tile_ks_deepimage_1403943.fits.fz'

Catalog and imaging data can be submitted using separate Phase 3 batches (within the same Data Collection). To allow proper resolution of PROV references the imaging data must be submitted before the catalog data can be successfully validated.

Q: How can I display the 9Gb fits files of the stacked UltraVISTA images?

A: You must have a 64 bit OS to display the images. It will work right away in ds9 if your machine has at least 9GByte of RAM. If not, you can use imcopy in IRAF or the stand-alone utility fitscopy that comes with CFITSIO. Compiling fitscopy is usually as simple as:
./configure
make
make fitscopy
You can then copy out a subsection e.g. as
fitscopy 'filename.fits[6317:42736,7063:37182]' filename.trim.fits
This particular subsection would be about 4 Gbyte (and requires about 4 Gbyte RAM to make); you can of course try something smaller. The format of the subsection is x1:x2,y1:y2 (just like in IRAF).

Q: How can I visualise the 1D spectra stored in binary table format as defined in the ESO science data products standard?

A: Please follow the instructions here.

Q: Can I use the CONTINUE FITS convention for values exceeding 68 characters?

A: No, the CONTINUE FITS convention for values exceeding 68 characters is not supported by ESO. Please remove offending keywords TRACE1 and ARC and the CONTINUE lines as well as the LONGSTRN related keywords.

Q: May you please clarify what the OBSTECH keyword values are?

A: We support the OBSTECH keywords listed in the table below, in addition to those listed in the SDP standard document

Q: May you please clarify what the WCS requirements are?

A:

• CDELTn and PCij: the SDP standard does not allow their usage. Please use the CD matrix.
  
• The WCS definition must be unique. Redundant and potentially conflicting keyword definitions will be rejected.

Q: Is NCOMBINE a mandatory keyword?

A: NCOMBINE in the primary header unit is mandatory for images and 1D spectra.

Q: Is TEXPTIME a mandatory keyword?

A: TEXPTIME is mandatory for: images (IMAGE, MEFIMAGE), 1D spectra (SPECTRUM), source lists (SRCTBL), and IFS data cubes (CUBE.IFS).

Q: Can you please explain what a mandatory keyword means?

A: For keywords of type 'string' it means that both the keyword shall be present and its value shall not be blank (except for REFERENC if a refereed publication is not available at the time of the data release going public). For keywords of type 'float' or 'integer', it means that the FITS card shall not be incomplete and that a valid value shall be provided.

Q: Is there a way to query the archive for the original filenames of the data?

A: Yes, please find more information about programmatic access at:
http://archive.eso.org/wdb/wdb/adp/phase3_main/query

A typical query is listed below:

http://archive.eso.org/wdb/wdb/adp/phase3_main/query?wdbo=html%2fdisplay&max_rows_returned=200&phase3_program=Any&tab_phase3_collection=on&phase3_collection=Any&tab_release_tag=on&release_tag=&tab_prog_id=on&prog_id=&pi_name=&p3orig=%25&target=&&tab_rel_date=on&rel_date=&tab_publication_date=on&publication_date=&tab_origfile=on&origfile=&tab_dp_id=on&dp_id=&tab_referenc=on&referenc=&order_main=&extra_columns=more&extra_columns=HDR&

Q: Which kind of data can be provided in an associated ancillary file?

A: An ancillary data product provided together with a data release should contain additional data that is not included in the science data product itself. The ancillary file should not replicate information present elsewhere. In particular keep in mind that Phase 3 generally mandates that the data documentation is to be provided in the release-description.pdf file. Referential information and other metadata must be defined in the header of the science file in terms of FITS keywords. ESO/ASG has the overall responsibility to assess if provided ancillary data qualify for ingestion into the ESO science archive facility.

Q: Can EXPTIME assume negative values?

A: No, it is no more allowed to set EXPTIME to a negative value.

Q: Can ORIGFILE be present in FITS extensions?

A: No, ORIGFILE must be present in the FITS primary header unit, and must not be present in FITS extensions to avoid potential inconsistencies.

Q: We like to include in our catalogue some data originating from a non-ESO facility. Is there a flag to indicate such products?

A: Please use the dedicated boolean keyword NOESODAT in the primary header to indicate that the product includes non-ESO data:

NOESODAT = T           / includes non-ESO data

Note: if the catalogue is predominantly based on ESO data then the Phase 3 keywords TELESCOP, INSTRUME and PROV should record the provenance of the ESO data component only.

Q: May you please clarify whether calibration data taken with a DPR CATG ='SCIENCE' should be taken into account in the set up of the MJD-OBS, MJD-END, T/EXPTIME, NCOMBINE, TELAPSE, and PROVi keywords?

A: It can happen that some calibration data have been acquired with a DPR.CATG = 'SCIENCE'. Those calibration exposures should be excluded from the calculation of the MJD-OBS, MJD-END, T/EXPTIME, NCOMBINE, and TELAPSE keyword values. The list of PROVi keywords recording the science files originating the data product should not contain any reference to those calibration files.

Example where this situation happens:

• XSHOOTER: the OFFSET frames with DPR.TECH = 'SKY' (even though taken as DPR.CATG = 'SCIENCE') should NOT be taken into account, while the ones with DPR.TECH = 'OBJECT' are the real science data.
  

Q: Can I provide a DOI as value for the REFERENC header keyword?

A: Both the 19-digit bibliographic identifier used in the Astrophysics Data System bibliographic database or the Digital Object Identifier are valid bibliographic reference information to point to the primary publication associated to a data product.
If you provide the DOI, the link should be given as in the following example:

REFERENC= 'doi:10.1051/0004-6361/201730605' / Bibliographic reference

while in the case you provide the BibCode, the link should be given as follows:

REFERENC= ' 1924MNRAS..84..308E' / Bibliographic reference

Q: How can I rename the P3 products from their ADP names to the original file names?

A: In the case of files downloaded via the Request Handler, the correspondence between file-id and original file-name is mapped in the README file associated to each data request. The renaming commands can be extracted directly from the README file using the following shall command:

awk '/\- ADP/{ print "mv "$2, $3}' README_123456.txt

In some cases it can happen that during the downloading the ':' of the file name is replaced by a '_',
if so, please use this other command line:

awk '/\- ADP/{ print "mv "$2, $3}' README_123456.txt | awk '{gsub(/\:/, "_", $2)}1'

Errata for §5.2.1.1: the IAU naming convention provided in the SDP standard is amended to:

Every source catalogue must have a unique source identifier, defined according to the IAU recommendations for nomenclature using the schema16

<prefix> J<hhmmss.ss+-ddmmss.s>

or, alternatively,

<prefix> J<hhmmss.sss+-ddmmss.ss>

where prefix denotes the survey programme.

In case you cannot use the IAU naming convention because of duplication/deblending problems, then please be aware of what follows.

Sources in the band merged catalogues with duplication/deblending problems should still have a unique identifier.

Q: Please can you clarify if the column is to be called anything in particular and confirm it'll be OK for it to be a LONG INTEGER.

A: There is no Phase 3 requirement for a particular column name. The names must comply with the restrictions specified in ESO SDP Standard, Issue 5, Sect. 5.1.3, keyword TTYPE*. The unified content descriptor (UCD) meta.id;meta.main shall be set to identify the catalogue column that represents the source identifier. Long integer type is ok.

Here is an example:

TTYPE1 = 'SOURCEID'
TFORM1 = 'K '
TCOMM1 = 'Unique source identifier'
TUCD1  = 'meta.id;meta.main'

Note: TNULL1 should not be defined here as the identifier is not nullable by definition.

Q: What are the differences between multi-band aperture-matched source lists and band-merged catalogue-like products.

A: Here are some useful information on multi-band aperture-matched source lists and band-merged catalogues.

Multi-band aperture-matched source lists (FITS tables) are produced from the single band source lists and are associated with the tiles observed in different bands for one pointing sky position. The astrometric and photometric calibrations for the ra/dec and magnitudes of the sources in these lists are based on night-calibrations. E.g. these are not global calibrations. The multi-band aperture matched source lists cannot be queried for content from the ESO archive; they can be downloaded as fits tables and will refer to the ra/dec sky position of tiles they were produced from.

Multi-band (or band merged) catalogue are released at the milestones for the survey releases. The first Phase 3 submission period for catalogue data resulting from VISTA public surveys occurs between 25 May and 30 June 2012. The calibration of the astrometry and photometry is global, on the whole region covered by the release. The catalogues will be querable for content, differently from the multi-band aperture-matched source lists , i.e. the individual sources in the catalogues will be accessed via the ESO query interface.

Sources in the catalogues may be detected based on the χ² image or other criteria, driven by the scientific goal of the project. They will not come necessarily from matching single band source lists from the CASU data reduction pipeline.

Furthermore catalogues delivered at the survey releases may contain additional data, that do not come from the VISTA observations: for example photo-z or matched information with X-ray or other multi wavelength observations, weak lensing information etc.

Q: What are multi-band aperture-matched source lists and band merged catalogues for the deep survey fields?

A: It deals with the specific properties of deep surveys with respect to the wide area surveys. As illustrated before multi-band aperture matched source lists are different data products from band-merged catalogues.

Multi-band aperture matched source lists may refer to shallower data products than the deep stacked tiles that are obtained from the data collected over the whole period.

In case of the deep fields, the survey teams may decide to release 1 hr or xx hrs deep tiles with associated multi-band aperture-matched source lists, which will be very valuable data-products for multi-epoch observations, for example monitoring AGN variability or SN searches. In this case the information contained in the multi-band aperture matched source lists is not equivalent to the catalogue extracted from the deep stacks, as the variability information is lost in this final catalogue.

About the multi-band aperture-matched source lists: for deep surveys, the distinction between these source lists and the catalogues expected for the survey releases is more difficult to draw. While for wide surveys the scientific quality difference is clear, it is less so for deep surveys.

Example specific for a deep field

A deep field is observed in Z, J, and Ks in a period  with 28 tile-OBs in Z, 21 tile-OBs in J and 21 tile-OBs in Ks successfully executed. The related data products would be 28 tiles in Z, 21 tiles in J and 21 tiles in Ks + weight maps + single band source lists as data products for these observations.

One would then have 21 multi-band aperture matched source lists in Z, J, and Ks computed from the single-band source lists closest in time.

Q: Are the data standards specified for light curves data products? When is the submission date for these products?

A: The ESO External Data Products Standard, Issue 3, Date 22.05.2012, now also covers multi-epoch photometric catalogues (Sect. 4.2.2) to support the Phase 3 delivery of light curves during the May/June 2012 submission period depending on survey progress.

Q: How to convert an ascii file (table) to a fits file?

A: You can perform the following steps:

1. Create a comma separated ascii table. For example if the table (cat1.txt) has n columns separated by white spaces:
   awk '{print $1,","$2,","$3,","$...,","$n-1,","$n}' cat1.txt > cat2.txt
2. Convert the new table (cat2.txt) into a fits file using the stilts tool:
   stilts tcopy cat2.txt cat.fits ifmt=csv ofmt=fits-basic
   Usage:
   stilts <stilts-flags> tcopy ifmt=<in-format> ofmt=<out-format> [in=]<table> [out=]<out-table>
3.  Add the missing header keywords according to the ESO External Data Products Standard.
   

The STIL Tool Set can be downloaded from:

For more information please check the following documentation:

Q: Are TCOMMi and TUCDi keywords required in the FITS header of catalogue data files?

A: Yes, to successfully pass Phase 3 validation each catalogue data file must include the indexed keywords TCOMMi and TUCDi, one for each catalogue column. Note that Table 15 of the ESO Science Data Products Standard, Issue 5, is lacking these keywords although they are required - an inconsistency which will be corrected in a forthcoming version of the document.

Q: Spectroscopic programmes like Gaia-ESO, PESSTO or zCOSMOS produce high-level results in the form of catalogues where each record contains the results of the analysis of one reduced spectrum (which is a Phase 3 product itself). How to encode the links between catalogue record and original spectra?

A: The catalogue must contain a column with a ORIGFILE or ARCFILE reference that identifies for each row the 1D spectrum from which the catalogued parameters are measured.

ORIGFILE refers to the filenames of the 1D spectra within the current release, ARCFILE refers to the ESO archive identifier of the form ADP.<timestamp>. The referenced ORIGFILE may be part of a previously submitted Phase 3 batch.

The special keyword TXLNKi = ORIGFILE (or ARCFILE) must be defined in the catalog file.

If the processing provenance has been defined in this way, i.e. per catalogue record, then the PROVi keywords are not applicable and should not be included.

Example:

Header listing for HDU #1:

PRODCATG= 'SCIENCE.CATALOG'

(..)

Header listing for HDU #2:

XTENSION= 'BINTABLE'

EXTNAME = 'PHASE3CATALOG'

(..)

TTYPE8 = 'FILENAME'

TFORM8 = '52A '

TCOMM8 = 'FITS file name of the original spectrum'

TUCD8  = 'meta.ref'

TXLNK8 = 'ORIGFILE' / Data link to the original spectrum

(..)

Q: Are the keywords listed in 5.1.2.1 of the ESO Data Products Standard mandatory only for the metadata file for catalogues in multi-file catalogue format?

A: No, now FILTER, FILTERi and REFERENC keywords are also required in the headers of the catalogue data files. In this way users can query catalogues tiles also selecting a specific filter. They will be returned as query results showing the bibliographic code of the related publication, enhancing its visibility.

Moreover, if possible, we encourage to define also values for the header keywords MAGLIMi, 5-sigma limiting AB magnitude for point sources, for such products, using the same indexing as for FILTERi

Example:

FILTER  = MULTI
FILTER1 = 'Z' / Filter name
MAGLIM1 = 20.53562164306641 / 5-sigma point source limiting AB mag
FILTER2 = 'Y' / Filter name
MAGLIM2 = 20.16515922546387 / 5-sigma point source limiting AB mag
FILTER3 = 'J' / Filter name
MAGLIM3 = 19.83752250671387 / 5-sigma point source limiting AB mag
FILTER4 = 'H' / Filter name
MAGLIM4 = 19.53595924377441 / 5-sigma point source limiting AB mag
FILTER5 = 'Ks' / Filter name
MAGLIM5 = 19.46832275390625 / 5-sigma point source limiting AB mag

Q: May you please explain how to specify a mask image as an ancillary product?

A: The mask image describes the data quality of an image array by flagging bad pixels using integer numbers >0. For example: bright stars (=1), readout spikes (=2) etc. Mask value =0 indicates 'good' data. The detailed definition of mask values >1 shall be documented in the Phase 3 release description.
The mask image is a FITS file with the same dimension and number of extensions (if any) as the FITS file that contains the image data. The mask has integer data type, e.g. BITPIX = 8 or 16.
The keyword PRODCATG is to be set in the mask image:
PRODCATG = 'ANCILLARY.MASK'

Q: How to fill in the values of the TL_RA, TL_DEC, and TL_OFFAN for VST data?

A: In what follows, the assumption is that the information provided by the VST survey teams in the OB does indeed refer to the nominal survey tile position.

• The position of the tile in terms of TL_RA and TL_DEC is shown in the HIERARCH ESO TEL TARG ALPHA and HIERARCH ESO TEL TARG DELTA keywords. That is, for an OB pointing to a certain coordinate for the acquisition template, the HIERARCH ESO TEL TARG ALPHA/DELTA keywords do not change for each exposure even if these are acquired at relative offsets with respect to the above reference coordinates. The exact coordinates for each exposure are logged in the RA / DEC keywords. The RA/DEC keywords in the image headers contain the precise position on the sky of each individual exposure.
• The unit and format of "HIERARCH ESO TEL TARG ALPHA" and "HIERARCH ESO TEL TARG DELTA" are HHMMSS.TTT and DDMMSS.TTT, respectively.
• The tile orientation is recorded in HIERARCH ESO ADA POSANG. POSANG = 0 means the usual N/S orientation, with North and East being aligned with positive y and x. Within the fixed x-y coordinate system, the North axis rotates clockwise with increasing POSANG towards East, thus following the same sign convention as the position angle on the sky. Hence: TL_OFFAN = -POSANG.

Finally, note that by definition, the TL_* keywords are identical for all data belonging to the same tile.

Q: How to submit image data without accurate astrometric/photometric calibration via Phase 3?

A: Imaging data need to be characterized in terms of their astrometric registration (WCS), photometric scale (PHOTZP) and dynamic range (ABMAGLIM, ABMAGSAT) in order to qualify as Science Data Product according to the ESO/SDP standard.

Taking into account that the accuracy of these calibrations varies depending on the scientific goals for which the data was obtained, the ESO/SDP standard does not specify a fixed level of accuracy for calibrations but allows to encode the quality using specific keywords, namely CRDERi, CSYERi, and PHOTZPER.

It means in practical terms that the image WCS may be defined based on the information given in the raw data if it is infeasible to register the image with respect to an astrometric reference catalogue. Similarly, the photometric scale and depth of the image data may be estimated based on the nominal instrumental characteristics (default instrumental zero points, exposure time calculator) if photometric standards have not been obtained as part of the observation. The larger uncertainty associated with the estimated photometric properties needs to be expressed by setting PHOTZPER to an appropriate value.

Q: How to submit an image without photometric and astrometric calibration?

A: An image without photometric and astrometric calibration can be submitted as an associated file of a main SCIENCE product (e.g. an acquisition image associated to a SCIENCE.SPECTRUM).

The uncalibrated image must include the following keyword definition within the FITS header:

PRODCATG = 'ANCILLARY.IMAGE'

Q: Could you please clarify the status of the NDITHER header keyword for OmegaCAM products?

A: NDITHER shall not be mandatory any more for OmegaCAM products.

Q: What is the correct EFFRON for median-combined SOFI images?

Example: There are 7 raw images, each resulting from averaging together 5 detector integrations (NDIT = 5). A science product is generated by reducing and median-combining those 7 raw images.

In this case:

EFFRON = 12 * sqrt(PI/2) / sqrt( 7 * 5)

where PI is 3.14159, and 12 is the detector readout noise of SOFI in electrons.

Q: Using the MEFIMAGE format in case of chip-by-chip processing of OmegaCAM data it occasionally happens that the PSF_FWHM quality parameter cannot be determined for one of the chips, normally due to the lack of suitable point-like reference objects just in this field. The PSF_FWHM recorded with the other detectors demonstrates that the overall image quality appears to be ok. How to define PSF_FWHM for the problematic detector chip (MEFIMAGE format)?

A: The MEFIMAGE format requries PSF_FWHM to be defined separately for each detector, i.e. image extension. The valid range is PSF_FWHM > 0. If PSF_FWHM cannot be determined for one of the detectors then the special value -1 should be assigned to PSF_FWHM in the respective FITS extension:

PSF_FWHM= -1

Note: the PSF_FWHM special value is not supported for the SCIENCE.IMAGE format i.e. for resampled survey tile images.

Q: How to assign a value to the EPS_REG keyword?

A: The EPS_REG keyword should uniquely refer to the name of the survey region or area. The keyword values are defined as follows:

As an example, the FITS headers for the VIDEO survey are defined as follows:

EPS_REG = 'VIDEO/CDFS'    / ESO public survey region name

EPS_REG = 'VIDEO/ES1'     / ESO public survey region name

EPS_REG = 'VIDEO/XMM'     / ESO public survey region name

Note: to identify individual survey fields, the OBJECT keyword should be used instead. Examples are:

OBJECT = 'VIDEO/ES1-North ' / Target designation
OBJECT = 'VIDEO/ES1-South ' / Target designation
OBJECT = 'VIDEO/CDFS2'      / Target designation
OBJECT = 'VIDEO/XMM3'       / Target designation

Q: How can I convert my spectrum to the binary table format required by tha Phase 3 Standard?

A: You can use the following Python script (requirements: Python-2.7 or greater, astropy library) but please make sure to modify it by providing the correct values for the required FITS header keywords to be added to the file. To help the users the script already includes the commands to add all the Phase 3 mandatory keywords but when the value must be inserted by the user, the line is commented. Please provide the correct value and de-comment the line in order to generate a Phase 3 complaint file.

• In case the original spectrum is an image with the wavelength in the WCS, use this script. You can give it a try by using this input file. The script should output the following converted spectrum.

Q: When shall I set the TOT_FLUX keyword to true for spectroscopic observations?

A: The TOT_FLUX keyword shall be set to T (true) if measurements were taken with a slit wide enough to collect all flux from the source or when correction for slit loss was applied to the measurements.

Q: May you please clarify the definition of the FLUXERR keyword for spectroscopic observations?

A: The FLUXERR is a mandatory quantity that provides the uncertainty on the flux scale. It applies to spectroscopic data with FLUXCAL set to 'ABSOLUTE'. In this case, the FLUXERR value is expressed as a percentage with values between 0 and 100, unless its value cannot be determined, in which case the special value -2. shall be used.

If the spectroscopic data are of type FLUXCAL = 'UNCALIBRATED', the FLUXERR keyword shall be set to -1.

Q: May you please clarify the relation between the LAMRMS and the SPEC_ERR keywords for spectroscopic observations?

A: The keyword LAMRMS refer to a well defined measurement on the calibration data whereas the keyword SPEC_ERR refer to an estimate of the error in the science data that might come from LAMRMS or other information.

More precisely, LAMRMS is the root-mean-square of the residuals, defined as:

LAMRMS = sqrt( sum_i(R_i²)/N )

where R_i= residual of wavelength solution for i-th arc line, N = numbers of arc lines = LAMNLIN.

It is an estimator for the uncertainty in each residual under the assumption that all uncertainties are equal and that the model that was subtracted is the correct/appropriate model for the data (so that the errors are random and not systematic).

Identifying a best estimate for SPEC_ERR requires the PI's judgement. In most cases, including UVES and XSHOOTER, the formula SPEC_ERR = LAMRMS / sqrt(LAMNLIN) is a good approximation if the scatter is mainly due to random measurements errors and those errors are similar for all lines.

Note that the value of LAMRMS / sqrt(LAMNLIN) is a lower limit to SPEC_ERR.

Q: May you please clarify the definition of the SPEC_SYE keyword for spectroscopic observations?

A: The keyword SPEC_SYE aims at capturing the systematic error in spectral coordinate and is expressed in nanometers.  Possible sources of systematic error include any residual offset of the wavelength calibration and any error related to broadening of the lines due to the observation setup and the observation conditions.

If systematic uncertainties are known to represent an insignificant contribution to the overall error budget on the wavelength axis, or all the systematic errors have been accounted for in the submitted data product spectrum, then SPEC_SYE can be set to zero.

Q: What is the correct position of the TFIELDS keyword in the binary table extension of a spectroscopic observation?

A: As per the FITS standard, the TFIELDS keyword that specifies the number of fields in each row of the binary table must be the 8^th keyword in the FITS extension containing the binary table.

Please note that the current version of the Science Data Product standard (v5) will require an update to comply to the FITS standard.

Q: Should I provide the resolution or the resolving power for a 1D spectrum?

A: The SPEC_RES FITS keyword must represent the spectral resolving power (defined as Lambda / Delta_Lambda), and not the spectral resolution. SPEC_RES is therefore a dimensionless (floating point) number.

Please note that the definition of the SPEC_RES keyword in paragraph 2.12 of the Science Data Product standard, Issue 5, is to be amended. 

Q: How to submit wavelength calibrated and distortion corrected 2D spectra?

A: A 2D spectrum denotes the 2D array with one axis oriented along the dispersion direction (calibrated in wavelength), the other axis being the spatial dimension, normally oriented along the slit. Wavelength calibrated and distortion corrected 2D spectra can be submitted as associated files of the SCIENCE.SPECTRUM products. They must include the following keyword definition within the FITS header:

PRODCATG = 'ANCILLARY.2DSPECTRUM'

Q: Are variable length arrays supported in the FITS binary table of a 1D spectrum?

No, variable length arrays are not supported in the spectral case.

The 1D spectrum is serialised in a binary table as a single record, with as many cells as needed (e.g. one for the wavelength, one for the flux, etc.), each cell containing one array of data points, and with arrays across all cells of equal size. Despite being supported by FITS, the usage of variable length arrays is not allowed in Phase3, as such format is not well supported by the existing spectral tools.

Please make sure that:

PCOUNT = 0
GCOUNT = 1

Q: Can I express the errors on the flux of a 1d spectrum as a percentage (%)?

No, percentage is not allowed. The values of the error array associated to the flux of a 1d spectrum must be provided in the same units of the values in the flux array.

Q: How to compute the EXPTIME and MJD-END of a SOFI spectrum?

EXPTIME

The EXPTIME value of a Phase 3 product must be set to the total integration time on target. In the case of a single SOFI observation, the correct value can be computed using the following formula:

EXPTIME = DIT * NDIT

where DIT and NDIT are to be found in the raw header, e.g.:

HIERARCH ESO DET DIT = 90.0000000 / Integration Time
HIERARCH ESO DET NDIT = 3 / # of Sub-Integrations

If instead the Phase 3 product is the result of a co-addition of multiple exposures, then obviously the EXPTIME will be the sum of the individual DIT * NDIT values.

MJD-END

The end time of a Phase 3 SOFI 1d spectrum product (MJD-END) must be computed using the following formula:

MJD-END = MJD-OBS of the last raw observation + NDIT * ( DIT + 1.8) / 86400

where the 1.8 seconds accounts for the necessary overheads, and 86400 scales back from seconds to days.

Q: Can I use BUNIT to specify the unit and a scaling factor for a 1d spectrum?

The BUNIT keyword must be dropped from the headers of the ESO 1d spectroscopic data products. To provide unit and a scaling factor please use the relevant TUNITn keyword(s).

Q: Q: May you please clarify the definition of the EXT_OBJ keyword for spectroscopic observations?

The EXT_OBJ keyword is mandatory for External Data Products, e.g. Spectroscopic Public Surveys.

The keyword EXT_OBJ is not applicable to XSHOOTER Internal Data Products, i.e. the keyword must not be listed in the header. It means the property is unknown.

For GIRAFFE:

EXT_OBJ  = F            / True if extended object

For pointlike UVES/Echelle data:

EXT_OBJ  = F            / True if extended object

Q: May you please explicitly list the possible keywords values that a spectrum taken with a specific instrument can take?

A: The table below shows some examples of the possible values for the keywords TELESCOP, OBSTECH and DISPELEM. If you still have doubts or the instrument you are looking for is not listed, do not hesitate to contact us.

* When combined

Q: May you please explain how to encode quality information in 1D spectra?

A: Relaxing the restrictions given in SDP standard v5, page 53, the data quality of each pixel is encoded in terms of the sum of powers of 2. Each 'bit' corresponds to a quality condition. Quality default to 0, i.e. good data.

Q: May you please clarify the properties of the wavelength array for 1D spectra?

A: The sequence of elements of the WAVE vector for 1D spectra must be in ascending order, i.e. strictly increasing. Same for the FREQ and ENER vectors.

Q: May you please clarify whether off source raw exposures should be taken into account in the set up of the EXPTIME, NCOMBINE, MJD-OBS, MJD-END and PROVi keywords for 1D spectra?

A: Off source raw exposures should be excluded from the calculation of the EXPTIME, NCOMBINE, MJD-OBS, and MJD-END keywords values. The list of PROVi keywords should not contain any reference to off source raw exposures.

Q: May you please clarify the definition of the WAVELMIN and WAVELMAX keywords on page 20, section 2.11 of the SDP standard, v.5?

A: For 1-D extracted spectra (SCIENCE.SPECTRUM):

WAVELMIN and WAVELMAX should describe the minimum and the maximum wavelengths in nanometers outside of which the described spectrum does not carry any valuable scientific information. For example, if a spectrum is zero-padded on the lower wavelengths, then the WAVELMIN should not be lower than the wavelength associated to the first bin where the spectrum has a flux > 0.

Of course, WAVELMIN cannot be lower than SPECTRUM.WAVE[1], nor higher than SPECTRUM.WAVE[NELEM].

Similarly, the WAVELMAX should provide the wavelength value of the last useful wavelength bin after which the spectrum is known not to carry any scientific value.

It must be true that:

SPECTRUM.WAVE[1] <= WAVELMIN < WAVELMAX <= SPECTRUM.WAVE[NELEM]

where NELEM is the pixel length of the wavelength array.

For 3-D data cubes (type "CUBE.IFS") WAVELMIN and WAVELMAX correspond to the physical wavelenth (nm) of the first and last plane of the science data cube carrying scientifically useful information.

Q: Which value should take the keyword CONTNORM in the case I provide two flux arrays, one normalized and the other not?

A: If multiple flux arrays are provided, then
CONTNORM = T
and the Spectral Data Model v2.0 applies (VOCLASS = 'SPECTRUM V2.0').
For example, if field 2 represents the continuum-normalised spectrum (field 3 its error)
and field 4 the non-normalised spectrum (field 5 its error), the TUTYPn are:

TUTYP1  = 'spec:Data.SpectralAxis.Value' / 'IVOA data model element for field 1'
TUTYP2  = 'spec:Data.FluxAxis.Value' / 'IVOA data model element for field 2'
TUTYP3  = 'spec:Data.FluxAxis.Accuracy.StatError' / 'IVOA data model element for
TUTYP4  = 'eso:Data.FluxAxis.Value' / 'ESO element for field 6'
TUTYP5  = 'eso:Data.FluxAxis.Accuracy.StatError' / 'ESO element for field 7'

Moreover the UCD must indicate which one is the continuum-normalised spectrum using the arith.ratio token
and indicate which one is the main one, by placing the meta.main token
at the end of the TUCD2 and TUCD3:

TUCD2 = 'phot.flux.density;em.wl;arith.ratio;meta.main'
TUTYP2 = 'spec:Data.FluxAxis.Value'

TUCD3 = 'stat.error;phot.flux.density;meta.main'
TUTYP3 = 'spec:Data.FluxAxis.Accuracy.StatError'

TUCD4 = 'phot.flux.density;em.wl'
TUTYP4 = 'eso:Data.FluxAxis.Value'

TUCD5 = 'stat.error;phot.flux.density'
TUTYP5 = 'eso:Data.FluxAxis.Accuracy.StatError'

Q: Could you please provide some information regarding the format of APEX 1-D spectra?

A: The format defined in the Phase 3 SDPS for 1-D spectral products applies also in the case of APEX reduced spectra, with few modifications:

1. The following keywords defined as mandatory for spectral products do not apply in the case of APEX spectra:
OBID1, DISPELEM.

2. While the keyword FEBE1 indicating the Frontend-backend combination becomes mandatory in this case.

Example of the values of some specific keywords:

Primary header
ORIGIN  = 'APEX'             / Facility
TELSCOP = 'APEX-12m'         / Telescope name
INSTRUME= 'APEXHET'          / Instrument name
FEBE1   = 'HET230-XFFTS2'    / Frontend-backend combination
OBSTECH = 'SPECTRUM'         / Technique of observation
PRODCATG= 'SCIENCE.SPECTRUM' / Data product category

Extension header
EXTNAME = 'SPECTRUM'           / FITS Extension name
TTYPE1  = 'FREQ    '           / Label for field 1
TUTYP1  = 'Spectrum.Data.SpectralAxis.Value'
TUNIT1  = 'GHz     '           / Physical unit of field 1
TUCD1   = 'em.freq '           / UCD of field 1
TDMIN1  =              229.011 / Start in spectral coord.
TDMAX1  =             233.000  / Stop in spectral coord.
TTYPE2  = 'FLUX    '           / Label for field 2
TUTYP2  = 'Spectrum.Data.FluxAxis.Value'
TUNIT2  = 'mJy     '           / Physical unit of field 2
TUCD2   = 'phot.flux.density;em.freq' / UCD of field 2
TTYPE3  = 'ERR     '           / Label for field 3
TUTYP3  = 'Spectrum.Data.FluxAxis.Accuracy.StatError'
TUNIT3  = 'mJy     '           / Physical unit of field 3
TUCD3   = 'stat.error;phot.flux.density;em.freq' / UCD of field 3

Please also note that the values of the following keywords need to be provided in nm:
WAVELMIN=        1291867.18750 / [nm] Minimum wavelength
It corresponds to the value of TDMAX1 converted from GHz to nm (conversion factor: 299792458)
WAVELMAX=        1309075.80566 / [nm] Maximum wavelength
It corresponds to the value of TDMIN1 converted from GHz to nm (conversion factor: 299792458)
SPEC_BIN=        111.744273793 / [nm] Wavelength bin size
SPEC_VAL=        1300471.49658 / [nm] Mean Wavelength
SPEC_BW =        17208.6181641 / [nm] Bandpass Width Wmax – Wmin

The following link may be useful to reconstruct the provenance information, it provides the mapping between the original file name to the archive id, and calculating the exposure time:
http://archive.eso.org/wdb/wdb/eso/apex_origfile/form

With respect to an earlier version this specification has been revised on 18/07/17 to the effect that TTYPE1 = 'WAVE' is independent of air or vacuum wavelength.

Q: How is calibration with respect to air wavelength indicated in the one-dimensional spectrum data format?

A: To discriminate wavelength measured in dry air (at standard temperature and pressure) from the wavelength measured in vacuum, the ESO/SDP one-dimensional spectrum data format requires TUCD1 to be set to 'em.wl;obs.atmos' and 'em.wl', respectively (see below).

For improved human readability when browsing the FITS header, it is recommended to add a respective comment to the line defining the TUCD1 keyword.

When the spectral axis is given in air wavelength:

TTYPE1   = 'WAVE'
TUCD1    = 'em.wl;obs.atmos' / Air wavelength

When the spectral axis is given in vacuum wavelength:

TTYPE1   = 'WAVE'
TUCD1    = 'em.wl'  / Vacuum wavelength

Note: WAVELMIN and WAVELMAX need to be specified in either air or vacuum wavelength consistent with the definition of the spectral axis.

Q: Definition of PRODCATG in case of multiple associated images.

A: Each data set must be associated by one unique white light image. Further images, like line images, may be associated in addition. To allow unique identification of the white light image, the following scheme of labeling must be adopted:

• if there are multiple images associated to the data cube, the white light image (and only this one) must be labeled PRODCATG=ANCILLARY.IMAGE.WHITELIGHT. Unless otherwise specified, the other images need to be labeled by the default PRODCATG=ANCILLARY.IMAGE.
• if there is only the white light image associated it is recommended to label it PRODCATG=ANCILLARY.IMAGE.WHITELIGHT, but PRODCATG=ANCILLARY.IMAGE is also accepted.

Q: How are exposure maps defined in the SDP standard?

A: In the context of co-adding multiple exposures, the exposure map records the variation of the accumulated exposure time (in seconds) of 2D image mosaic and 3D IFU data cubes per pixel and per spaxel, respectively. It encodes the pattern of multiple offset exposures being combined into the final product while taking any possible masking of bad or invalid pixels into account.

The exposure map is indicated by the FITS keyword in the primary header:

   PRODCATG = 'ANCILLARY.EXPMAP'

Dimensions: The exposure map is a 2D FITS image associated to the main science data as an ancillary file. It shares the dimensions (AXIS1, AXIS2), WCS definitions and number of image extensions (if any) with the 2D image mosaics, or, in case of 3D IFU data cubes, the exposure map inherits the dimensions and WCS definition of the first two axis of the data cube (AXIS1, AXIS2, etc).

Units and data encoding: The exposure map encodes the total exposure time per pixel/spaxel in units of seconds (BUNIT = 's' / exposure time in seconds) using floating point decimals (BITPIX = -32) or 16 bit scaled decimals (BITPIX = 16, see FITS standard).

The total exposure time per pixel is the sum of the exposure times of the exposures contributing to the given pixel/spaxel possibly including fractional contributions depending on the interpolation scheme.

The exposure map is constraint to values >0 for data pixels/spaxels representing a valid signal, or value=0 for void pixels/spaxels. Note: It is recommended to set the values of void pixel/spaxel in the science data to NaN.

BLANK, NaN or any other special value are not allowed in the exposure map.

Relation to the EXPTIME keyword: EXPTIME, to be recorded in the header of the main science file, is defined to be the median of the pixels values of the exposure map excluding any value=0 pixel.

Data compilation: If the effective exposure time varies within the science image or data cube, it is strongly recommended to provide the exposure map along with the main science file.

Note: The exposure map does not serve as substitute for the WEIGHTMAP.

Q: How is the variation of the number of contributing exposures defined in the SDP standard?

A: In the context of co-adding multiple exposures, the pixel count map records the variation of the number of contributing exposures of 2D image mosaic and 3D IFU data cubes per pixel and per spaxel, respectively.

If the contributing exposures have different exposure times, the pixel count map does not apply, and the exposure map should be associated instead.

The pixel count map is indicated by the FITS keyword in the primary header:

   PRODCATG = 'ANCILLARY.PIXELCOUNTMAP'

Dimensions: The pixel count map is a 2D FITS image associated to the main science data as an ancillary file. It shares the dimensions (AXIS1, AXIS2), WCS definitions and number of image extensions (if any) with the 2D image mosaics, or, in case of 3D IFU data cubes, the pixel count map inherits the dimensions and WCS definition of the first two axis of the data cube (AXIS1, AXIS2, etc).

Units and data encoding: The pixel count map encodes the total number of contributing exposures per pixel/spaxel as integer number (BUNIT = ' ' / dimensionless) using 16 bits (BITPIX = 16, see FITS standard).

The pixel count map is constraint to integer values >0 for data pixels/spaxels representing a valid signal, or value=0 for void pixels/spaxels. Note: It is recommended to set the values of void pixels/spaxels in the science data to NaN.

BLANK or any other special value are not allowed in the pixel count map.

Data compilation: If the effective exposure time varies within the science image or data cube the pixel count map may be optionally provided along with the main science file in addition to the exposure map.

Q: Are OBJECT keywords in the FITS extension of SCIENCE.CUBE.IFS required? Is the specific suffix indicating the type of extensions (i.e. data/stats/err) required?

A: No, the requirement for the OBJECT keyword in the extension of data of type SCIENCE.CUBE.IFS has been dropped. It is optional now.

If such keyword does exist in the extensions, and space permits, it is recommended to adopt the OBJECT keyword  value from the primary header and to append an indicative suffix for the type of extension.

Typo: Page 47 of the SDP standard, v.5

Fix: The PRODCATG for the OmegaCAM pawprint should be set to SCIENCE.MEFIMAGE.

Typo: Page 33 of the SDP standard, v.5

Fix: The 'ancillary.weightmap' value should be uppercased on the fourth line of the page 33.

Typo: Section 4.1 of the SDP standard, v.5

Fix: In case the spectral data have been calibrated to absolute density, the flux scale should be given in physical units, e.g. in erg/s/cm2/A. 'A' should be 'Angstrom' instead.

Typo: Section 2.4.3 of the SDP standard, v.5

Fix: It should read 'exceeding 9999 records'.

Typo: Page 17 of the SDP standard, v.5

Fix: The format string should be TFORM1 = '35A'.