qfits Reference manual

6.2.0


Introduction

qfits is a stand-alone library written in C to interact with files complying with the FITS format. It is fast and portable over any kind of POSIX-compliant platform.

Rather than going through the FITS format specification and trying to implement some support for everything that is described there, this library was built day after day upon request. This guarantees that all the functions you will find in the library have been written for some purpose in the VLT pipeline context, and all of them are used daily in a production environment.

Functionalities offered by this library are:


History

qfits was born from the need to have a simple but powerful C library to handle FITS file access. There are already many libraries on the Net to handle FITS files, but they tend to be bloated with far too many features and may raise portability and maintenance issues.

qfits was written to take care of all low-level aspects of the FITS format, and only these. You will find there a wealth of optimized functions to access everything inside FITS files, but nothing about what you could do with the data: this is left to other (e.g. image processing) libraries. There is no suggested image or cube type, and the table object only loads the data without trying to interpret them.

The idea is that people wanting to work with FITS might have different requirements: somebody writing an image viewer might just want to load a pixel buffer, somebody handling headers only might want to have easy access to header information without touching pixel data. qfits allows you to get your hands on what is contained in a FITS file but does not force you to use any specific high-level data type to describe your data.


Header handling

This section gives you an overview of the functionalities offered to work on FITS headers with qfits. For a complete reference, check out the reference manual: qfits.h.

FITS headers are simply formatted as 80-char lines (cards) containing ancillary data represented in ASCII format like:

keyword = value / comment

If you want to retrieve data from a FITS file header, you will find various useful query routines to get a keyword value from main or extension headers. Values are returned as strings and not coerced to any particular (e.g. numerical) type. See e.g.

Since all FITS values are returned as strings, you may need to identify types. See e.g.

Of course, you can use the usual atof(), atoi() and scanf() functions to convert a string to a numerical type.

You may also want to perform more complex operations on FITS headers, like loading one, modifying it and saving it back to a new file. The qfits_header data structure is offered for that purpose. It comes with utilities like:

And of course all manipulation tools you can think of:

There are other functions to create new FITS cards, dump a header to screen for debugging purposes, etc. See the reference manual for an exhaustive description.

An important feature of the qfits_header object when loaded from a file, is that it keeps the memory of the initial FITS card. The data structure that is carried around for each FITS card contains:

This feature is fairly useful if you want to perform header manipulation without getting too intrusive. You can use this to load a header, modify a couple of cards and dump all back to disk. The dumped header will only be modified for the cards you have touched, all others will be forwarded verbatim as they appeared in the initial file. This least-modification policy ensures that you will not modify the lines you do not need to touch.

Notice that ESO's HIERARCH keywords and DICB ordering are natively supported by qfits, i.e. when you work with qfits_header objects you can be sure that they will properly recognize and sort keywords following ESO conventions.


Data handling

This section gives you an overview of the functionalities offered to work on FITS data segments with qfits. For a complete reference, check out the reference manual.

Data segments are quite simply stored: they always contain data contiguously in uncompressed format, so reading them is mostly a matter of applying an fread() statement at the right place in the file, with the right size. To find out about offsets to data and header parts in a FITS file (possibly containing several extensions), you can make use of:

qfits includes a caching mechanism to speed up accesses to large files to an optimal access time (only one parsing of the file is needed, even with multiple queries on the same file). This mechanism is internal to this module and invisible to the programmer using qfits. This means that you can loop on all sections on a huge file to retrieve file offsets for each extension, and pay the price of file parsing only at the first call.

If you want to dive further into data loading, you may want to have a look at table and image handling functionalities.

Table handling

A table is stored in a FITS file in an extension. The extension header contains all needed informations to read the data (number of columns, the column types, the columns labels, the columns unit, etc...).

A FITS table is composed by columns. Within one column there can only be data of one defined type, but you can store several values by field.

In BINARY tables, you can find 11 different data types, in ASCII tables, there are 5 different types. All these types are supported by qfits:

The qfits_table object provided by qfits contains all necessary informations to define how the data are stored in the file: the name of the file it comes from, the table type (QFITS_BINTABLE or QFITS_ASCIITABLE), the number of columns and for each column a pointer to a qfits_col object.

Each qfits_col object contains informations defining the associated column: the type (e.g. TFITS_BIN_TYPE_L), the number of rows, the number of atoms per field, the label, the unit, etc...).

This qfits_table object is returned by qfits_table_open() and should only be destroyed with qfits_table_close().

This qfits_table object does not contain the table data, but has to be used to load the data with functions like:

qfits_query_column() returns you simply the data (byte swapped if necessary) as they were read in the file as a byte array. It is up to the user to interpret the data (e.g. to try to find out where the NULL values are).

qfits_query_column_data() does the same, but identifies the NULL values and replaces them by a user-provided value. In this case, the returned array is an array of data of the right type. It is returned as a void*, the user just has to cast it to a double*, a short* or what is needed.

For example, if you require in a table that contains 15 rows a column where you can find 3 values of type TFITS_BIN_TYPE_M (double complex), qfits_query_column_data() will return an array of 15 * 3 * 2 double values.

qfits also provides functions to write a table to a FITS file (qfits_save_table_hdrdump()), or to print out a table value or a complete table on the screen or in a file (qfits_table_field_to_string()).

Image handling

The qfitsloader and qfitsdumper objects are offered to simplify the action of reading/writing image data. The corresponding operators take care of retrieving all necessary ancillary information (image size, pixel type, buffer position), perform a memory allocation or mapping and read the file into memory. See the following functions:

As for tables, the idea is that you first launch an analysis of the file to get back a number of informations about what is present there. In the case of images, you use qfitsloader_init() to see if pixel loading could be done, then use qfits_loadpix() to perform the actual load.

Loading pixels as floats means that whichever pixel format (BITPIX) is used in your input FITS file, all pixels will be converted to your local representation of float upon loading. The 3 offered types basically allow you to work with 3 kinds of pixels internally, choose the one that best suits your needs. 'int' is not recommended for loss of precision, and 'double' for memory and performance issues.

Notice that as soon as you start using pixel loading functions, you must comply with the qfits memory model.


Features

This section describes various unique features of qfits.

Portability

This library has been ported and tested on the following platforms:

Speed

The offered routines make use of caching mechanisms and memory-mapping system calls to enhance FITS file parsing and speed up the code.

Numerical precision

Since FITS headers are stored as strings, numerical precision is limited by the number of digits used to write a number in a FITS card, which is in theory larger than what a 32-bit floating-point can store. Using qfits, these values are available to a C programmer as a string, making sure that precision has at least not be lost in the reading process.

Conservative headers

As mentioned above, if you only need to load a header, modify a card and save it back, you will find out that only the card you touched has been modified, the rest was verbatim transferred. This is useful to ensure that the library formatting does not disturb your format.

Native HIERARCH support

Native support for ESO's HIERARCH keywords, as well as keywords of any length (up to 80 chars per line).

ESO/DICB keyword ordering

Native support for DICB (ESO only) ordering of keywords in the headers. FITS files created with this library will be DICB compliant with respect to keyword ordering.

Memory model

qfits does not only offer pixel loading mechanisms, it also comes bundled with a memory module based on a model optimized for the handling of very large data segments. Calls to memory allocators will yield valid data pointers past the hardware limits of your machine, up to 2 or 4 Gb on a 32-bit workstation and insanely high values on 64-bit processors.


Installation instructions

In the main qfits directory, type:

    
./configure --prefix=install_dir ; make ; make install

To use the library in your programs, add the following line on top of your module:

#include "qfits.h"

And link your program with the qfits library by adding -lqfits to the compile line.


Frequently Asked Questions

Where should I start to use qfits?

Try to build the library on your system first. You should then have a new library file called libqfits.a. To use the library functionalities in your programs, you must add the following include in your list of includes:

#include "qfits.h"
      

And then compile your program adding the correct flags to indicate where the qfits.h header file is, where the libqfits.a file is, and that you want to link against libqfits. Example: if the header file is in /usr/local/include and the library in /usr/local/lib, you would use:

% cc -o myprog myprog.c -I/usr/local/include -L/usr/local/lib -lqfits
     

Now you are all set. Refer to this documentation for more information about the offered data structures and associated methods.

What should I know about the cache mechanism?

Parsing FITS files to find extensions is a lengthy process, because the FITS format does not declare in the main header where the following extensions are located in the file. Any access to a FITS file with this library will cache a number of offset pointers in the file to allow fast access to extensions. This means that the file is parsed completely only once, the first time it is accessed through any of the qfits functions.

The cache is implemented (since version 4.2) as a rotating buffer of modest size: 128 FITS file information structures can be held simultaneously. If you do need to perform accesses on a list of more than 128 files and want to use the caching mechanism at its best, it is recommended to isolate all your FITS requests to a given file in the same code area.

As a programmer, you do not need to initialize or shutdown the cache, this is done automatically.

The rotating buffer is a global data structure private to the cache module. The immediate side-effect is that the library is thread-unsafe. If you are concerned about writing multi-threaded applications, you should use a mutex upon accessing this library. Making a thread-safe compliant version of this library should not be too hard, but the need has not been felt yet.