Everything you always wanted to know about MIDAS

and that it would take you weeks to find in the manual

O.Hainaut, 2001-10-22
(ohainaut@eso.org)
URL: http://sc6.sc.eso.org/~ohainaut/midas.html

This is a starter's guide for Midas. Section 1 is for basic interactive Midas, Section 2 for programming in Midas.

0- Generalities

The official MIDAS page is at http://www.eso.org/projects/esomidas; you will find there a link to the full documentation. You may also find some useful hints and tips in the FAQ.

OK, now you can get started:
inmidas 00 : start midas, session number 00 (should be between 00 and 99)
inmidas -p 01 : start another midas without killing the 1st one ("parallel")
inmidas xx : start Midas without graphical capabilities.

helpmidas : help interface (sometime does not work in LinuX)

1- Basic Midas:

1.1- Unix commands in Midas:

OK, this is not strictly speaking Midas, but really useful:

Use $ to pass the command to UniX

$ ls
$ more toto.dat

Warning: $ cd will not work as expected ($ starts a shell, in which the directory is changed, then you exit from the shell and end up in the original directory!). Use

CHANGE/DIR ../toto
(warning for those using my login.prg: cd has been aliased to chan/dir, which is why it works as one though it should, but as it should not if it had not been redifined).

1.2- Command conventions

commands are not case sensitive:

LOAD/IMAGE = load/image

commands and parameters can be abreviated as much as possible:

LOAD/IMAGE toto.bdf scale=full cuts=force,ihap

loa toto sc=fu cu=f,i

all commands have a mini-help:

LOAD/IMAGE ??

all commands have a long help:

HELP LOAD/IMA

HELP

HELP LOAD

HELP/QUALI IMAGE

In the generic HELP, you will see that there are several topics between [brakets]; these are general help about these topics. I recomment specially HELP [DESCRIPTORS]

In a command, parameters can be replaced by a default value using ?

STAT/IMA ? cursor

1.3- Read images in

Images usually come in FITS (toto.fits or toto.mt); midas _can_ deal with FITS frames, but it is usually a good idea to translate the FITS into midas' internal format, BDF:

INDISK toto.fits toto.bdf

(note that FITS is universally standard, but BDF changes from machine to machine, from midas version to midas version: NEVER archive BDF files).

As we are playing with images:

OUTDISK toto.bdf toto.fits

will convert an image from BDF to FITS.

Note that these commands also work on tables stored in FITS:

INDISK toto.fits toto.tbl
OUTDISK toto.tbl toto.fits

1.4- play with images

Once you have an image you can look at it:

CREATE/DISPLAY creates a display window to look at images

LOAD/IMAGE toto.bdf

or with additional parameters:
LOAD/IMAGE toto.bdf cuts=100,1000 scale=-3 center=500,500
cuts: min and max of the color scale
scale: positive to zoom in, negative to zoom out
center: the pixel of the image set at the center of the display.
These come with more fancy options:
LOAD/IMA toto cu=f,i sc=fu ce=ce
cu=f,i: forces the minimum just below the mean of the image, the max
well above the mean. Works well on images
sc=fu: adjust the scale to see the whole image
ce=ce: center the center of the image

Note that once the parameters are set, they are remembered for the considered image; next time, it is enough to do LOAD/IMA toto to reload it as the last time.

GET/CURSOR

Note that the XY coordinates are given in 2 coordinate systems:

"pixel coordinates": first pixel is 1,1, and they increase by 1 per pixel

"world coordinates": a more complex coordinate system. Two cases are possible:

simple image: x_world = (x_pixel -1) * x_STEP + x_START (and same for y), where STEP adn START are descriptors of the image. For example, on SuSI and EMMI (and many La Silla instruments), START is the first pixel of the window of the CCD, and STEP is the binning of the CCD. In that case, x,y_pixel are the number of the pixel of the image you work on, and x,y_world correspond to the "real" pixel of the physical CCD, no matter which window and bin are set.

As a consequence, if bin=1 and the whole CCD is read out, world=pixel

World Coordinate System (WCS), e.g. for SofI and many instruments on the VLT: a more complex rule is set to convert the x,y_pixel into x,y_world so that x,y_world are the actual RA, Dec of the pixel.

As a convention in Midas, "world" x,y are typed as simple numbers, e.g. 123,456, while "pixel" x,y are typed with the @ symbol, e.g. @123,@456. So, in the LOAD/IMA example above, we centered the image on world coordinates 500,500, which may or may not be the same as @500,@500.
While the pixel coordinates of a pixel can change if you do something to an image (e.g. crop it), its world coordinates will always be preserved whatever you do (unless of course you mess around with the START and STEP).

STATISTICS/IMAGE

STAT/IMA ? cursor
you can specify that you want the statistics of only a sub-region. Measure with Mouse-left, exit with Mouse-middle

Beware of combining both:
STAT/IMA toto cursor
can give rather stupid results as you define the region on the image that is displayed, and you measure on toto.bdf, which might be another image!\

CENTER/GAUSS

READ/DESCRIPTOR toto

READ/DES toto *
will read _all_ the descriptors (there might be many...)
READ/DES toto ESO.INS.FILT1.NAME
will read one of the descriptor
Some useful ones:
O_TIME contains the date of the image in different formats and the exposure time
O_POS contains the RA and Dec (in degrees, although some telescope store this in hours and deg, which is wrong...)
START the world coordinates of the first pixel (usually the first pixel of a subwindow defined to read the CCD).
STEP the size a pixel in world coordinates (usually the bin of the CCD)

1.5 Arithmetics with images

COMP/IMA toto.bdf = ( image - bias ) / flat / 900.

will do exactly what it seems it will do: make the difference between image and bias, divide the result by flat, and finally divide the result by number 900.

Some notes:

there must be a space before and after the =
you can mix images and numbers
images can have different sizes (the result will have the size of the intersection of all the images), but they must have the same STEP.
many functions are defined, look into the help.

1.6 Catalogues

A catalogue is an ASCII file that contains a list of stuff. There are 3 kinds:

image catalogues: ICAT, containing a list of bdf files (or fits, or bdf.gz compressed images)
table catalogues: TCAT, containing tables
ASCII catalogues: ACAT, containing ascii files (typically other catalogues, or whatever...)

These are really usefull, as many commands can be executed on a whole catalogue. In the example below, I use only ICATs, but it works the same for the others:

CREATE/ICAT oli toto*bdf

READ/ICAT oli

Notes:
catalogues are just ASCII files. You can edit them with Emacs (or whatever editor). It is often simpler to do
CREA/ICA oli *bdf then to remove the lines you don't need than to make a very accurate CREA/ICA oli ONT*21T23*bdf,ONT*22T00*bdf...

EXECUTE/CATALOGUE LOAD/IMA oli.cat sc=fu

Note that you can NOT create new files with different names: this will NOT work as you could think it would.
EXEC/CAT comp oli_new.cat = oli.cat / 1000.
To do that, you should use a small procedure, c.f. STOR/FRAM in Section 2.

1.7 Tables

Tables are really very powerful. The can be created

From a data file:

From another Midas command, eg

Some useful commands:

READ/TABLE toto      reads the whole table

READ/TABLE toto @1 #1 #2 #3 reads only line 1, columns Nr 1, 2 and 3
READ/TABLE toto :xcent :ycent      reads only columns named :xcent and :ycent, all the lines.

COMP/TAB toto :out = :x + :y     arithmetics; there are many pre-defined functions, cf HELP COMP/TAB

SELECT/TABLE toto :xcent .ge. 0.    will select the lines for which the condition is true

SELECT/TABLE toto all     re-select the whole table. Note that a COMP/TAB reselects the whole table too

SORT/TAB toto :x :y    sort the table along :x, then :y. Be careful when doing a sort on a SELECTed table, then reading out the result. May be better to re-do the SELECT

1.999 The end

At this point, you should be able to survive in front a Midas terminal, and even to convince Midas to do what you want to do on an image. Next section is for doing the same thing on many images (cf "Error is human, you need a computer program to really screw things up").

Oh, yes. In case you did not find out, the command to exit is

QUIT

BYE

EXIT

2. Basic Midas procedures

2.1 Intro

It can be *very* useful to make small Midas procedures, even for beginners. Basically, just put a few commands in a file, and it is already a procedure. Be bold, try it. The only think you risk is to wipe the data from your disk, but you would need to do something really stupid for that to happen.

Midas procedures are expected to have the .prg extension, and they are called using @@:

@@ example 1234 toto.bdf out.fits

will call example.prg, passing 1234, toto.bdf and out.fits as parameters.

The procedures are looked for
- first in the local directory
- second in the ~/midwork directory

Note that comments are marked with ! :

LOAD/IMA toto.bdf ! everything after a ! is a comment

2.2 Variables

2.2.1 Global and local

All variables _must_ be declared.
All variables are considered as "vectors", with n elements, the first one not necessarilly being number 1 (it can be any arbitrary number).
Variables can be Integer, Real, Double precision or Characters.

The definition will look like

variable/type/first_element/number_of_elements

Local variables are known only inside the procedure, while global variables (called keywords) are known everywhere.
Use global variables only when needed, because
- the total space for global variable is limited (big, but limited)
- once defined, you have to keep the definition compatible (i.e. if you define i as global integer, you cannot redefine it later as real, unless you DELETE/KEY i first)

Local variables:

DEFINE/LOCAL olinteg/i/1/5 0,1,2,3,4

Global keywords

WRIT/KEY olireal/r/1/1 0.

2.2.2 Parameters

The parameters are special case of variable:

they are called P1, P2, ... P8
their values are set from the parameter of the procedure
they are defined/initialized with

DEFI/PAR Px default type "query string"

Examples:

DEFI/PAR P1 oli.cat C "input catalogue"
DEFI/PAR P2 ? N "size" ! note that no default is given
DEFI/PAR P3 ? ? "something" ! type is free

2.2.3 Variable substitution

(tricky and important. In case you were wondering about a coffee, it is a good time to have it NOW)

In a procedure, you can use a variable either DIRECTLY, as in

COMP/KEY result = toto1 + toto2

in that case, the variable is used with its full precision, or use its SUBSTITUTED VALUE, putting it between {}, as in

COMP/KEY result = {toto1} + {toto2}

In that later case, the {toto1} is first substituted using the current format, and THAT value is used.

As a general rule:

if you compute something, as in COMP/KEY, use the variables directly:

COMP/KEY result = toto1 + toto2

if you use a variable in a non-computing command, use its substituted value:

LOAD/IMA toto.bdf cuts={toto1},{toto2}

Example: toto.prg

DEFI/LOC toto1/r/1/1 1.001
DEFI/LOC toto2/r/1/1 2.002
DEFI/LOC result/r/1/1 0
result = toto1 + toto2 -3 ! note that COMP/KEY is implicit
READ/KEY result
SET/FORM f5.2
result = {toto1} + {toto2} -3
READ/KEY result

Midas 011: @@ toto

keyword: RESULT type: real no_elems: 1 modtime: 1003791723
0.00300014
keyword: RESULT type: real no_elems: 1 modtime: 1003791723
0

The first COMP/KEY was executed in full precision, the second in the precision of the format!

(note that {toto} used to be written 'toto' - you might still find it in old programs; it is obsolete, as you cannot nest them, as in {toto{i}} )

The format is defined as in FORTRAN, with the following syntax:

SET/FORM Format_for_integer format_for_real,format_for_double_precision

(note the , ), e.g.

SET/FORM I5 F5.2,E15.5

2.2.4 Special functions:

Many functions exist in Midas. Their name is always M$SOMETHING, where SOMETHING is what you would expect, eg

M$ABS(arg) M$LN, M$LOG10, M$EXP, M$SIN, M$COS, M$TAN(arg), M$SQRT, M$ASIN, M$ACOS and M$ATAN(arg)

Some special functions are also defined, e.g. M$EXIST(arg) returns 1/0, if file contained in character key `arg' exists or not. There are many M$EXIST* functions, most of then useful. Refer to HELP COMP/KEY for complete description

NOTE that many of the math functions (e.g. M$LOG) also have a counter-part for images,

COMP/IMA res = log(ima)

or for tables,

COMP/TAB table :res = log(:input)

In those cases, there is no M$, which is only for keywords (what do you mean, "confusing"? M$ is only for keywords, not for tables nor images. Simple).

2.3 Structures

2.3.1 IF

General structure:

IF something .condition. something THEN
write/out "it is true"
ELSE
write/out "it is false"
ENDIF

.condition. is like in FORTRAN: .le. .lt. .eq. .ge. .gt. .ne.

ELSE can be ommitted.

Example:

if clei .eq. 4 then
   write/out "Loading result..."
   load/ima 'p1'
   clea/ch ov
   clei = 0
else
   write/out "not loading yet"
endif

Same remark than above about substituted and direct use of the variables; the condition above could be written

IF {clei} .EQ. 4 THEN

or even

IF "{clei}" .EQ. "0004" THEN

assuming format set to I4).

2.3.2 DO

General structure:

DO variable = start end step
write/out {variable}
ENDDO

Note that
- the start end step are separated by spaces,
- step can be ommitted, and is set to 1 by default.
- step=0 is OK and corresponds to an infinite loop
- the loop is always execute at least once!

Example:

DO LL = 22 OUTPUTI(4) -2
WRITE/OUT {LL}
ENDDO

(do I still need to make a remark about substituted variables?)

2.3.3 GOTO

Goto is quite useful (I don't want to hear comments about GOTOs being unelegant, outdated, etc... the ARE useful). The command is

GOTO LABEL

Where LABEL is a label located somewhere else, marked with :, as in:

again:
! ( note the : after again: this is a label )
... ! some other stuff
if i .ge. 0 then
goto again:
endif

which simulate a DO-WHILE loop.

2.3.4 Catalogue loops

If you want to loop on all the frames contained in a catalogue,
STORE/FRAME is what you need. From the help:

STORE/FRAME key frame [indx] [exit_label]
key = name of keyword where frame or catalog entry will be stored in
frame = name of frame (syntax = nnnnn) (a)
or
name of catalog (syntax = nnnnn.cat) (b)
indx = index into local keyword CATAL, only used with option (b);
defaulted to 1, i.e. use CATAL(1)
exit_label = label to jump to on termination of catalog processing,
only used with option (b)

Example and notes:

DEFINE/LOCAL CATAL/I/1/1 0 ! you NEED it, and it MUST be called CATAL
CAT_LOOP: ! label for the beginning of the loop
STORE/FRAME IN_A astro.cat 1 FINITO
.... commands using frames stored in key IN_A, eg
COMP/IMA new_{IN_A} = ({IN_A} - bias)/ff
LOAD/IMA new_{IN_A} cu=f,i sc=fu
GOTO CAT_LOOP ! loop the loop
FINITO: ! jump here when the catalogue is finished
.... any other commands ....

Notes:
- if STOR/FRA receives a frame (toto.bdf) instead of a catalogue (with .cat), the loop will be executed once for that frame: great to make procedures that work both on catalogues and single frames.
- this is much more powerful that the EXEC/CAT: you can create new frames and play with their names, as in the example.

2.4 Deal with tables and images

2.4.1 Simple stuff

COMP/IMA mag_{inima} = -2.5* log10( ({inima} - {bias})/{ff} ) + {ZP}

will be interpretted as

COMP/IMA mag_ntt0012 = -2.5*log10((ntt0012 -bias_night1)/ff_night1)+26.42

(note the combination of images and numbers).

2.4.2 Play with descriptors

To read a descriptor into a variable, you can use COPY/DK (cf the help), but it is faster and nicer to address directly the header. Let's assume you want to read in the exposure time of a frame, which is stored in O_TIME(7)

DEFI/LOC exp/r/1/1 0.
exp = {ima.bdf,O_TIME(7)}

et voila. If the name of the image is in a variable (for instance the parameter P1), just nest the {}:

exp = {{p1},O_TIME(7)}

Note that this is a case of variable substitution. Beware of the format.

In the case of tables, it is the same trick, but you have to specify the .tbl extension.

table_length = {{p1}.tbl,TBLCONTR(10)} ! remark: in this example, P1 is expected to come without the .tbl, so we add it.

Note that TBLCONTR(10) is the length of the table

2.4.3 Play with tables

If you want to address just an element of a table:

x = {table.tbl,:x,@1}

of more fancy:

x = {{myTable},:{myColumn},@{i}}

If you want to write in one element:

{myTable},:mjd,@{i} = JD - 250000.5

(note that the left side is NOT surrounded by {}! Go and re-read the section on variable substitution if it is not clear!)

2.4.999 Various tricks

Many commands output their values in the generic global keywords OUTPUTR, OUTPUTI, OUTPUTC. READ/KEY these keywords just after executing the command to see who is who. For instance

stat/ima {p1}
min = outputr(1)
max = outputr(2)
mean = outputr(3)
sigma = outputr(4)
median = outputr(7)

In addition to the previous output in OUTPUT*, many commands can output their results in table, cf the help of CENT/GAU, MAGN/CIR, etc...

In the case of tables: any table has two special columns

SEQ which contains the number of the line (in the current way of sorting it!)
SEL which is set to 1 if the line is selected, 0 if not. So, if you want to make an additional selection to something already selected:
SELEC/TAB toto sel.eq.1 .and. :x.le.0.

2.999 Final notes

More hints can be found in

HELP [BATCHES]

Also, remember that the fastest way to program is often to modify a program that does something vaguely similar to what you want. Look in ~/midwork for inspiration.

--oOo--

[ O.Hainaut | comments | ESO | La Silla | NTT ]