| |
|
|
|
4 THE PRE-PROCESSOR
4.1 SOFTWARE ARCHITECTURE
The incoming data are read via DMA into a ringbuffer. The capture process provides the synchronization between the DMA-driver and the calculation process. A ringbuffer overflow is also detected by the capture process. When processing the data, the result frame will be stored in a ringbuffer to guarantee continuous data flow. When the frame has been processed and should be transferred, it is given to the data server. From here the frames can be distributed in parallel to several hosts via parallel output link threads. The frames can also be displayed on one or more local real-time displays (RTD's). Data sets like flatfields or bad-pixel masks are stored in shared memory and can be shared between acquisition processes.
4.2 DMA INTERFACE
The DMA interfaces for SBus (EDT-SCD 20/40/60) and PCI-Bus (EDT-PCICD 20/40/60) are pin compatible. As the drivers are completely different, a driver interface library is included in the sdma submodule. Depending on which type of Ultra-Sparc is used a different driver library is linked with the acquisition process. This is fully transparent to the acquisition software. Both DMA-interfaces support a maximum input data rate of 30 MHz x 16 Bit (SCD/PCICD 60).
4.3 ACQUISITION PROCESSES
The acquisition processes can be started as individual tasks on any computer running a Sun-Solaris or HP-UX (simulation only) operating system.
Command line for process execution:
Most readout modes additionally support the `-nx <pixels>', `-ny <pixels>' option to specify the input frame format. The `-ndet' option specifies the number of detector instances. This is needed by the output server for the correct handling of window requests, if the frame buffer contains formatted images of more than one detector. The application can access this parameter via the global variable sdmaNumDet. If the read-out mode supports more than one detector instance, it should allocate buffers for [NX * NY * sdmaNumDet] sized frames. The frame header elements should be set to:
If the `-i' option is not specified, the acquisition process starts the data capture and processing itself, using the default parameter setting (auto-start). This mode is reserved for test purposes. For using a command interface the acquisition process has to be invoked with the `-i' option.
4.4 HOW TO DEVELOP AN ACQUISITION PROCESS
Now we will develop a new acquisition process from scratch. The final version `sdmaTemplate.c' is part of the IRACE module and can be found together with the standard acquisition processes in the test directory of the sdma sub-module. The standard vltMakefile should be used to build the process. The following flags have to be set:
USER_INC = -I/opt/EDTscd -I$(DFLOW_ROOT)/include/rtd -I$VLTROOT)/include/rtd
xxxx_LIBS = sdma sdmaDrv sdmaThr icmdSock icmd socket nsl intl m rtdImgEvt
USER_INC = -I/opt/EDTpcd -I$(DFLOW_ROOT)/include/rtd -I$VLTROOT)/include/rtd
xxxx_LIBS = sdma sdmaDrv edt sdmaThr icmdSock icmd socket nsl intl m rtdImgEvt
USER_CFLAGS = -DHP -DTHR_POSIX
USER_INC = -I$(DFLOW_ROOT)/include/rtd -I$VLTROOT)/include/rtd
xxxx_LIBS = sdma sdmaThr icmdSock icmd m rtdImgEvt
4.4.1 BASIC SYSTEM INITIALIZATION
In a first approach, a main process is generated and the basic system initialization is done. The sdmaStartup() function initializes the data acquisition according to the command line arguments. It also sets up the data output server and the command interpreter. The ringbuffer size (number of bytes per element) is passed as parameter to the function. It is up to the application to compute the needed size. Afterwards the dynamic parameter structure (myPARAM_TYPE) has to be passed to the system. This is done via the sdmaSetupDynParam() function. The structure myPARAM_TYPE and the associated parameter names have to be defined by the application:
static char *paramId[] = {"DET.NC.DIT",
Then some default parameter-values have to be set. The sdmaParamDefault() routine informs the system, that the default parameter setup has been done. Now the command interpreter is able to receive a double buffered mirror of all parameters defined in myPARAM_TYPE. The first main process now looks like:
static myPARAM_TYPE *param; /* pointer to parameter structure */
int main (int argc, char *argv[])
int nx=256; /* default chip size */
int ny=256; /* default chip size */
if (sdmaStartup (argc, argv, (nx*ny)*2, erms) == -1)
printf ("\nsdma: %s \n\n", erms);
if ((param = (myPARAM_TYPE *)sdmaSetupDynParam (sdmaNUM_ELE(paramId),
(char **)paramId, 0, erms)) == NULL)
printf ("\nsdma: %s \n\n", erms);
* set default parameter values
* signal, that all initialization is done
4.4.2 THE ACQUISITION LOOP
After the initialization phase, the acquisition process has to wait for the start command. After receiving the start command, it can do some further initialization steps and will then signal the capture process, that the main process is now ready to get data. The sdmaStartCapture() function will enter the real time DMA-loop. From now on all further processing has to keep up with the incoming data flow. So the acquisition process should enter the acquisition loop immediately. It waits for the next data buffer, processes it and calls the acknowledge-function sdmaAckData(), when no more processing is done with the current data buffer. The main loop would look like:
* signal capture process, that process is ready to get data
* check, if stop signal has been received
if (!(active = sdmaCheckStop()))
* get pointer to dynamic parameter structure
param = (myPARAM_TYPE *)sdmaParamPtr();
* now process the data somehow
* transfer computed data frames
* let capture process know, that data has been processed
} /* end of acquisition loop */
4.4.3 DATA FRAMES
A frame consists of a frame header and an output ringbuffer. With each frame a frame type (DIT, INT, ...) and a data type (int, float, ...) are associated. There are default frame-types defined within the sdma-submodule:
sdmaFRAME_SNAPSHOT snapshot frame
sdmaFRAME_INTERMEDIATE_DIT intermediate DIT-frame
sdmaFRAME_INTERMEDIATE_INT intermediate INT-frame
sdmaFRAME_SDV standard-deviation frame
sdmaFRAME_HCYCLE1 first half-cycle frame
sdmaFRAME_HCYCLE2 second half-cycle frame
It is also possible to add new frame types to the system. The frame-type has to be a single bit value. The least significant bits are used for the pre-defined default frames. The first unused bit can be retrieved by defining myFRAME_TYPE in the following way:
#define myFRAME_TYPE (sdmaFRAME_USER << 1)
To introduce a new frame type to the system, one has also to specify a frame name and a parameter name (both ASCII-strings). If the parameter string is not empty, the specified parameter should be used to switch the generation of the frame on/off.
if (sdmaAddFrameType(myFRAME_TYPE,
printf ("\nsdma: cannot add frame type - %s \n\n", erms);
Each frame has to be at least double-buffered. Science-frames, which have to be stored in any case should have three ringbuffer elements (this depends on the expected computation rate of the frame and the transfer time including storage on disk). The output ringbuffer has to be allocated by the acquisition process. Then the frame has to be passed to the system:
static int *resMyFrame[2]; /* data-ring buffer of my-frame */
static sdmaFRAME myFrame; /* frame structure for my-frame */
resMyFrame[i]=(int *)malloc (numPix * sizeof(int));
printf ("\nsdma: allocation error \n\n");
myFrame.h.dtype=sdmaDTYPE_INT32;
sdmaInitFrame (&myFrame, (char **)resMyFrame, 2);
The ringbuffer element, that has to be used for processing the frame, is assigned by the system and is always stored in the structure element myFrame.dptr. So all processing has to be done on resMyFrame[myFrame.dptr]. If the frame should be transferred, the output request function has to be called:
* transfer computed data frames
myFrame.h.setupId = param->setupId;
* request for parallel output of my-frame
sdmaReqOut (sdmaQUEUE_SKIP, &myFrame, 1, -1, -1);
The setupId header element is used by the receiving process to identify frames belonging to a certain parameter setup. This is needed, as parameters may change, while the system is running.
The first parameter of the sdmaReqOut() function is used to specify the behavior, when the output ringbuffer of the frame is full. It has to be one of the following values:
sdmaQUEUE_SKIP - skip frame, if queue is full
sdmaQUEUE_BLOCK - block until queue is free
sdmaQUEUE_SETERR - set error in frame header and skip
The last three parameters specify the transfer target. The first flag determines, whether the frame should be given to the data-output server for transfer via the network. The next is used to specify a camera descriptor for output on a local rtd (this is obsolete and should be set to -1). The last is used to specify a file-descriptor for output on the local disk. The file descriptor has to be retrieved by the sdmaInitFileServer() function. If the file-descriptor is set to -1, no output on the local disk is done.
Our acquisition process now looks like:
static short *data; /* pointer to data buffer */
static myPARAM_TYPE *param; /* pointer to parameter structure */
static int *resMyFrame[2]; /* data-ring buffer of my-frame */
static sdmaFRAME myFrame; /* frame structure for my-frame */
int main (int argc, char *argv[])
int nx=256; /* default chip size */
int ny=256; /* default chip size */
short *dataIn[1]; /* pointer to data buffer */
if (sdmaStartup (argc, argv, (nx*ny)*2, erms) == -1)
printf ("\nsdma: %s \n\n", erms);
if ((param = (myPARAM_TYPE *)sdmaSetupDynParam (sdmaNUM_ELE(paramId),
(char **)paramId, 0, erms)) == NULL)
printf ("\nsdma: %s \n\n", erms);
if (sdmaAddFrameType(myFRAME_TYPE,
printf ("\nsdma: cannot add frame type - %s \n\n", erms);
resMyFrame[i]=(int *)malloc (numPix * sizeof(int));
printf ("\nsdma: allocation error \n\n");
myFrame.h.dtype=sdmaDTYPE_INT32;
sdmaInitFrame (&myFrame, (char **)resMyFrame, 2);
* set default parameter values
* signal, that all initialization is done
* signal capture process, that process is ready to get data
* check, if stop signal has been received
if (!(active = sdmaCheckStop()))
* get pointer to dynamic parameter structure
param = (myPARAM_TYPE *)sdmaParamPtr();
* now process the data somehow
/* compute resMyFrame[myFrame.dptr]... */
* transfer computed data frames
myFrame.h.setupId = param->setupId;
* request for parallel output of my-frame
sdmaReqOut (sdmaQUEUE_SKIP, &myFrame, 1, -1, -1);
* let capture process know, that data has been processed
} /* end of acquisition loop */
4.4.4 PARALLEL PROCESSING
The processing algorithm is application dependent. All calculation should be done using parallel threads, to use full computing bandwidth of the multi-processor CPU. The sdma-library contains functions for starting and synchronizing calculation threads:
static void calcThread(void *arg)
int id = (int)arg; /* process id */
int n = sdmaProcNum(id); /* process number */
int size = numPix/numProc; /* number of pixels per process */
pMyFrame = resMyFrame[myFrame.dptr]+(n*size);
* just fill buffer with some data
int main (int argc, char *argv[])
int numProc = 4; /* should be number of CPUs */
if ((setId = sdmaProcCreate ((void *)&calcThread, numProc, erms)) < 0)
printf ("\nsdma: error: %s\n\n",erms);
* now process the data in parallel
4.4.5 USER DEFINED COMMANDS
It is also possible to add user defined command handling to the acquisition process. A command and a reply buffer are passed unchecked via sdmaExecUsrCmd() to an external command handling routine. This routine is set by the acquisition process and can vary between the readout modes. To implement such a command handler one has to set the global function sdmaUsrCmdHandler to a command handling routine during the initialization phase:
static void myCommandHandler (char *command, char *reply)
* parse command string (format is unspecified)
if (strcmp(command, "MYCMD") == 0)
* all synchronization with the main thread
else if (strcmp(command, "NOP") == 0)
sprintf(reply, "unknown command");
int main (int argc, char *argv[])
sdmaUsrCmdHandler = myCommandHandler;
The sdmaUsrCmdHandler function defaults to NULL (no external handler). The sdmaExecUsrCmd() function can pass any command/reply string to the external handler. The string length is limited to sdmaMAX_USRCMD_LEN characters.
4.4.6 DOWNLOADING OF DATA-SETS
The processing task has the possibility to reserve memory that can be accessed from remote via the command interpreter. This memory area(s) may contain flatfield(s), badpixel-mask(s) or other data sets used for the pre-processing of image data. The buffer can be set from the host process by passing dataSetId and size to the command interpreter with sdmaDownloadBuf(). A blocksize for the transfer has also to be specified. If this blocksize exceeds the maximum command-buffersize, the command interpreter would return a socket (server address and port number) in the reply to the host process. Afterwards it waits for size bytes to be received on this socket. As the reserved memory is double buffered, the readout-mode process would not block while a download is in progress. If all buffers are set, a sync command (sdmaMsync()) has to be sent to the command interpreter to perform the memory swap. The memory areas can be placed in shared memory to be shared between different readout modes.
#define myDATA_SET_ID 4 /* has to be a single bit value */
sdmaShDataGetPtr ((char **)&myDataSet, myDATA_SET_ID, size, erms);
/* process data frame: frame = f(dataIn, myDataSet) */
4.4.7 COUNTER RESET AND EXPOSURE END FLAG
There are two global flags for exposure control. They are set asynchronously via the command interpreter task and can be checked at run-time within the acquisition loop. After checking both have to be reset by the application.
The counter reset flag (sdmaResetCnt) is set, when the exposure should (re-)start while the sequencer is left running. Typically at this point the observing conditions have changed (telescope move etc.). This means, that the DMA-input buffer has to be flushed (sdmaFlushInput()) and the current integration has to be skipped. This has to be taken into account when estimating the exposure time. In the worst case one integration is lost.
The exposure end flag (sdmaEndFlg) is set, when the exposure should end immediately. Typically now an intermediate result should be transferred.
4.4.8 REAL-TIME PERFORMANCE
To get real-time performance, super-user privileges must be given to the acquisition process:
The priorities for the command interpreter task, for the capture task and for the data-output task are set internally. The processing task has to set its own priority during the initialization phase:
sdmaSetPrio (sdmaPRI_PROC, sdmaPRI_RT, sdmaTQ_INF);
sdmaSetPrio (sdmaPRI_PROC, sdmaPRI_TS, sdmaTQ_DEF);
The sdmaPRI_PROC constant defines the maximum priority, that can be used by the processing task. When sdmaPRI_RT is used, the process runs as real-time class. sdmaPRI_TS lets the process run as time-sharing class. Either an infinite (sdmaTQ_INF) or default (sdmaTQ_DEF) time quantum can be used. It is also possible to specify the time quantum in microseconds. The time quantum value is the maximum amount of time a process may run assuming that it does not complete or enter a resource or event wait state (sleep). Note that if another process becomes runnable at a higher priority, the currently running process may be preempted before receiving its full time quantum. Each thread can set its own priority via the sdmaSetPrio() function.
4.5 COMMAND INTERFACE
The command interpreter task has the highest real-time priority (to be able to send commands at any time). It handles the commands and synchronizes them with the capture task, the data-output task and the main processing task. All parameters defined by the processing task (plug-in) are double-buffered and can be set also during run-time. The parameter values are overtaken synchronously with a sync-command.
4.6 DATA INTERFACE
The data frames can be requested in parallel from the output link processes. The data transfer is done via socket (tcp). The sdmaOpenDataLine() routine distinguishes between video- and science-data transfer. In science mode the oldest available, but not yet transferred frame, which matches the frame-type in the request structure is transferred. In video-mode for each frame-type matching the request, the latest not yet transferred frame is selected. From this selection the oldest frame is transferred. In both cases `not yet transferred' means, that the frame has not yet been transferred to the requesting client. In video-mode additionally the retransmit flag in the request-structure is checked, to allow frames of the specified type to be transmitted again. This can be used, to display different windows of the same frame during long integrations.
Format of the request structure:
int type; /* requested frame type */
int blocking; /* block until frame is ready */
int retransmit; /* transfer last buffer again */
int start_x; /* window start-x */
int start_y; /* window start-y */
Format of the returned frame structure:
int start_x; /* window start-x */
int start_y; /* window start-y */
int overrun; /* overrun flag */
int frames; /* available frame types */
sdmaFRAME_H h; /* data header */
char * fbuf; /* data buffer */
The sequence for the function calls should be the following:
- sdmaPingDataLine() (optional);
- sdmaOpenDataLine(serverName, &socket, ...);
- sdmaSendDataRequest(socket, request, ...);
- sdmaWaitDataReply(socket, &frameHeader...);
- if frame type is sdmaFRAME_NO_FRAME, then send a new data request. Otherwise do either transfer the frame (window parameters in request can be updated according to the frame type) with sdmaAcceptFrame(socket, request, &frame, ...) or skip it with sdmaSkipFrame(socket, ...);
- the frame name can be resolved with the sdmaGetFrameNameByType() function (optional);
- the frame can be scaled using the general sdmaScale(&frame) function (optional);
- continue with sdmaSendDataRequest(socket, request);
The socket can be used for a select() call before sdmaWaitDataReply. To cancel a request one has to call sdmaCancelReq(socket, ...) before sdmaWaitDataReply() and then send a new request with sdmaSendDataRequest().
If any of these functions fails, one should call sdmaCloseDataLine() and then try to reopen with sdmaOpenDataLine() to recover the data-channel from failure.
The calling program should not exit with an open request. When the process is waiting for the data reply, sdmaAbortReq(serverName, pid, ...) or sdmaCancelReq(spcket, ...) have to be called before exiting. It is recommended to add sdmaAbortReq() or sdmaCancelReq() to a signal handler.
The sdmaAcceptFrameN(socket, request, &frame, partNo, numDiv, ...) function can be used instead of sdmaAcceptData() to request a frame partition. The additional parameters <partNo> and <numDiv> have to be supplied to indicate that the partition number <partNo> out of <numDiv> sub-divisions of the frame has to be taken as base for the requested window.
The data reception task sdmaDart can be used to test the data transfer. It uses the above routines and can be taken as a template for other customized data reception tasks:
 Quadralay Corporation http://www.webworks.com Voice: (512) 719-3399 Fax: (512) 719-3606 sales@webworks.com |
| |
|
|
|
