leda Pipeline processing 
| HyperLeda home | Documentation | Search by name | Search near a position | Define a sample | SQL Search |

| HFA general doc. | Structure | Keywords | FAQ |

The data contained in the FITS archive can be processed on-line with pipeline procedure.

This page introduces the syntax of the pipeline processing and describes the commands implemented on the pipeline. You may also be interested to see examples of pipelines.

Introduction

The first version of the pipeline processing has been implemented during the period 1999 January to June. The procedures currently publicly available allow to apply the basic steps of data reduction to the files archived in the Fits archive. In the future, new procedures will be implemented and they will be applicable to other material through data-mining over the network.
The data are archived as they are provided to HyperLeda. They are at various stages of processing; some are fully calibrated (flat-field corrected, wavelength calibrated ...), other are raw data. The task of archiving consists in completing and standardizing the description of the data given in the FITS headers. This description must allows an automatic processing of the data.

The pipeline processing is divided in three parts:

  1. Basic pipeline (pipe0). It consists in the basic calibrating operations (flat-fielding, wavelength and flux calibrations). By default, these operations are applied when extracting the data but the user can desable some of them.
  2. Data processing and analysis pipeline. Different data processing functions are implemented on the pipeline:
  3. Data evaluation In addition these functions affecting the data, visualisation and data evaluation function are available and can be inserted at any place along the pipeline. (the data evaluation function have no effect if the output of the pipeline is not an HTML page).

The output of the pipeline can be directed to an Hypertext page, in such case the effect of the pipeline processing can only be evaluated through the visualisation and data evaluation functions. It can also be downloaded as a FITS file, either to a file or to a fits viewer.

The pipeline functions may be selected from the HTML editor displayed when opening a file form the fits archive. They can also be easily accessed with a utility program like wget. The parameters to the HyperLeda request are normally passed with the GET method, and you simply have to learn the syntax and the meaning of the arguments to make requests more elaborated than what permet the editor. We are trying to keep the editor simple enough to allow an easy and fast evaluation of the content of the archive.

The evolution of the system has been very rapid in the recent months and in many places the documentation is out of date. We are currently working to remedy this point. We do not foresee important change in the pipeline software before the end of this year (1999).

Flat-Field correction

The Flat-field correction subtracts the bias and divides by the flat-field. The bias may be a file in the archive indicated by the keyword H_ASP001. If this keyword is not defined, the bias is assumed to be a constant value taken from the keywords H_BIAS. The Flat-field is a file in the archive indicated by the keyword H_ASP002.

Wavelength resampling

The wavelength resampling uses the wavelength calibration relation pointed to by the keyword H_WCALIB.
The program allows to resample in wavelength or logarithm of wavelength and to choose the sampling limits and the step. The pipeline editor, however, provides only a wavelength resampling with a resampling factor of 1.
The resampling is based on a cubic spline interpolation.

Flux calibration

Only for spectroscopy.
The flux calibration consists in dividing each scan of the spectrum by the flux calibration relation stored in a FITS HDU pointed to by a keyword H_FCAPHY, H_FCANOR, ....

Flux normalisation

Only for spectroscopy.
The flux normalisation consists in dividing each scan of the spectrum by a constant determined by averaging the concerned scan between two wavelengths.

The flux can be returned in a linear scale (this is the default: Lfc=1) or in decimal logarithm Lfc=2. The second argument Syst indicates the coordinate system used for the last two arguments. The value can be m (default) n or w meaning respectively: maximum range, world coordinates or natural coordinates. W1 and W2 are these coordinates.

The vector used to normalize the flux is stored in an HDU appended to the current FITS file.

Extraction of a sub-image

The extraction has two methods. c for an extraction specified by the coordinates of the region, or o for and extraction around the current object. arguments are passed respectively to xtc or xto

Extraction of a sub-image given by its coordinates

e1 to e4 are the coordinates of the first pixel, first line, last pixel and last line. Each coordinate has the form Si where i is a number and Sspecily the coordinate system:

Extraction of a sub-image around an object

Extract a region around the object currently selected.
The extraction region is determined using the keywords: H_OBJP1, H_OBJP2 and H_RADIUS, and possibly uses also the keywords defining the scale.

e1 and e2 are the extraction sizes along each axes, if there is only one spatial axis, e2 may be omitted and otherwyse it is ignored. system stand for n,w,w or a, with the meaning:

Project (2D-> 1D)

Combine the lines or columns of a 2D image to produce a 1D image.
method can be a, m or o corresponding to average, median or optimum extraction. axis is the number of the axis onto which data are projected.

Sky subtraction

The sky is computed on both side of the object and is subtracted to the whole image.
The sky regions are detemined using the keywords: H_CENT2 and H_RADIUS.

In each sky region the sky is computed in each wavelength bin as the median of the pixel values in the sky region.
The sky spectrum is stored in the fits extension named: SKY.

Present limitations:
The subtracted sky is the average betwwen the two sky region. It will probably be upgraded to a linear interpolation between these two regions.

Remove the spikes

The algorihm detects spikes, ie. features narrower than fwhm which are above a threshold of signi times the noise. The process may be iterated up to iter times in order to remove elongated events.

It is advised to insert this function on the pipeline before any resampling since spikes contain information at the cut-off spatial frequency (by definition) and hence are poorly interpolated. Interpolation usually result in a smearing (making their afterward detection harder) or worse in oscilations (eg. with splines interpolator) producing some "replicas" of the spike in its wings.

Shift and broaden a spectra

Shift a spectra by the amount cz (in km/sec), broaden it by a gaussian of standard deviation sigma and multiply it by g.

The convolution proceeds in the Fourier space from spectra sampled in log-wavelength. Hence, if necessary (ie. if the spectra is calibrated in wavelength), this function calls wrs before the convolution to resample in logarithm (oversampling by a factor 2) and calls it again after the convolution to restore the sampling. This is time consuming and resampling is a source of degradation, so, try to minimize the number of resampling.

vsg cannot be performed if the spectrum has not been calibrated in wavelength.

Mask a sub-image

e1 to e4 are the coordinates of the first pixel, first line, last pixel and last line. Each coordinate has the form Si where i is a number and Sspecily the coordinate system:

repval is the replacement value . If it is not given the pixels will be set to the usual "not a number" value.

This procedure may be used, for example, to mask the regions of telluric lines in a spectrum.

Apply a filter

filter is the name of the filter p1, p2 ... are the parameters of the filter.

Available filters are (i) gaussian convolution and (ii) gaussian unsharp masking.
gaussian convolution: The image is convolved by a gaussian of standard deviation p1 (in pixels) along both axes.
gaussian unsharp masking: The image is convolved by a gaussian of standard deviation p1 (in pixels) along both axes and the function returns the difference between the original image and the convolved image.

Gray scale display

Ommit the arguments, or set them to blank to have an automatic determination of the cuts.

HyperLeda Questions: leda@univ-lyon1.fr