leda The FITS archive: How to archive a new dataset site

| HyperLeda home | Documentation | Search by name | Search near a position | Define a sample | SQL Search |


| Fist Archive | Structure of HFA | Keywords | FAQ
| Introduction | Archived data | Software | Pipeline |



This page explains the different steps for archiving a new dataset.

Introduction

Archiving a dataset should be implemented in the "data-flow" between the acquisition at the telescope and the reduction, in other words the data are archived as the are produced by the telescope and the reduction is performed on the archived data using the pipeline procedure. Archiving data consists in adding in the FITS header all the keywords necessary for a full description of the data.
This principle is often going against the usual practice, we will explain now why you should nevertheless adopt this line and maybe change your habits.
The goal of an astronomer when reducing data is to get his results as fast as possible. In practice, very often the procedures fail and the work has to be re-done in an improved manner. There are two basic alternative ways to make the "re-do" easy:
The second solution is of course better because only one file will need to be edited to change the reduction method. The second method needs the relevant keywords to have been created: this is exactly what "archiving" will do. Anyway, whatever method is adopted the information will have to be given once...
If necessary the two methods can be combined by using the header information when available and otherwise adding specific information in the command files. This will make the command files lighter, but I still think it is not the best solution.
The overwhelming advantages of archiving the data before the data-reduction appear when the data are to be re-used, sometimes years after, in order to measure other parameters (for example do spectrophotometry of data which were first used for kinematics) or to apply new reduction methods (eg. improved description of the LOSVD). Even if the command files were carefully kept, it will not be straightforward to re-use them: information like object identification or position will have to be re-determined, and hence possibly incompatible with the initial reduction.

My advice: Load as much information as possible early in the data-reduction

Pre-requisites

Steps

  1. If you have enough space on disk, copy the whole dataset to a new directory (for security). The name of the directory containing the fits file should be the identifier of the dataset. This is a 9 digit character string, obeying a (somewhat flexible) convention. For example, L99021HP2 is decoded as:
    L  -> archived in Lyon
    99 -> millesime of the year of observations
    02 -> month of observations
    HP -> Haute-Provence Observatory
    2 -> 1.2 m telescope (1 is for the 1.9 m)
  2. Create the files: AAAReadMe and AAAHistory
    These files must be filled with (free text format)
       AAAReadMe: Detailled description of the dataset.
    AAAHistory: Log of the different steps for archiving in HFA.
  3. Uncompress the FITS files (if necessary)
  4. Allow write access to the FITS files (if necessary; chmod +w *.fits)
  5. Rename all the FITS files into filenames formatted as: nnnnn.fits
    Where nnnnn is a running number.
    The raw-observation files must be in the range 00001 to 98999.
    The order is not really important, but it is recommended to use consecutive numbers.
       (to create the numbers you can use something like:
    ls p*.fits | awk '{printf "mv %+13s %05d.fits\n", $0, FNR}'
    )
    The numbers between 99001 and 99999 are reserved for naming the "computed" files, as eg. average flat-field or mean calibration file.
    The program archilist can be used to display a listing of the dataset
  6. Create a file SETUP.fits which will be used to set the default value of some keywords common to the whole dataset execute the PLEINPOT command: archiset
    The program archiset asks the default value for some keywords, try to give as much as possible. If you do not set some values, the other archiving programs will ask them...
    SETUP.fits is used by the archiving programs: archispe and archiver
    You may update SETUP.fits at any time using archiset or possibly using any keyword editing program, for example futil. You can initialize any keyword (associated with a PLEINPOT descriptor), even those not initialized by archiset.
    Note: When you run archiset for the first time, the program will probably not be able to suggest default value for the bias and flatfield files. You may omit to initialize this value directly, and go directly to the program archispe. After archispe you may re-run archiver.
    If the dataset consists in spectra not calibrated in wavelength, archiset asks the "identifier" of the relation of calibration (keyword H_WCALIB). It can be a separate FITS frame, hence identified as eg. L1999101HP1/990005, or an extension attached to any FITS frame, for example: L1999101HP1/990001[WCA] (extension named WCA in L1999101HP1/990001.
  7. Execute archispe to load the keywords in the special files.
  8. Execute archiver to load the keywords in the "regular" files. You may also use any keyword editing program, for example futil, or archikey to edit a keyword in a series of files in the dataset. For example, to set the value of H_AXIS1 for all the regular files of the dataset L1996KINN, you may use: ls L1996KINN/0*.fits | xargs -i futil {} ke H_AXIS1 2
    archiver initialize some kewords mandatory for the archiving. User is prompted to provide (or confirm) the value for these keywords. The default values for the un-initialized keywords are taken from SETUP.fits (see above).
  9. To define the position of the object in the frame, you can use: archiview. This program visualize each frame in turn, and you can use the usual keyboard commands (as in grlise) to set the center (actually ctrl-C).
  10. Spectroscopy: To define the pointers to the arc files taken before and after each science observation, use archiarc
  11. To move the dataset into the HFA, contact the HFA administrator, at present Ph. Prugniel.

drawbacks

For the moment the calibration files (eg. flatfield...) are referred with their identification in the HyperLeda FITS archive: as eg. fa:L95011HP2/99004. For this reason the PLEINPOT programs for processing these files will not work until the dataset is actually moved in HFA.

There is not program (yet) for writing the pointers to wavelength calibration and flux calibration relations (this must e done by hand).

Documentation of PLEINPOT programs used for archiving


HyperLeda Questions: leda@univ-lyon1.fr