6. Local Calibration Database

The Recipe System has a system to retrieve processed calibration automatically. This system must work with a Calibration Manager. Currently, only one public Calibration Manager is available, the Gemini Calibration Manager, GeminiCalMgr. This must be installed as a DRAGONS dependency; a conda install will take care of that (see Installation).

The Calibration Manager contains the calibration association rules and database access hooks. The Gemini Calibration Manager uses exactly the same calibration association rules as the Gemini Observatory Archive (GOA).

The calibration facility requires a database. The Recipe System’s caldb application helps the user configure and create a local, lightweight sqlite database, and add or remove calibration files to and from that database.

In this chapter, we explain how to use caldb to add processed calibrations that the Recipe System will pick up when needed.

Note

We intend to improve the Calibration Manager side of things to make expanding the association rules for new instruments or non-Gemini instruments feasible.

6.1. Configuring caldb

The first time caldb is used for a project, either via command line or API, it needs to be configured and initialized. The configuration is stored in a text file in a special directory named ~/.geminidr/, in a file called rsys.cfg. The ~ means the user’s home directory. The very first step, to be done only once, is to create the directory and the configuration file.

$ mkdir ~/.geminidr
$ touch ~/.geminidr/rsys.cfg

The rsys.cfg file must contain the following lines:

[calibs]
standalone = True
database_dir = ~/.geminidr   # set this path to whatever you want.

The standalone option tells caldb if you are using a local database when it is set to True. standalone = False is used only internally at Gemini when using the internal data manager.

The database_dir parameter points to the directory hosting the calibration database. The database name is always cal_manager.db, this cannot be set, only the directory where it lives. It is possible to have more than one database as long as they are in different directory. Which one will be picked up will be set through the database_dir parameter in rsys.cfg.

6.2. Using caldb on the Command Line

The caldb tool is used to interact with the local calibration database. This is where the Recipe System will look for processed calibrations. For a reminder of its basic usage, one can always use the --help flag:

$ caldb --help
usage: caldb [-h] {config,init,list,add,remove} ...

Calibration Database Management Tool

positional arguments:
  {config,init,list,add,remove}
                        Sub-command help
    config              Display configuration info
    init                Create and initialize a new database.
    list                List calib files in the current database.
    add                 Add files to the calibration database. One or more
                        files or directories may be specified.
    remove              Remove files from the calibration database. One or
                        more files may be specified.

optional arguments:
  -h, --help            show this help message and exit

There can be only one positional argument given to caldb, this means only one file at a time can be added or removed from the database.

Once the configuration file is in place (see Configuring caldb), one can verify the configuration by doing:

$ caldb config

Using configuration file: ~/.geminidr/rsys.cfg
Active database directory: /Users/username/.geminidr
Database file: /Users/username/.geminidr/cal_manager.db

The 'standalone' flag is active, meaning that local calibrations will be used

To initialize a new database with the selected configuration:

$ caldb init

Once the database is initialized (created), it is ready for use.

To add a file:

$ caldb add /path/to/master_bias.fits

If the path is not given, the current directory is assumed. The addition of a file to the database is simply the addition of the filename and its location on the disk. The file itself is not stored. If the calibration file is deleted or moved, the database will not know and still think that the file is there.

To see what is in the database:

$ caldb list
master_bias.fits    /path/to/

To remove a file from the database:

$ caldb remove master_bias.fits

Warning

If a file that is already stored within the database needs updating, it will need to be removed and added again. caldb has no update tool.

To see caldb used in a complete example along with the other tools see Full Command Line Example.

6.3. Using the caldb API

Before being usable in a Python program, the local calibration manager must be configured. This cannot be done from the API. See Configuring caldb for instructions.

The calibration database is initialized and the configuration are read into the the calibration service as follow:

>>> from recipe_system import cal_service
>>>
>>> caldb = cal_service.CalibrationService()
>>> caldb.config()
>>> caldb.init()
>>> cal_service.set_calservice()

The calibration service is then ready to use. This must be done before Reduce is instantiated.

To add a processed calibration to the database:

>>> caldb.add_cal('/path/to/master_bias.fits')

If the path is not given, the current directory is assumed. The addition of a file to the database is simply the addition of the filename and its location on the disk. The file itself is not stored. If the calibration file is deleted or moved, the database will not know and still think that the file is there.

To see what is in the database:

>>> for f in caldb.list_files():
...     print(f)
...
FileData(name=u'master_bias.fits', path=u'/path/to')

To remove a file from the database:

>>> caldb.remove_cal('master_bias.fits')

Warning

If a file that is already stored within the database needs updating, it will need to be removed and added again. caldb has no update tool.

To see it used in a complete example along with the other tools see Full API Example.