Update on 12/02/2008

---------
Lin/Gain:
---------

Templates and Data to be sent from Paranal.
Integration into as many pipelines as possible after receiving previous data.

Tickets open: 4918, 4942 to be implemented in this order.

Future integration in CPL: 
Definition of API and level of function to be moved to CPL.

Documentation

---------
RON/Bias:
---------

Test Report from DFO.

Extension to NIR?

Documentation

---------
PerNoise:
---------

Test Report from DFO.

Documentation

---------
Dark:
---------

Test Report from DFO.

Documentation


CHECKS on 0.0.6
Feedback from wolfgang:

===============================
Hi Lander,

The contamination values of the gainlin opt (2 and 5) are not correct.

Wolfgang


HIERARCH ESO PRO REC1 PARAM18 NAME = 'llx1    '
HIERARCH ESO PRO REC1 PARAM18 VALUE = '500     '
HIERARCH ESO PRO REC1 PARAM19 NAME = 'lly1    '
HIERARCH ESO PRO REC1 PARAM19 VALUE = '500     '
HIERARCH ESO PRO REC1 PARAM20 NAME = 'urx1    '
HIERARCH ESO PRO REC1 PARAM20 VALUE = '510     '
HIERARCH ESO PRO REC1 PARAM21 NAME = 'ury1    '
HIERARCH ESO PRO REC1 PARAM21 VALUE = '510     '
HIERARCH ESO PRO REC1 PARAM22 NAME = 'llx2    '
HIERARCH ESO PRO REC1 PARAM22 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM23 NAME = 'lly2    '
HIERARCH ESO PRO REC1 PARAM23 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM24 NAME = 'urx2    '
HIERARCH ESO PRO REC1 PARAM24 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM25 NAME = 'ury2    '
HIERARCH ESO PRO REC1 PARAM25 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM26 NAME = 'llx3    '
HIERARCH ESO PRO REC1 PARAM26 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM27 NAME = 'lly3    '
HIERARCH ESO PRO REC1 PARAM27 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM28 NAME = 'urx3    '
HIERARCH ESO PRO REC1 PARAM28 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM29 NAME = 'ury3    '
HIERARCH ESO PRO REC1 PARAM29 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM30 NAME = 'llx4    '
HIERARCH ESO PRO REC1 PARAM30 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM31 NAME = 'lly4    '
HIERARCH ESO PRO REC1 PARAM31 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM32 NAME = 'urx4    '
HIERARCH ESO PRO REC1 PARAM32 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM33 NAME = 'ury4    '
HIERARCH ESO PRO REC1 PARAM33 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM34 NAME = 'llx5    '
HIERARCH ESO PRO REC1 PARAM34 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM35 NAME = 'lly5    '
HIERARCH ESO PRO REC1 PARAM35 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM36 NAME = 'urx5    '
HIERARCH ESO PRO REC1 PARAM36 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM37 NAME = 'ury5    '
HIERARCH ESO PRO REC1 PARAM37 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM38 NAME = 'exts    '
HIERARCH ESO PRO REC1 PARAM38 VALUE = '0       '
dfo27 isaac:/data27/detmon/detmon_demo/test 1286 >dfits out_0011.fits | grep QC
HIERARCH ESO QC CONTAM1 = 877.166687011719
HIERARCH ESO QC CONTAM2 = 3.66667175292969
HIERARCH ESO QC CONTAM3 = 811.333312988281
HIERARCH ESO QC CONTAM4 = 861.
HIERARCH ESO QC CONTAM5 = 5.66667175292969
HIERARCH ESO QC CONAD FIT = 1.59323828370586
HIERARCH ESO QC GAIN FIT = 0.627652505106775
HIERARCH ESO QC GAIN FIT MSE = 107661.003986851
HIERARCH ESO QC READNOISE = -193.108842124702
HIERARCH ESO QC GAIN MED = 0.634522694142863
HIERARCH ESO QC GAIN STD = 0.00699729865446722
HIERARCH ESO QC LAMP FLUX = 1565.62485738455
HIERARCH ESO QC LIN COEF1 = 0.00060547302685002
HIERARCH ESO QC LIN COEF2 = -1.50220905656487E-10
HIERARCH ESO QC LIN COEF3 = 4.04248060512791E-16
HIERARCH ESO QC AUTOCORR = 1.00274129219596
HIERARCH ESO QC GAIN CORR = 0.632828992957655
HIERARCH ESO QC NUM BPM =   0.
HIERARCH ESO PRO REC1 PARAM18 NAME = 'llx1    '
HIERARCH ESO PRO REC1 PARAM18 VALUE = '500     '
HIERARCH ESO PRO REC1 PARAM19 NAME = 'lly1    '
HIERARCH ESO PRO REC1 PARAM19 VALUE = '500     '
HIERARCH ESO PRO REC1 PARAM20 NAME = 'urx1    '
HIERARCH ESO PRO REC1 PARAM20 VALUE = '510     '
HIERARCH ESO PRO REC1 PARAM21 NAME = 'ury1    '
HIERARCH ESO PRO REC1 PARAM21 VALUE = '510     '
HIERARCH ESO PRO REC1 PARAM22 NAME = 'llx2    '
HIERARCH ESO PRO REC1 PARAM22 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM23 NAME = 'lly2    '
HIERARCH ESO PRO REC1 PARAM23 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM24 NAME = 'urx2    '
HIERARCH ESO PRO REC1 PARAM24 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM25 NAME = 'ury2    '
HIERARCH ESO PRO REC1 PARAM25 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM26 NAME = 'llx3    '
HIERARCH ESO PRO REC1 PARAM26 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM27 NAME = 'lly3    '
HIERARCH ESO PRO REC1 PARAM27 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM28 NAME = 'urx3    '
HIERARCH ESO PRO REC1 PARAM28 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM29 NAME = 'ury3    '
HIERARCH ESO PRO REC1 PARAM29 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM30 NAME = 'llx4    '
HIERARCH ESO PRO REC1 PARAM30 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM31 NAME = 'lly4    '
HIERARCH ESO PRO REC1 PARAM31 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM32 NAME = 'urx4    '
HIERARCH ESO PRO REC1 PARAM32 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM33 NAME = 'ury4    '
HIERARCH ESO PRO REC1 PARAM33 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM34 NAME = 'llx5    '
HIERARCH ESO PRO REC1 PARAM34 VALUE = '20      '
HIERARCH ESO PRO REC1 PARAM35 NAME = 'lly5    '
HIERARCH ESO PRO REC1 PARAM35 VALUE = '900     '
HIERARCH ESO PRO REC1 PARAM36 NAME = 'urx5    '
HIERARCH ESO PRO REC1 PARAM36 VALUE = '30      '
HIERARCH ESO PRO REC1 PARAM37 NAME = 'ury5    '
HIERARCH ESO PRO REC1 PARAM37 VALUE = '910     '
HIERARCH ESO PRO REC1 PARAM38 NAME = 'exts    '
HIERARCH ESO PRO REC1 PARAM38 VALUE = '0       '
dfo27 isaac:/data27/detmon/detmon_demo/test 1286 >dfits out_0011.fits | grep QC
HIERARCH ESO QC CONTAM1 = 877.166687011719
HIERARCH ESO QC CONTAM2 = 3.66667175292969
HIERARCH ESO QC CONTAM3 = 811.333312988281
HIERARCH ESO QC CONTAM4 = 861.
HIERARCH ESO QC CONTAM5 = 5.66667175292969
HIERARCH ESO QC CONAD FIT = 1.59323828370586
HIERARCH ESO QC GAIN FIT = 0.627652505106775
HIERARCH ESO QC GAIN FIT MSE = 107661.003986851
HIERARCH ESO QC READNOISE = -193.108842124702
HIERARCH ESO QC GAIN MED = 0.634522694142863
HIERARCH ESO QC GAIN STD = 0.00699729865446722
HIERARCH ESO QC LAMP FLUX = 1565.62485738455
HIERARCH ESO QC LIN COEF1 = 0.00060547302685002
HIERARCH ESO QC LIN COEF2 = -1.50220905656487E-10
HIERARCH ESO QC LIN COEF3 = 4.04248060512791E-16
HIERARCH ESO QC AUTOCORR = 1.00274129219596
HIERARCH ESO QC GAIN CORR = 0.632828992957655
HIERARCH ESO QC NUM BPM =   0.

===================================
The crires raw frames have keys like:

EXTNAME = 'CHIP1.INT1' / FITS extension name
EXTNAME = 'CHIP2.INT1' / FITS extension name
EXTNAME = 'CHIP3.INT1' / FITS extension name
EXTNAME = 'CHIP4.INT1' / FITS extension name

These keys are missing in the products.

As recently discussed,
the main header of the raw frame should go to the main header of the product and
the extension header of the raw frame should go to the extension header of the product.

Wolfgang

PS;I will soon get UVES red-arm frames with extension, for which I can test the
recipe as well.

=================================
The crires raw frames have keys like:

EXTNAME = 'CHIP1.INT1' / FITS extension name
EXTNAME = 'CHIP2.INT1' / FITS extension name
EXTNAME = 'CHIP3.INT1' / FITS extension name
EXTNAME = 'CHIP4.INT1' / FITS extension name

These keys are missing in the products.

As recently discussed,
the main header of the raw frame should go to the main header of the product and
the extension header of the raw frame should go to the extension header of the product.

Wolfgang

PS;I will soon get UVES red-arm frames with extension, for which I can test the
recipe as well.

================================
uves:

I get two products MASTER_BIAS and BPM
The BPM I cannot load into rtd

dfits out_0001.fits   | grep NAXIS
NAXIS   =                    2 / number of data axes
NAXIS1  =                 1074 / length of data axis 1
NAXIS2  =                  125 / length of data axis 2
dfits out_0000.fits   | grep NAXIS
NAXIS   =                    2 / number of data axes
NAXIS1  =                 1074 / length of data axis 1
NAXIS2  =                 1000 / length of data axis 2

=================================
the preover scan regions are currently speciief in the recipe config.
The PREOVER region can be read from the fits header:
The configuration options should be dropped.

NAXIS   =                    2          / # of axes in data array
NAXIS1  =                 2048          / # pixels/axis
NAXIS2  =                 1034          / # pixels/axis
HIERARCH ESO DET OUT1 X      =            1 / X location of output
HIERARCH ESO DET OUT1 Y      =         1024 / Y location of output
HIERARCH ESO DET OUT1 NX     =         2048 / valid pixels along X
HIERARCH ESO DET OUT1 NY     =         1024 / valid pixels along Y
HIERARCH ESO DET OUT1 PRSCX  =            0 / Prescan region in X
HIERARCH ESO DET OUT1 OVSCX  =            0 / Overscan region in X
HIERARCH ESO DET OUT1 PRSCY  =            5 / Prescan region in Y
HIERARCH ESO DET OUT1 OVSCY  =            5 / Overscan region in Y

Note: OUT1.PRSCY + OUT1.NY + OUT1.OVSCY = NAXIS2

=================================
Hi Lander,

I tested UVES red -arm data that have two extensions.
The products have only one extension although :

# --exts
# Activate the multi-exts option.
detmon.detmon_opt_lg.exts=1

=================================
Hi Lander

- please rename the detmon_dark recipe to detmon_opt_dark recipe, as it is
intended for OPT only.

- please rename the detmon_ronbias to detmon_ron, as it is used for BIAS and NIR-DARK
frames.

- the detmon_ron recipe works on single raw frames. So it cannot be applied for NIR frames,
and other BIAS frames with signatures like string fixed pattern.

- For methods RANDOM, HISTO and REGION, there should be an option to apply the readnoise to raw difference frames.
an QC parameter should indicate to which kind of frame the method was applied.
When applied to difference frames, RON must be divided by sqrt(2).
In any case, for the ir version the RON must be scaled by sqrt(NDIT), to get the ron at NDIT=1.

Wolfgang 

=================================
Her is the list of  instrument specific setup keys to be included in the monthly detmon recipe
(gainlin recipe). They should be included in PAF and hence in OBS log file:

Wolfgang

SINFONI:
INS.SETUP.ID


ISAAC:
DET.CHIP.NAME
DET.MODE.NAME

NACO:
DET.MODE.NAME
DET.NCORRS.NAME
INS.OPTI7.NAME
INS.OPTI6.NAME
INS.OPTI5.NAME
INS.OPTI4.NAME

CRIRES:
DET.CHIP.NO
DET.NCORRS.NAME
EXTNAME

UVES:
INS.PATH
DET.NAME
DET.READ.CLOCK
DET.WIN1.BINX
DET.WIN2.BINY
DET.READ.SPEED

GIRAFFE:
DET.READ.CLOCK

FORS1
DET.READ.CLOCK
DET.CHIP.ID
INS.COLL.NAME
DET.WIN1.BINX
DET.WIN1.BINY

FORS2:
DET.READ.CLOCK
DET.CHIP.ID
INS.COLL.NAME
DET.WIN1.BINX
DET.WIN1.BINY

VIMOS:
DET.READ.CLOCK
OCS.CON.QUAD
=================================

CHECKS on 0.0.5
---------------------------
notes on detmon-kit-0.0.5:

Installation worked only after changing automake files and install_pipeline 
script. 

FFTW seems still not to be usable: (lg recipe issue a warning)

detmon_dark:
man-page is too compact: I would add a brief description of what the recipe 
does, what are input and output files (with PRO.CATG)

input recipe parameters should be clearly described with all allowed options
(eventually indicating when each option is appropriate).


The input temperature value should appear in the paf and possibly also in the table (as an extra column).

RON computation?


make distcheck is failing


Requests on 0.0.1:
DET WIN1 UIT1 should be used instead of EXPTIME
QC related keys should get a comment
DET.NCORRS.NAME READMODE are still missing in paf file



--------------------------
high:                -the gain value computed using autocor is not as expected. This is probably due to the improper
                      input data. We have to solve this issue with Paola (get proper data or drop the method).

high: ronbias        -The size of the BPM is wrong.  It should be the same of the frame(s) it refers to.
                      In case you analize a frame sub region, it should flag bad pixels only in that sub-region

high: ronbias        -The intensity of the BPM is wrong. I would expect 0-INTEGER_CODE_FOR_EACH_KIND_OF_BAD_PIXEL

low                  -is there a difference between llx,lly,urx,ury reference region's corners and 
                      the llx#...contamination region corners? if so may be one can add a prefix to the parameter name.

low                  -gain computation without FFTW: for the estimation time put a value per nb: 3m30s*nb, 
                      which is more realistic and useful if one does not have a given number of data ...  
                      update warning message

med                  -profile gain computation without FFTW (using a single data set). Then try to speed-it-up



question:            -For OPT the function gain(DIT) is quite strange (iperbola). What do we expect? 
                      The main problem is to find the constant.
question:            -For NIR the function gain(DIT) is quite strange . What do we expect?
                      The main problem is to find the constant.



/**************************/
/*     ALREADY SOLVED:    */
/**************************/

high:                -LINEAR_INFO table contains NULL in ADL column. Why?Fix it.
SOLVED, it wasn't implemented. Implemented now following SINFONI 
example.


high: ronbias        -Pls add comments to QC values. Some QC has strange values:
QC.BIAS.RANDOM.RMS    0.05441153511:  why so low if other methods gives reasonable values of 2-5?
SOLVED, there was a bug.

QC.BIAS.HISTO.BIAS.STDEV 0         : Why 0? Is there a typo in the QC name?
This is the stdev of biases computed through histograms for each raw frames. I do no interpolation yet to compute
the max of the histogram, so it is logic that all raw frames will have the max at the same bin and therefore STDEV will be 0.

QC.BIAS.HISTO.RMS     1.128775856a : This differes from other results. Check formula is correct.
SOLVED, there was a bug.


QC.BIAS.HISTO.RMS.STDEV 1.307324768: What is this (compared to previous?)
The previous value is the mean of rms (computed through histogram) for each raw frame. This is the stdev of those
values computed for each frame.



QC.BIAS.OVERSCAN.RMS  60.653797    Why so high? Does it include bad pixels?
SOLVED, bad pixels taken now into account when computing.

QC.BIAS.PRESCAN.MEAN.STDEV 0.0485800401 :Why so low?
QC.BIAS.PRESCAN.MED.STDEV 0             :Why 0?  shouldn't be ~141
QC.BIAS.PRESCAN.RMS.STDEV 0.008790412464: Why so low? 
Same as above, this are stdev through raw frames... it is logic that PRESCAN.MEAN will be similar for different raws, that's why
these values are low (note QC.BIAS.PRESCAN.RMS = 3.79 and != QC.BIAS.PRESCAN.RMS.STDEV)

QC.BIAS.OVERSCAN.MEAN.STDEV 0.2790323214: Why so low?
QC.BIAS.OVERSCAN.MED.STDEV 0              Why so low
QC.BIAS.OVERSCAN.RMS.STDEV 75.26719867    Why so high? Does it include bad pixels?
Same as above about .STDEV (High value to be investigated yet, too).

QC.BIAS.REGION.RMS    13.87589966         Quite high
SOLVED, bad pixels taken now into account when computing.

QC.BIAS.REGION.MED.STDEV 0                Why 0?
QC.BIAS.REGION.RMS.STDEV 7.810128747
QC.MASTER.MEAN        141.9918225
QC.MASTER.RMS         15.57306709         Quite high
SOLVED, bad pixels taken now into account when computing.

low    detmon_opt_lg: call the reference region parameters with the same index as the corresponding order:
                      llx1: refers to.....first region...
                      llx2: refers to.....second region...
                      ...
                      llx5: refers to.....fifth region...
SOLVED, and thanks for spotting such a stupid error :-)

question:            -is it possible to specify the value of a PRO.CATG from the function interface?
                      Should we use common PRO.CATG for different instruments (this would mean a change for DFO)
                      or be able to set new names ?
SOLVED.
The PRO.CATG are part of the interface.


SOLVED
PRO.CATG of products should be:
MASTER_DARK
DSNU_TABLE
DSNU_MAP

SOLVED
linearity recipe: if temporary products are created it fails.
