Description of the Raw Data Format at BL4/ID2
This is a very fast written description, of what is stored in the header together with the image data. For a summary of all keywords that are currently in use with SAXS data look into the file ~boesecke/programs/work/include/SaxsDefinition.h. This C-header file contains also all transformations that are in use. The actual raw data file of the gasfilled detector, that is produced by spec is defined at the end (see also the C-program: ~boesecke/programs/work/include/GetDldData.C
Center_1 = 23 ;
Center_2 = 24 ;
DDummy = 0.1 ;
Dim_1 = 1024 ;
Dim_2 = 1024 ;
Dummy = -1 ;
Image = 1 ;
Offset_1 = 0 ;
Offset_2 = 0 ;
Psize_1 = 0.025 ;
Psize_2 = 0.026 ;
SampleDistance = 0.995386 ;
SaxsDataVersion = 1.0 ;
Title = 0 ;
WaveLength = 7.69043e-11 ;
The actual structure was implemented on the base of the ESRF data format, which has been constructed Peter Daly. The ESRF data format does not have a very consistent definition and has been criticised for it. The programs used at BL4/SAXS are using the ESRF data format only as a way to store the data. The definition given below are independent of the format and can be implemented in other data bases later. Even the programs are not based on the ESRF data format and can be updated to an other format, if necessary. The data is stored in an ASCII header followed by a binary section. A basic header contains the following keywords (apart from the ESRF data format keywords which define the backbone) are:
/n{/n
...
HeaderID = EH:000001:000000:000000 ;
ByteOrder = HighByteFirst ;
DataType = FloatValue ;
Size = <length of binary data in bytes>;
...
Center_1 = 23 ;/n
Center_2 = 24 ;/n
DDummy = 0.1 ;/n
Dim_1 = 1024 ;/n
Dim_2 = 1024 ;/n
Dummy = -1 ;/n
Image = 1 ;/n
Offset_1 = 0 ;/n
Offset_2 = 0 ;/n
Psize_1 = 0.025 ;/n
Psize_2 = 0.026 ;/n
SampleDistance = 0.995386 ;/n
SaxsDataVersion = 1.0 ;/n
Title = 0 ;/n
WaveLength = 7.69043e-11 ;/n
}/n
Exactly <Size> bytes of <binary data>
Coordinate System
The origin (0,0) is at the lower left corner. The first pixel touches with its lower left corner the origin. The uper edges of the first pixel are 1.0 in both directions. All following pixels are defined accordingly. This definition makes the handling of binning procedures easier.
Remark: This is like it should be, but for the moment there is still a shift by 0.5, 0.5, i.e. the center of the first pixel is (0.0,0.0)).
Therefore: SaxsDataVersion = 1.0 ;
Description of keywords
ESRF Data Format Keywords
HeaderID, ByteOrder, DataType, Size
These are internal keywords of the ESRF data format and will not be discussed.
The header must be padded with blanks that its size is a multiple of 512 bytes.
An ESRF data format file can contain an unlimited number of header/binary data
blocks.
SAXS Format Keywords
The following keywords are followed by integer values:
- Image: |
sequence number of the data in a single data file. The data is processed according to the number following after Image (default 1, but can be any integer number). The image number must be unique in the data file. |
- SaxsDataVersion = 1.0: |
Version of the SAXS data format (must be 1.0) |
- Dim_1, Dim_2: |
Dimension of the actual data array (column, row) |
The following keywords are followed by float values:
- Dummy: |
Dummy value |
- DDummy: |
All value between Dummy-DDummy and Dummy+DDummy are not valid. |
- Offset_1, Offset_2: |
Actual start position (in pixel) the data (in pixels) |
- Center_1, Center_2: |
Position of the primary beam on the detector (in pixels) |
- Psize_1, Psize_2: |
Pixel size in meter |
- SampleDistance: |
Distance between sample and detector center |
(Center_1, Center_2). |
|
- WaveLength: |
wavelength in meter |
- Title: |
A single line string the describes the data. |
Remark: The definition of the center as the point where the primary beam (virtually) hits the detector is only valid if the detector is perpendicular to the primary beam.
Raw Data Format Keywords
The previously defined keywords are used through all the treatment of the data file, raw data keywords are only available in the raw data file.
The raw data files contain a lot of other keywords which are defined in the following way. HM stands for Historgramming Memory, HS is an abbreviation for Histogramming Scaler.
DetectorName = X-ray Image Intensifier with FRELON CCD ;
-should be a unique name of the used detector
DetectorPosition = 0.995139 ;
+ Position of the detector on the bench in meter (inside the detector tube)
The sample distance can be calculated from the DetectorPosition with an
offset. E.g. if the detector is at home (closest distance):
DetectorPosition = 0. The SampleDistance of the X-ray Image Intensifier
from the Sample is DetectorPosition + 1.xxx meter.
HMFile = /data/opid02/inhouse/ccd220897/flatfield2208970002 ;
- full path of the raw data file (info only)
HMFrame = 1 ;
+histogramming memory frame one (number is converted to image number)
HMStartTime = Fri Aug 22 19:18:51 1997 ;
-starting time of the first frame (info only)
HS32C01 = 1.09726e+07 ;
HS32C02 = 1.17135e+07 ;
HS32C03 = 0 ;
HS32C04 = 369815 ;
HS32C05 = 1.35419e+07 ;
...
HS32C15 = 7.67908e+06 ;
...
HS32C32 = 0 ;
+content of 32 channel histogramming scaler 1 to 32 (HS32C01 to HS32C32)
HS32Depth = 24 ;
+depth of the value (usally 24 bit)
HS32F01 = 7.56e+07 ;
HS32F02 = 7.27e+07 ;
HS32F03 = 3.73e+07 ;
HS32F04 = 1.1913e+09 ;
HS32F05 = 2.11339e+07 ;
HS32F06 = 6.9124e+08 ;
...
HS32F15 = 1e-06 ;
...
HS32F31 = 1 ;
HS32F32 = 1 ;
+multiplication factor for channel 1 to 32
HS32Len = 32 ;
+number of the used channels
HS32N01 = PIN1 ;
HS32N02 = PIN2 ;
HS32N03 = PIN3 ;
HS32N04 = PIN41 ;
HS32N05 = PIN42 ;
HS32N06 = PIN5 ;
HS32N07 = PIN6 ;
HS32N08 = PIN7 ;
...
HS32N15 = time ;
...
HS32N31 = 0 ;
HS32N32 = 0 ;
+names of channel 1 to 32
HS32Z01 = 44.76 ;
HS32Z02 = 26.92 ;
...
HS32Z15 = 0 ;
...
HS32Z31 = 0 ;
HS32Z32 = 0 ;
+ zero values of the channels
HSAnode = 6 ;
+channel number in which contains the anode counts (channel 1 to 32)
HSAnodeS = 231 ;
+alternative channel that contains divided the anode counts (0 if not used)
HSI0 = 4 ;
+channel number of the I0 monitor
HSI0S = 3 ;
+alternative channel of the I0 monitor that contains divided counts (0 = unused)
HSI1 = 6 ;
+channel number of the counter for the transmitted intensity I1
HSI1S = 8 ;
+alternative counter for I1
HSTime = 15 ;
+channel which contains the counting time
HSTimeS = 23 ;
+alternative channel of the counting time
MachineInfo = Ie=141.81mA,gap46=21.80mm,taper46= 0.01mm,gap26=200.00mm,taper26
= 0.00mm ;
-single line string with machine information (info only)
OpticsInfo = optics ;
-single line string with optics information (info only)
StationInfo = id2 ;
-single line string with station information (info only)
SubTitle = 3to4 ;
-single line string with sample or experiment information (info only)
DetectorInfo =
- single line string with detector information (not used for the moment)
Example:
The time is measured in channel 15 (HSTime = 15). The value HS32Z15 is
not used for the time:
Channel 15 has the name: time
The calibrated value of channel 15 is:
(HS32C15)*HS32F15 = 7.67908e+06*1e-06 = 7.67908 seconds
Channel 1 hast the name: PIN1
The calibrated value of channel 1 is (the value HS32Z15 is not used for the
time):
(HS32C01-HS32Z01*time)*HS32F01 = (1.09726e+07 counts-44.76 counts/sec*7.67908 sec)*7.56e+07 photons/count = 8.29503e+14 photons
Remark: Normalization of the data to absolute scattering intensities Together with the absolute detector response, these values are sufficient to normalize the data automatically to absolute scattering intensities.
Spec raw data file of the gasfilled detector data
The first line of a block contains its number of lines, the last line contains -1 as a separator
The header filename is followed by hm (for histogramming memory file) The scaler data are defined in the Scalar header block. The binary data are defined in the Image header block. The current version number is 2.2. Older version are checked by a string comparison. The run number is just a number between 1 and 999, which is used to distinguish subsequently taken data. One line per item is used
General Header: 4, run number, 2.2 (version number)
and separator(-1)
Scalar header: Length of header (including separator),
scalar memory ID (2 for scalar, 3 for
histogramming memory),
first channel,
last channel,
first scalar,
last scalar,
two zero for padding,
length of scalar data in bytes,
separator (-1),
Scalar data: Length of scalar data block (including sep.),
scalar data ID (5 for scalar data)
(last scalar - first scalar) *
(last channel - first channel) lines of data
separator (-1),
Image header: length of header (including separator),
3 (presumably signifying hist. memory ???),
data length in bits (16 or 32),
x size,
y size,
number of images,
two zero for padding,
size of each image in bytes,
x offset
y offset
separator (-1)
Timing Header: length of header (including separator),
"TIMING"
DLD_HMSTARTTIME
for (i=first_frameno;i<=last_frameno;i++) {
DLD_HMDELTATIME
}
separator (-1)
Scaler Calibration Header:
length of header (including separator),
"SCALERCALIB"
DLD_SCALER_DEPTH
DLD_SCALER_I0
DLD_SCALER_I1
DLD_SCALER_ANODE
DLD_SCALER_TIME
first_scalerno
last_scalerno
for (i=first_scalerno-1;i<last_scalerno;i++) {
DLD_SCALER_NAME
DLD_SCALER_ZERO
DLD_SCALER_CALIB
}
DLD_SCALER_I0S
DLD_SCALER_I1S
DLD_SCALER_ANODES
DLD_SCALER_TIMES
separator (-1)
Experiment Header:
length of header (including separator),
"EXPERIMENT"
DLD_CENTER_1
DLD_CENTER_2
DLD_PIXSIZE_1
DLD_PIXSIZE_2
DLD_WAVELENGTH
DLD_SAMPLEDISTANCE
DLD_TITLE
DLD_SUBTITLE
DLD_DETECTORPOSITION
separator (-1)
Info Header:
length of header (including separator),
"INFO"
DLD_DETECTORTYPE
DLD_MACHINEINFO
DLD_OPTICSINFO
DLD_STATIONINFO
DLD_PROPOSALINFO
separator (-1)
EXAMPLE:
It is actually an CCD file but there is not a big difference.
(Only data size).
4
634
2.2
-1
10
2
1
32
1
1
0
0
128
-1
35
5
1219
865
0
2287
23015
19152
3182
3590
0
0
0
0
1021533
499
20000357
9766
0
0
0
0
0
3337
20479643
20479643
0
0
0
0
0
0
0
0
-1
12
3
16
1024
1024
1
0
0
2097152
0
0
-1
5
TIMING
Wed May 7 19:30:45 1997
0.006
-1
110
SCALERCALIB
25.585
5
7
13
15
1
32
PIN1
20.99
7.5617e+07
PIN2
1.49
72700000
PIN3
0
37300000
PIN41
106.15
4.29349e+08
PIN42
933.65
1.03513e+07
PIN5
1014.9
-65985562914.776428
PIN6
238.35
1.95772e+06
PIN7
150.2
126627372650.48988
I0
0
9670000
I02
0
1
I1
0
1
I2
0
1
anode
0
1
anode2
0
1
time
1e-06
time2
0
0.000205
thc1
0
0.002048
thc2
206.8
2009.7
vfc23
0
1
vfc24
0
1
thc5
0
1
vfc26
0
1
vfc27
0
1
thcsa
0
0.0005
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
10
7
14
16
-1
12
EXPERIMENT
256
256
0.000366
0.000361
9.88717e-11
0.999968
working proteins
olivier
0.160975
-1
8
INFO
X-ray Image Intensifier with FRELON CCD
Ie= 0.00mA,gap46=199.98mm,taper46= 0.02mm,gap26=20.40mm,taper26= 0.00mm
optics
id2
stopped flow
-1