NcsDataDirectoryOrganization

This page is moderated by WB. Please note:

Terminology

Ncs Data

If not stated otherwise, we understand that NCS data means observation data. NCS produces of course more data, but speaking always of NCS observation data ... is a bit clumpsy.

Archive, Backup

Please note: While working on this document, I (wb) realized that we use the term archive and backup not correctly. Archive should mean that something is stored permanently and backup should mean that we create a copy. Some names of scripts, etc. ... are incorrect and therefore confusing. I hope to clearify this while working on this document.

Overview

This wiki page describes the directory structure used by the NCS. Details of the file structure are given in specifc pages. The directories are distributed over several computers, but are normally accessible via NFS from all user computers as well as from specific processors used for telescope control.

We distinguish:

Architecture

Data is produced in the NCS in:

Data Archiving Overview

The data of a scan is kept in the NcsScanDirectory, e.g. /ncsServer/mrt/ncs/data/20080324/scans/1/ and at the end of the scan a copy is put into the NcsProject, e.g. /vis/100-07/observationData/20080324/1.

The data produced for an observation is archived three times:

Datastream directories

Several subsystems of the NCS can produce so-called [NcsDatastreams datastreams]. The datastreams are written into directories: /mrt-lx1/ncsServer/mrt/ncs/data/datastreams/ Currently the following datastreams are produced:

mrt@mrt-lx1:datastreams>pwd
/mrt-lx1/ncsServer/mrt/ncs/data/datastreams
mrt@mrt-lx1:datastreams>ls -1F
100khz/
1MHz/
1mhz -> 1MHz/
4mhz/
abba/
antmd/
continuum/
fts/
FTS -> fts/
hera/
secondary/
vespa/
Vespa -> vespa/
wilma/
Wilma -> wilma/
mrt@mrt-lx1:datastreams> 

Note: The above directory listing is not literal

Example of a datastream directory:

mrt@mrt-lx1:datastreams>ls -l 1MHz/ | tail -3
-rw-r--r--  1 root root    5760 2007-01-10 16:38 iram30m-1mhz-20070110t163848.fits
-rw-r--r--  1 root root    5760 2007-01-10 16:39 iram30m-1mhz-20070110t163949.fits
-rw-r--r--  1 root root    5760 2007-01-10 16:40 iram30m-1mhz-20070110t164049.fits

Most datastream files are in FITS format. The filename is of format iram30m-<datastream>-<dataOfCreation>.fits. Files of other format might be there for test purposes.

Moving datastream files to day directory

du /ncsServer/mrt/ncs/data/20070108/datastreams/
11540   /ncsServer/mrt/ncs/data/20070108/datastreams/100khz
140832  /ncsServer/mrt/ncs/data/20070108/datastreams/4mhz
552916  /ncsServer/mrt/ncs/data/20070108/datastreams/antmd
88428   /ncsServer/mrt/ncs/data/20070108/datastreams/continuum
23112   /ncsServer/mrt/ncs/data/20070108/datastreams/secondary
149644  /ncsServer/mrt/ncs/data/20070108/datastreams/Vespa
185224  /ncsServer/mrt/ncs/data/20070108/datastreams/1mhz
4496    /ncsServer/mrt/ncs/data/20070108/datastreams/hera
1156196 /ncsServer/mrt/ncs/data/20070108/datastreams/

Archiving day directories

Moving day directories to tape

NCS Scan directories

The NCS creates a directory and several subdirectories per scan. The root directory is called the scan directory. All directories of one day are put into directory: /mrt-lx1/ncsServer/mrt/ncs/data/<date>/scans. Example:

mrt@mrt-lx1:scans>pwd
/mrt-lx1/ncsServer/mrt/ncs/data/20070110/scans
mrt@mrt-lx1:scans>ls -trl
total 1200
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 00:05 1
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 00:08 2
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 00:09 3
...
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 16:15 298
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 16:21 300
drwxr-xr-x  4 mrt mrt 4096 2007-01-10 16:21 299
mrt@mrt-lx1:scans> 

An example listing of a scan directory is:

mrt@mrt-lx1:scans>ls -ltr 299/*
-rw-r--r--  1 mrt mrt    18679 2007-01-10 16:15 299/iram30m-scan-20070110s299.xml
-rw-r--r--  2 mrt mrt  6082560 2007-01-10 16:21 299/iram30m-4mhz-20070110s299-imb.fits
-rw-r--r--  2 mrt mrt 16208640 2007-01-10 16:21 299/iram30m-wilma-20070110s299-imb.fits

299/debug:
total 4
-rw-r--r--  1 mrt mrt 2006 2007-01-10 16:21 antmd.commands

299/log:
total 896
-rw-r--r--  1 mrt mrt  65537 2007-01-10 16:15 iram30m-statebefore-20070110s299.xml
-rw-r--r--  1 mrt mrt  72842 2007-01-10 16:15 iram30m-statebefore-20070110s299.html
-rw-r--r--  1 mrt mrt 180290 2007-01-10 16:15 iram30m-statebefore-20070110s299.dbm
-rw-r--r--  1 mrt mrt  46795 2007-01-10 16:15 iram30m-scan-20070110s299.pickle
-rw-r--r--  1 mrt mrt  65548 2007-01-10 16:21 iram30m-stateafter-20070110s299.xml
-rw-r--r--  1 mrt mrt  72853 2007-01-10 16:21 iram30m-stateafter-20070110s299.html
-rw-r--r--  1 mrt mrt 180359 2007-01-10 16:21 iram30m-stateafter-20070110s299.dbm
-rw-r--r--  1 mrt mrt    523 2007-01-10 16:21 scanoverview.txt
-rw-r--r--  1 mrt mrt   6393 2007-01-10 16:21 iram30m-sync-20070110s299.xml
-rw-r--r--  1 mrt mrt   7153 2007-01-10 16:21 iram30m-sync-20070110s299.html
-rw-r--r--  1 mrt mrt  17126 2007-01-10 16:21 iram30m-sync-20070110s299.dbm
-rw-r--r--  1 mrt mrt    683 2007-01-10 16:21 scanoverview.html
-rw-r--r--  1 mrt mrt   9448 2007-01-10 16:21 logMessages.xml
-rw-r--r--  1 mrt mrt   4603 2007-01-10 16:21 makeimbfits4mhz.err
-rw-r--r--  1 mrt mrt  51057 2007-01-10 16:21 makeimbfits4mhz.log
-rw-r--r--  1 mrt mrt   6259 2007-01-10 16:21 makeimbfitswilma.err
-rw-r--r--  1 mrt mrt  50214 2007-01-10 16:21 makeimbfitswilma.log
mrt@mrt-lx1:scans>

See [NcsScanDirectories] for details on the files in a scan directory.

Archiving and moving scan directories

The scan directories are archived on /mrt-lx1/ltmpDataStream and moved to a backup tape at the same time as the datastream files. See details above.

However, the scan direcories are duplicated to the project directory at the end of a scan. As the project directories are moved to a CD or DVD and the observer gets another copy, we finally end up with at least three copies of the scan directories.

IMBFits directory

At the end of scans, and possibly subscans, [FileOrganizationIMBFits IMBFits files] are created. The "imbfits" files are stored in the scan directory (see listing above) and, in addition using a unix hard link, in directory /mrt-lx1/ncsServer/mrt/ncs/data/imbfits/het and .../bol.

Note: "imbfits" files can also be created offline if the other files in a scan directory are available and datastream files for the subscan periods.

Note: We also store all imbfits files of a project in /mrt-lx3/vis/<project>/observationData/imbfits/

Archiving and backup of IMBFits files

IMBFits files are archived as the other project files on the observer and IRAM DVD.

Note: we right now also keep an archive of imbfits files per month.

Project directories

Each project has its own home directory /mrt-lx3/vis/<project/.

Example:

ls /mrt-lx3/vis/204-06
3mm_10.30m  Desktop1  gopako  mira  mydata           observationData  PaKo  tmp
Desktop     FDveSv    goPako  Mira  observationdata  pako             RCS
mrt@mrt-lx1:het>

During the creation of a project ([CreateProjects]), in the project home a directory observationData is created. The owner of this directory is "mrt" and access protection is setup such that only "mrt" can read/write.

Please note: read access to project files should only be possible from account mrt and the project account. However, currently also world can read. We have to investigate this.

Example:

ls -ltr /mrt-lx3/vis/204-06/observationData
total 136
drwxr-xr-x  3 mrt 204-06   4096 2007-01-06 01:06 mira
drwxr-xr-x  3 mrt mrt      4096 2007-01-06 01:06 20070106
...
drwxr-xr-x  3 mrt mrt      4096 2007-01-11 00:04 20070111
drwxr-xr-x  2 mrt mrt    106496 2007-01-11 07:41 imbfits
mrt@mrt-lx1:het>

In the directory

Observer backup

Observers can get a copy of the project directories on CD, DVD. They also might just transfer the data themself via FTP. We also encorage Observers to bring their own USB-disk or memory-stick.

Archiving project directories

After the monthly backup, projects that have finished 2 weeks ago are removed from disk:

Please note:

Results of online dataprocessing

We store in a project directory also results of online dataprocessing. The observer can read those files but not writem them. We store:

Example:

mrt@mrt-lx1:observationData>pwd
/vis/wbtest/observationData
mrt@mrt-lx1:observationData>ls
20010203  20060510  20060531  20060805  20060926  20070103
20051122  20060529  20060620  20060808  20061107  imbfits
20060509  20060530  20060804  20060810  20061214  mira
mrt@mrt-lx1:observationData>ls -l mira
total 1636
drwxr-xr-x  3 mrt wbtest   20480 2007-01-03 16:49 plots
-rw-r--r--  1 mrt mrt    1649664 2007-01-03 16:49 spectraOdp.30m

Comments:

Pool Projects

Here we shall describe spefic organization of pool projects.

Data recovery procedures

Recreate IMBFits

Here we shall explain how to create IMBFits files offline.

Recalibrate observations

Here we shall explain how to recalibrate observation.

Incidents

Here we shall list observed problems.

Evaluation

Performance can be documented here.

Requests

Online Dataprocessing

spectraOdp.30m

The observer shall be able to rename spectraOdp.30m files.


CategoryNcs

NcsDataDirectoryOrganization (last edited 2011-09-27 09:59:07 by WalterBrunswig)