Online Data Processing

Overview

Note: The odp_fts queue is currently linked to the odp_spectra queue queue.

OnlineDataProcessing (short: ODP) is executed at the end of subscans and scans. (ODP after subscan has been done only for pointings, but is also currently disabled.)

ODP does a first processing of the data of a scan:

Check the page http://mrt-lx1.iram.es/mainWiki/OperationNotesObserversMonitoring on how to monitor the results of ODP.

Links:

The rest of this page describes details that are of less importance for the observer.

Operation

This section shall help the operator in case of problems with ODP.

Automatic Start

ODP start: the processes needed for ODP are started on mrt-lx3 during boot by executing: /ncsServer/mrt/ncs/etc/rc_mrt-lx3.local

Monitor ODP processes

You can use command top -c -u mrt -d 5 under mrt@mrt-lx3 to monitor ODP processes.

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                          
 2606 mrt1    20   0 87568 1944  988 S    0  0.0   0:00.00 sshd: mrt@pts/12                                                                                 
 2607 mrt1    20   0 22960 2172 1552 S    0  0.0   0:00.01 -bash                                                                                            
31784 mrt1    20   0 59364  15m 3472 S    0  0.0   0:02.27 python ./scanInfo.py -m odp -q -v 1                                                              
31785 mrt1    20   0 32336 5776 2484 S    0  0.0   0:02.97 python /ncsServer/mrt/ncs/packages/coordinator /spooler.py -v 2 -s odp/continuum -d               
31786 mrt1    20   0 32336 5768 2484 S    0  0.0   0:03.08 python /ncsServer/mrt/ncs/packages/coordinator/spooler.py -v 2 -s odp/spectra -d                 
# there is currently no extra queue for FTS:
# 31787 mrt1    20   0 32336 5772 2484 S    0  0.0   0:02.87 python /ncsServer/mrt/ncs/packages/coordinator
# /spooler.py -v 2 -w 1 -s odp/fts -d                

You will also see that processes execute MIRA at the end of scans.

Check ODP Processes

Under mrt@mrt-lx3 (!) do:

mrt@mrt-lx3:.log>ps uxa | grep scanInfo
mrt       1352  0.0  0.0   8224   784 pts/8    S+   09:11   0:00 grep scanInfo
mrt      21657  0.0  0.0  90180 14324 pts/9    S    03:07   0:02 python ./scanInfo.py -m odp -q -v 1
mrt@mrt-lx3:.log>ps uxa | grep spooler
mrt       1350  0.0  0.0   8224   784 pts/8    S+   09:10   0:00 grep spooler
mrt      32071  0.0  0.0  35044  4400 pts/8    SN   06:30   0:01 python ./new/spooler.py -v 2 -s odp/continuum -d
mrt      32072  0.0  0.0  35044  4400 pts/8    S    06:30   0:01 python ./new/spooler.py -v 2 -s odp/spectra -d
# no specific spooler process for FTS:
# mrt      32073  0.0  0.0  35048  4400 pts/8    S    06:30   0:01 python ./new/spooler.py -v 2 -s odp/fts -d

    mrt@mrt-lx3:log>ls -ltr odp_*
-rw-r--r-- 1 mrt mrt       0 2011-01-18 15:02 odp_continuum.err
-rw-r--r-- 1 mrt mrt     106 2011-05-18 10:10 odp_spectra.err
-rw-r--r-- 1 mrt mrt   52399 2011-05-18 10:19 odp_fts.log
-rw-r--r-- 1 mrt mrt     748 2011-05-18 10:19 odp_fts.err
-rw-r--r-- 1 mrt mrt 2347228 2011-06-01 10:06 odp_continuum.log
-rw-r--r-- 1 mrt mrt 8757379 2011-06-01 16:03 odp_spectra.log
mrt@mrt-lx3:log>ls -ltr mrt-lx3*
-rw-r--r-- 1 mrt mrt     1242 2011-05-18 14:32 mrt-lx3_scanInfo.err
-rw-r--r-- 1 mrt mrt 32106651 2011-06-01 16:03 mrt-lx3_scanInfo.log

Restart ODP

If ODP does not work, please check its processes. Under mrt@mrt-lx3 (never in mrt-lx1) you might stop all processes with command kill -9 <pid> (pid: second column in the information shown using commands "ps uxa | grep scanInfo" and "ps uxa | grep spooler" , see above) and then enter

ps uxa | grep scanInfo
ps uxa | grep spooler
echo "check if ODP processes are still running - ignore processes runnging grep"
echo "sleeping 10s"
sleep 10
cd /ncsServer/mrt/ncs/etc
bash -v rc_ODP.sh
cd -

After that, do goKill+goOnceKill+goOnce+go.

ODP Queues

The new version of ODP, uses queues to process scans. The queues are directories: /ncsServer/mrt/ncs/var/spool/odp/*/ and the "job" files end with ".job". If there pending jobs, you might delete those jobs or (better) move them to a subdirectory:

cd  /ncsServer/mrt/ncs/var/spool/odp/spectra/
mv -vb *.job tmp/

A restart of the processes, as described above, is also recommended in this case.

Unfinished MIRA processes

ODP start subprocesses running MIRA and waits for the processes to finish. Sometimes, the processes do not finish (I suspect MIRA prompts for some interactive input though we run it with "interactive no"). In case, a MIRA process does not finish and the corresponding ODP queue is blocked, kill the process and check afterwards the ODP queue processes the next job.

Architecture

Overview

Implementation

Source code documentation can be found here

Example of ".job" file

#<job name='2011-10-18.70' file='/ncsServer/mrt/ncs/var/spool/odp/bbc/job-2011-10-18T15.34.54.782-2011-10-18.70.job' submittedAt='2011-10-18T15:34:54.782'>
#<script>
. /etc/profile.local; . /etc/IRAM/gag_ODP; mira -niw @onlineDataProcessing.mira 001-11 70 20111018 0 0 BBC #scanId=2011-10-18.70 queue=bbc backend=BBC
scanInfoOdpDone.py 001-11 70 20111018 calibrate BBC
#</script>
#</job>

Observations with the heterodyne receivers

ODP is controlled by process scanInfoODP when it receives the a message saying 'scan done':

Observation with Bolometers

Currently, we do not offer Bolometers.

Messages


This is page https://mrt-lx1.iram.es/mainWiki/OnlineDataProcessing, moderated by WB

CategoryOdp CategoryNcs CategoryOperator

OnlineDataProcessing (last edited 2013-04-24 21:44:58 by FredericDamour)