NCS: Telescope Operator Guide

/!\ Information included is up-to-date, more to be included /!\

This guide contains information for the Telescope Operators on how to operate the NCS (New Control System). This guide is complemented by operator training, see OperatorTrainingComputerGroup. Please note that currently information is missing about the VME systems.

In case, information given in this guide does not agree to other information, the information given in this guide shall be used.

New

Current problems

mrt-lx1 boot problem (14-10-24 WB)

mrt-lx1 has a problem when restarting propcesses after a reboot. Check https://mrt-lx1.iram.es/mainWiki/mrt-lx1 for information

News

Consult NCS Issues

NCS issues that might be important for the Telescope Operator: see below section Related Information

network monitoring (15-02-27 WB)

Use script ./networkCheck on root@mrt-lx1 to ping several important nodes.

Maintenance Day

During Maintenance

The operator shall:

SystemTest

The operator shall:

Coordinator Software

The NCS is based on may processes (programs) running on computer systems (include VME procesors). Most of the processes are started during boot of the computer. The operator interactivly starts additional programs with the scripts goOnce and go.

How to start and stop

After a mrt-lx1 boot,

Check processes started during boot

# under mrt@mrt-lx1 do:
cd /etc/init.d
./tcs status
# in case there failed processes
cd tcs.d
./$PR status
./$PR stop
./$PR start

NcsEtc/rc_<host>.local

On mrt-lx1 and mrt-lx3, we also start several processes via script /ncsServer/mrt/ncs/etc/rc_<host>.local

Please check the script, possibly stop a process via ps -kill and start it again with cut/paste under account mrt. Needs to be improved

goOnceAfterReboot

start process antennatraced once:

# under mrt@mrt-lx1:
cdCoordinator
./goOnceAfterBoot

goOnce

Start/restart a set of helper processes and web pages:

# change default directory to coordinator subdirectory
cdCoordinator
# stop and start helper processes
./goOnce
# you can just stop them with:
./goOnceKill
# however the web pages continue to be shown
# in order to start the helper processes without starting again the web pages, use:
./goOnce x

go

Start/stop control processes

# start control processes
# change default directory to coordinator subdirectory
cdCoordinator
# stop processes (if they are running)
./goKill
# start processes (stop them before if they are running)
./go

Problems and Solutions:

NCS Control web page and command line

The NCS can be controlled from:

NCS Alarm System

. /ncsServer/mrt/ncs/etc/profile; ncsMessageSend.py warning test:warning ignore

In case the web page of alarms (https://mrt-lx1.iram.es/cgi-bin/dbQuery/ncsAlarms.py) stops working fine, run the following:

mrt-lx1:/etc/init.d/tcs.d# ./ncsMessagesToDB stop
Stopping ncsMessagesToDB: ncsMessagesToDB                            [ OK ]
mrt-lx1:/etc/init.d/tcs.d# ./ncsMessagesToDB start
Starting ncsMessagesToDB: ncsMessagesToDB                            [ OK ]

Then you can test if the alarm system is working again by running the previous test alarm command.

Monitoring

The following monitoring pages shall be checked by the operator (how often ?):

mrt-lx10: monitor for observer

mrt-lx10 is used as a monitor for the observer:

antMD

The operator shall monitor the antMD datastream via the web page https://mrtweb.iram.es/plotAmd/ . Related issue: http://issues:8000/ncs/ticket/588 .

Observation Queues

See Observation Queues.

Online Data Processing (ODP)

ODP can be monitored by the operator, details are given in https://mrt-lx1.iram.es/mainWiki/NcsOdpOverview .

Operation Scenarios

Reboot a computer

In case a reboot of a computer is needed, check:

How to do technical observations

Calibration without sky

Ask the observer to specify a scan with cal /sky no. Note that the source given has to be in the accesprt az/el range.

Observe without moving the antenna

See https://mrt-lx1.iram.es/mainWiki/TelescopeControlWithoutMovingAntenna

NCS Fallback

There is currently no fallback version of the NCS, see also NcsVersions

Restart a VNC server

The NCS uses several wiki:VNC for system monitoring and remote observing on mrt-lx1. These VNCs are started during boot, see /etc/init.d/vncServers. A VNC can be restarted from account operator@mrt-lx1 (better root@mrt-lx1):

sudo vncServerRestart <display number>
# ex: sudo vncServerRestart -h ; sudo vncServerRestart 30

(issue ncs#936)

On some VNCs (nr:26,27,31), a script goMrtMonitor<nr> shall be executed in a xterm after restarting that VNC.

If we want to reset the vnc servers needed for remote (28, 30 and 32), e.g when the system is overloaded because the astronomer forgot closing the sessions, we can reset vnc by doing (in root mrt-lx1):

# restartRemoteVncSessions

Change Pointing Model

https://mrt-lx1.iram.es/mainWiki/OperationNotesOperatorsHowToChangePointingModel

Observing Projects

Fallback project

For each scheduling period (starting in May), a fallback project 000-yy is created that shall be used if the scheduled project does not exist. ncsOperator@iram.es shall be informed. issue

List of projects per two-weeks scheduling

https://mrt-lx1.iram.es/ncs/Projects/

Manual checks

PR=000-13
ypcat passwd | grep $PR

If no line is printed, the account does not exist. Check also if the home directory of the project is on /mrt-lx3/vis. Example:

ypcat passwd | grep $PR
#                                             : home directory    : 
000-13:x:4788:4788:brunswig@iram.es,,,,pw=2.0;:/mrt-lx3/vis/000-13:/bin/bash

Change Current Project

Project Passwords

For projects starting from scheduling year 13 (nnn-yy, yy>=13) on, we have introduced a new password policy:

alias pp=". /ncsServer/mrt/ncs/etc/profile; ncsProjectPassword.py"
# pp <projectId>:
pp 000-13
# result:
# 000-13durcal13
#              000-13: 30m/beqi/000

. /ncsServer/mrt/ncs/etc/profile
find /mrt-lx3/vis/*  -maxdepth 0 -name "*-13" \
  -exec ncsProjectPassword.py {} \; |
  grep -v "^Please" 
# or check help
ncsProjectPassword.py -h

Observation Data

to do include link

Receiver Control

HERA

Operation

Check: Read or Set Frequency for HERA2

The frequency of HERA2 is set via an APSIN synthesizer. Check the device:

Note:

Other Items

Remote Observing

See RemoteObserving

Skype

Skype can be used for communication between the operator, the (remote-)observer, and the (Granada-)AOD. See NcsUG#skype for details.

Allow remote access to a desktop (X11VNC)

We have installed a program x11vnc on several computers that allow to control (or monitor) the computer desktop, e.g., remote access for tuning by a receiver engineer from Granada. For that, the operator shall start program x11vnc:

When somebody connects, you have to accept the connection then.

x11vnc can also be used to give the "Remote AOD" access to a visitor desktop.

Related issue: http://issues:8000/system/ticket/1053

Spare Material

See http://issues:8000/system/wiki/SpareMaterial.


Page Info

NcsTOG (last edited 2018-08-17 08:17:21 by mellado)