Project Management: Overview

This page describes the observing project management of the MRT.

ProjectManagement (done by ncsOperation@iram.es) consists of:

NOTE

User Info

See also below Project FAQs

Information about Projects

Reloaded Projects

Projects can be reloaded from the archive. Note: normally "*imb.fits*" and "*.30m" files are not copied back to /vis.

Operation

This is not yet organized and would have to be put into the NcsOG:

Maintenance

On the maintenance day, do:

See how-to below. Log in Logbook

Logs

Project Maintenance

Standard Projects

Fallback projects

We create fallback projects that can be used in case the account of a project is not working correctly. The current fallback project is 000-12. In the fallback project, we will synchronize files in subdirectory pako during maintenance.

Create Projects

Create Projects From Scheduler Web Pages

The following commands will:

(see example below)

# under archive@mrt-lx3 do:
useNCS
cd ~/archive/wb
# check ReadMe.txt for more information !
./scheduler.sh +0
# or ./scheduler.sh +2 for weeks to come
# scheduler.sh creates a file work/actions.sh, check it with:
grep -v "^#" work/actions.sh
# note: the createUser for Tapas is always created,  account might already exist
# the next command can also be done from root@mrt-lx3 (without passwords asked)
cd work
./doActions
cd -
# the passwords to use will be printed, use cut&paste when asked for them
# all other questions: just <Enter>

After creating projects, do:

ncsProjectPassword.py -l | sort | a2ps -n 2 -Pmrt-pr1

cd /vis
org/checkQueues.py

(last rev: 13-09-17 WB)

Create Projects Manually

In order to create a project manually, do under archive@mrt-lx3:~/archive/wb :

PR=085-10
grep -i $PR projects.lis
# if the  project is not in projects.lis: add the project to file projects.lis:
  ci -l projects.lis
  nano projects.lis
#
./planner.py $PR | tee work/actions.sh
# the next command can also be done from root@mrt-lx3 (no passwords asked)
cd work
./doActions

Create Accounts for Pools

See http://issues.iram.es:8000/ncs/ticket/657

Inform operator/AOD on passwords

After creating new projects: print passwords for AOD and operator. See https://mrt-lx1.iram.es/mainWiki/NcsOG#Project_Passwords

Synchronize mrt-lx3:/vis and Project Archive

On maintenance day do normally

Review There is a cron job doing this!

Compress dbm files

Under root@mrt-lx3:

DIR=/vis
cd $DIR
logger -p local6.notice "id=ncsOperation.dayDirs.stateDbm.compress state=started"
ionice -c 3 find ./ -mindepth 7 -maxdepth 7 -name "*state*.dbm" -size +300k -mmin +60 -print -exec /ncsServer/mrt/ncs/tools/dbmCopy.py {} \; | tee -a org/log/dbmCopy.log
# and remove old versions
ionice -c 3 find ./ -mindepth 7 -maxdepth 7 -name "*state*.dbm~*" -exec rm -v {} \;
logger -p local6.notice "id=ncsOperation.dayDirs.stateDbm.compress state=done"

Compress day directories

Under root@mrt-lx3

# all projects
DIR=/vis
cd $DIR
logger -p local6.notice "id=ncsOperation.dayDirs.compress project=all state=started"
find ./ -maxdepth 1 -mindepth 1 -type d -print -exec org/dayDirsTar.py -v 2 -a {} \; 2>>org/log/dayDirsTar0.err | tee org/log/dayDirsTar0.log
cat  org/log/dayDirsTar0.log >>org/log/dayDirsTar.log
cat  org/log/dayDirsTar0.err >>org/log/dayDirsTar.err
echo "+++ done, error file:"
cat org/log/dayDirsTar0.err
logger -p local6.notice "id=ncsOperation.dayDirs.compress project=all state=done"

# or just one project
PR=xxx
logger -p local6.notice "id=ncsOperation.dayDirs.compress project=$PR state=started"
org/dayDirsTar.py -v 2 $PR >>org/log/dayDirsTar.log 2>>org/log/dayDirsTar.err
logger -p local6.notice "id=ncsOperation.dayDirs.compress project=$PR state=done"

Synchronize one project with project archive

Under root@mrt-lx3:

BW=25000
PR=xxx
YEAR=yy
cd /vis/org
DATE=`date +%Y-%m-%dT%H-%M`
LOG=/vis/org/log/$PR-$YEAR.log_$DATE
./pm_syncProject $PR $YEAR $BW >$LOG &
xterm -e tail -f $LOG &

See /vis/org/ReadMe.txt.

PR=xxx
YEAR=yy
cd /vis/org
./pm_checkProject $PR $YEAR

Scenario: Sync projects to archive and remove from /vis

several projects

# under root@mrt-lx3:

PAT="0*-14" # example
cd /vis
python org/syncAndRemove.py $PAT
# just lists commands - execute them with cut&paste

just one project

Preparation:

Remove only projects that had not been observed since last (incremental) backup of its corresponding project archive.

Under root@mrt-lx3:

PR=xxx-
YEAR=yy
ypcat passwd | grep $PR$YEAR

echo "check if project is scheduled soon"
grep $PR$YEAR /ncsServer/mrt/ncs/Projects/s*.lis
# if project is scheduled in the next weeks, leave it on mrt-lx3.

echo "ReadMeIRAM.txt"
cat /vis/$PR$YEAR/ReadMeIRAM.txt
sleep 10

echo "starting removal"
cd /vis/org
DATE=`date +%Y-%m-%dT%H-%M`;LOG=/vis/org/log/$PR$YEAR.log_$DATE
./pm_removeProject $PR $YEAR | tee $LOG 

Synchronize all projects with project Archive

to be reviewed

DIR=/vis
cd $DIR
DATE=`date +%Y-%m-%dT%H-%M`
logger -p local6.notice "id=ncsOperation.project.sync project=all state=started"
find ./ -maxdepth 1 -mindepth 1 -type d -print \
  -exec org/rsyncProject.py {} \; | tee -a org/log/sync.log
# gzip -c org/log/sync.log >org/log/sync.log_$DATE.gz; rm -v org.log/sync.log
# gzip -c org/log/sync.err >org/log/sync.err_$DATE.gz; rm -v org.log/sync.err
# tar  rvf org/log/sync.tar --remove-files org/log/sync.gz
echo "maintenance of org/log not yet programmed"
logger -p local6.notice "id=ncsOperation.project.sync project=all state=done"
cd -

Adapt passwd file

For one Project

Please note: Project directories in the archive are in /prArch/prArch<year>.

#
# change home directory for moved projects
# under root@mrt-lx1 /etc
PR=xxx-xx
# if PR=xxx-10, UNIT=0 else set to UNIT=1
UNIT=0
cd /etc
ci -l passwd
replace -v /mrt-lx1/vis/$PR /prArch/prArch$UNIT/closed/$PR -- passwd
logger -p local6.notice "id=ncsOperation.project.chHome project=$PR state=prArch"
make -C /var/yp

General Check of /etc/passwd

You can also do a general check of the password files, see mrt-lx1:/etc/tools/showPasswd.py for details. The program will print the "replace" commands needed.

Reload projects from archive

For project that have been observed and are scheduled again:

Under root@mrt-lx3 do:

cd /vis/org/
YEAR=yy
PR=xxx
bash -v pm_reloadProject $PR $YEAR

Note:

# Example:
Queues=/ncsServer/mrt/ncs/observationQueues
find $Queues -name "$PR-$YEAR" -type d
# if it does not:
#   if it is in old/:
cd $Queues
mv -vb old/$PR ./
#   else: create it
mkdir -vp $Queues/$PR/.done
find  /ncsServer/mrt/ncs/observationQueues/$PR -type d -exec chown -v $PR:mrt {} \;
find  /ncsServer/mrt/ncs/observationQueues/$PR -type d -exec chmod -v 775 \;

Rename Projects

How to change project names:

(see also issue)

Scenario: Send Project Data

Scenario VisitorBackup

This section needs a review and should not be used.

Give remote access to a project in archive

under root@mrt-lx1 do

PR=wbtest
# get new password
/ncsServer/mrt/ncs/packages/coordinator/sandbox/projectPassword.py $U
# mod password
yppasswd -p $U
# set user info
chfn -o pw=v2.1 $U
# inform observer
grep $U /etc/passwd

Send e-mail:

Dear observer,
in order to access your project from Internet, your project password 
has been changed. It is now: 30m/xxxx/ppp with xxxx being a four letter 
word you will receive in a separate e-mail and ppp being the first three 
letters of your projectID, ex: 30m/abcd/001. 
You can log into your project via slogin: slogin <projectID>@enter.iram.es .
Best wishes from the NCS Operation team.
ncsOperation@iram.es

Small projects burned by the operator

For small projects, the CD/DVD is created by the operator. For larger one, the observer has to request if via the VisitorBackupForm.

Burning DVDs in Granada

Obsolete: we now rather use USB memory and disks - needs review

In case the DVD was not being produced by the operator, it can be made in Granada. For this it is needed to synchronize mrt-lx1:/vis and the project archive (described above and normally done by ncsOperation during maintenance). Steps:

until the project fits on the DVD

Project Archive

We are building up a project archive that shall contain the files of projects observed at the 30mRT.

The archive is located (projectArchive == mrt-lx2 currently) :

The project archive is regularly synchronized with mrt-lx:/vis

Project FAQs

I want to have access to an old project

We keep for some time the files of projects that have been observed in our ProjectArchive. Check if the project account still exists:

PR=xxx-yy
ypcat passwd | grep $PR

You can also see there where the project files are.

If the project is in the archive, connect to projectArchive.iram.es. From there you also have access to the files of current projects via /projects/<projectId> .

Project Classes

Recurrent Projects

Recurrent projects are kept on mrt-lx3:/vis for observation. However, old data (scans, imbfits) might only be in the archive. Details will be given in file ReadMe.txt of the project home directory.

Technical Projects

Same rules a recurrent projects

Pool Projects

Pool projects of current pool stay for now on mrt-lx3:/vis. We will check with the pool manager when we can remove projects from mrt-lx3:/vis . The files of those projects would then be accessible in the archive.

Examples

Create Projects

archive@mrt-lx3:~/archive/wb$ ./scheduler.py -o +2
options: <Values at 0xdb7cb0: {'offline': True, 'verbose': 0}>, args: ['+2']
Projects: ['003', '003-10', '004', '004-10']

++++++++++ 003-10 ++++++++++
account 003-10 exists: True
directory /gra-lx4/prArch0/closed/003-10 exists: True
directory /mrt-lx3/visMirror/003-10 exists: False
directory /mrt-lx3/vis/003-10 exists: True

/gra-lx4/prArch0/closed/003-10 True
/mrt-lx3/vis/003-10  True
/mrt-lx3/visMirror/003-10 False
PI                   Michal Drahus
account              exists
California at Los Angeles, Los Angeles, CA 90095 (USA)
email                mdrahus(at)ucla.edu
home                 exists
project              003-10


++++++++++ 004-10 ++++++++++
account 004-10 exists: True
directory /gra-lx4/prArch0/closed/004-10 exists: False
directory /mrt-lx3/visMirror/004-10 exists: False
directory /mrt-lx3/vis/004-10 exists: True

/gra-lx4/prArch0/closed/004-10 False
/mrt-lx3/vis/004-10  True
/mrt-lx3/visMirror/004-10 False
PI                   Nicolas Biver
account              exists
affiliation          LESIA, Observatoire de Paris, 92190 Meudon (France)
email                nicolas.biver(at)obspm.fr
home                 exists
project              004-10


# actions for project 003-10
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 003-10 mdrahus@ucla.edu # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 003-10 # action root@mrt-lx1
#
# create tapas account # action archive@mrt-lx3
#<  003-10 = Drahus M. = mdrahus@ucla.edu = -1 = -1 = -1 = to be scheduled 
##./createUser.py '003y10' '003-10' 'Drahus M.' 'mdrahus@ucla.edu' '003-10'
cd /local/users/archive/archive/wb; ./createUser.py '003y10' '003-10' 'Drahus M.' 'mdrahus@ucla.edu' '003-10' # acti
on archive@mrt-lx3
#
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 003-10 # action root@mrt-lx3 
# action done

# actions for project 004-10
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 004-10 nicolas.biver@obspm.fr # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 004-10 # action root@mrt-lx1
#
# create tapas account # action archive@mrt-lx3
#<  004-10 = Biver N. = nicolas.biver@obspm.fr = -1 = -1 = -1 = to be scheduled 
##./createUser.py '004y10' '004-10' 'Biver N.' 'nicolas.biver@obspm.fr' '004-10'
cd /local/users/archive/archive/wb; ./createUser.py '004y10' '004-10' 'Biver N.' 'nicolas.biver@obspm.fr' '004-10' #
 action archive@mrt-lx3
#
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 004-10 # action root@mrt-lx3 
# action done

archive@mrt-lx3:~/archive/wb$ cd work
archive@mrt-lx3:~/archive/wb/work$ bash -v doActions
Account=root
Where=mrt-lx1
grep $Account@$Where actions.sh >actions_$Account@$Where
cat actions_$Account@$Where
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 003-10 mdrahus@ucla.edu # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 003-10 # action root@mrt-lx1
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 004-10 nicolas.biver@obspm.fr # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 004-10 # action root@mrt-lx1
ssh $Account@$Where cd /local/users/archive/archive/wb/work\; bash -v actions_$Account@$Where
root@mrt-lx1's password: 
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 003-10 mdrahus@ucla.edu # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 003-10 # action root@mrt-lx1
# create account root@mrt-lx1 # action root@mrt-lx1
# /usr/local/ncsPam/addAccount.sh 004-10 nicolas.biver@obspm.fr # action root@mrt-lx1
# /usr/local/ncsPam/addDirectory.sh 004-10 # action root@mrt-lx1

Account=root
Where=mrt-lx3
grep $Account@$Where actions.sh >actions_$Account@$Where
cat actions_$Account@$Where
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 003-10 # action root@mrt-lx3 
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 004-10 # action root@mrt-lx3 
ssh $Account@$Where cd /local/users/archive/archive/wb/work\; bash -v actions_$Account@$Where
root@mrt-lx3's password: 
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 003-10 # action root@mrt-lx3 
# under root@mrt-lx3 do: # action mrt-lx3
# /vis/org/rsyncArchToVis.sh 004-10 # action root@mrt-lx3 

Account=archive
Where=mrt-lx3
grep $Account@$Where actions.sh >actions_$Account@$Where
cat actions_$Account@$Where
# create tapas account # action archive@mrt-lx3
cd /local/users/archive/archive/wb; ./createUser.py '003y10' '003-10' 'Drahus M.' 'mdrahus@ucla.edu' '003-10' # acti
on archive@mrt-lx3
# create tapas account # action archive@mrt-lx3
cd /local/users/archive/archive/wb; ./createUser.py '004y10' '004-10' 'Biver N.' 'nicolas.biver@obspm.fr' '004-10' #
 action archive@mrt-lx3
ssh $Account@$Where cd /local/users/archive/archive/wb/work\; bash -v actions_$Account@$Where
archive@mrt-lx3's password: 
# create tapas account # action archive@mrt-lx3
cd /local/users/archive/archive/wb; ./createUser.py '003y10' '003-10' 'Drahus M.' 'mdrahus@ucla.edu' '003-10' # acti
on archive@mrt-lx3
452414
user 003y10 created
1st message sent, sleep 10s
(220, 'mrt-lx3.iram.es ESMTP Exim 4.69 Tue, 19 Oct 2010 08:31:29 +0000')
{}
2nd message sent
# create tapas account # action archive@mrt-lx3
cd /local/users/archive/archive/wb; ./createUser.py '004y10' '004-10' 'Biver N.' 'nicolas.biver@obspm.fr' '004-10' #
 action archive@mrt-lx3
873996
user 004y10 created
1st message sent, sleep 10s
(220, 'mrt-lx3.iram.es ESMTP Exim 4.69 Tue, 19 Oct 2010 08:31:39 +0000')
{}
2nd message sent

archive@mrt-lx3:~/archive/wb/work$


Page info

ProjectManagement (last edited 2015-10-07 16:12:09 by WalterBrunswig)