Discussion: New Organization
With the experience of NCS running for about six months we plan to reorganize disk space. This chapter discusses what to do in the future.
We have two mayor requests:
- Install a computer that can be backup for mrt-lx1 and mrt-lx3 (WB, AS)
- Keep pool data permanently on disk (SL)
Any new organization has to take the following known problems into account:
- Disk total backup currently might take more than maintenance time and put heavy load on the computers.
Project directories (/vis) shall be kept physically not on mrt-lx1 but on a second file server. DONE, /vis is now on mrt-lx3.
- The partition size should be increased from 250 GB to 500GB (to fit on one LTO tape).
- Would that be sufficient for the next two years ?
- Is it too much to do backups during maintenance ?
A fallback project shall be defined (name ?) that resides on mrt-lx1 only. This project shall allow obervations just with mrt-lx1 being available (not needing mrt-lx2, -lx3). This also might require that we keep /ncsdefaukt on mrt-lx1 .
Some projects can stay permanently on disk (pointing, local users). However, we should be more restrictive:
- permanent projects have to be requested, maybe using names without '-'
- observers can just request a project to stay for up to 3 months
Backup of these projects will be made as part of the monthly/yearly backup.
Backup for Visitors
Visitors can request a copy of project files on CD, DVD, Memory stick, or usb disk. Exceptionally it would also be possible to put them on LTO. DAT will not be supported in the future. Several computers will allow to mount a memory stick or usb-disk:
- a linux system in the visitor office
Backup for IRAM
For projects that have finished:
A copy for IRAM of the project files (/vis/<project>/...) will be put on CD or DVD if it fits. Should we rather save them on LTO ?
- The project will be removed from disk after the next monthly backup if the PI does not request to keep the files on disk.
A copy for the IRAM of the project files (/vis/<project>/...) will be put on CD or DVD if it fits. Should we rather save them on LTO ?
- Should we keep '.30m' files or the imbfits files on disk for some time ?
Keeping finished projects
Project files can stay on disk upon request by the PI. Initially the files then will stay 3 more months. The request shall indicate an email address for contacts. Extensions can be requested by the PI.
- The IRAM backup will be made as for projects that will be removed and it is recommended that the visitor takes a copy of the files nevertheless.
- The subdirectory observationData/... that cannot be changed by the visitor will be moved to partition /ltmpObservationData .
A link in /vis/<project> will make this transparent for the user. /ltmpObservationData is a partition that is not backuped up as a backup had been made before.
- When the project is finally removed after a monthly backup another backup for IRAM will be made.
- The current pool period is kept in /vis/pool
- When a pool scheduling period is finished, then we will make a backup of /vis/pool (on LTO) and move the files in subdirectory observationData to /ltpmPool . By that, astronomers can still work on old pool data but we avoid to make more backups of observationData/... .
- For /ltmpPool we will offer initially a 500GB Sata disk.
- AS suggests to reorganize archiving of datastreams:
at regular intervals move datastreams older than ? from /ncsServer/mrt/ncs/datastreams/<unit> to /ncsServer/mrt/ncs/<date>/datastreams/<unit> .
- make a tape backup and move scan data and datastreams to an archive disk every maintenance day
- AS suggests to keep imbfits files in the project directories and use a soft-link from
- /ncsServer/mrt/ncs/data/imbfits. Invalid soft-links shall be removed during maintenance, or via a cronjob.
- Should we request a daily backup time, e.g., stopping observation during or dinner time ?
The design is under construction.
On host mrt-lx1, we will have the following data:
- /ncsServer : everything needed to run NCS software:
- sources, binaries, scripts, libraries.
- log files
- documentation, part of the doc will also be in the mrt-lx1 wiki area
- /ncsServer is currently on a 40GB partition and uses 7GB. This is ok.
- /ncsData (:
- data streams, data stream files will be moved to ncsArchive on mrt-lx3 every 6 hours
- scan data /ncsData is currently on a 205GB partition and uses 127GB including OCS projects. The ocs projects will be removed. Partition can stay as it is for now.
- local user files for: root, hello, mrt
On host mrt-lx3, we will have the following data:
- /vis : project directories
- Currently on a 240GB partition, using 94%. Old projects will be removed soon. We will try to keep the partition of this size.
- /users : files of local users
- Currently, partion 154GB, 50% used. Is on mrt-lx1 and will be moved to mrt-lx3.
- /ncsArchive : data stream files older than 6 hours
this is page https://mrt-lx1.iram.es/mainWiki/FileOrganization, moderated by WB
- please mark personal opinions with your initials