Differences between revisions 91 and 92
Revision 91 as of 2021-06-22 19:43:30
Size: 13334
Editor: navarro
Comment:
Revision 92 as of 2021-06-22 19:45:51
Size: 13441
Editor: navarro
Comment:
Deletions are marked like this. Additions are marked like this.
Line 9: Line 9:
   * Finished installation of barycentric cables. Now they go from RX cabin to Backend room (JLS, GP, JP)

Weekly Maintenance Log

  • June 22, 2021
    • - Frontend (SN, DJ)
      • Second leak test performed on NIKA2. No leak found.
      • Second PLL reference box has been tested. Frequency switching is working. It has to be integrated in NCS (JLS & GP).

      • A water leak has been found on the HERA cooling system. HERA will be warmed.
      • Safety switch installed on the back of NIKA M6 mirror. Not connected yet.
      • Second VLBI synthesizer (new one) has been compared to old one. Same phase noise on both (SS).
      • Finished installation of barycentric cables. Now they go from RX cabin to Backend room (JLS, GP, JP)
  • June 15, 2021
    • - Computer
      • rxCommander, changed the apsyn set/get frequency for hera1 and hera2 commands to anapico 23 and 14.
      - Frontend (SN, DJ, JLS)
      • Heavy maintenance activities during the revision of the gear boxes:
      • Replaced NIKA2 duoPomp relay. The control of the duoPomp is back to the graphical interface and no longer on the IP switch.
      • EMIR has been warmed up to ambient temperature and cooled again to clean the vacuum.
      • Tested new 16 GHz oscillator for EMIR outer bands down conversion (GP)
      • Two coaxial cables, for the barycentric reference, have been installed in the spiral.
      • Measured EMIR IF cables. They are all OK.
      • Tested holography receiver and transmitter.Weather did not allow to put the receiver on the outside so no check of the satellite has been done.
      • Installed, but not yet in service, EMIR PLL second reference box (replacement of ADRET synthesizers)
      • Rewired EMIR PLL first reference cables (Anapico synts) through the VLBI switch box.
      • Revised tuning table of E150 local oscillator.
      • Installed a local control for the insertion of the VLBI quarter wave plates.
      • Installed a water leaks sensor in the RX cabin.
      • Cleaning of some unnecessary boxes (old E0/E3 switch, E3 amplifier and its power supply).
  • June 8, 2021
    • - Computer
      • Nothing: busy with administrative stuff
      - Frontend (SN)
      • Most of the time before lunch was spent on checking the inventory of the frontend group and the workshop
      • Replaced the 2mm LO box by the spare (Gunn H245).
      • Checked several frequencies (135 li, 150 li and 170 ui) with the new 2mm LO box before leaving the telescope.
      • Revised the vertex tunnel to check for obstacles in the signal path: nothing found
      - Telescope
      • Solving overtemperature problem in the receiver cabin
      • Revising with MC the process for purging the antenna glicol circuit, qpod and BUS
      • Putting glicol in the antena glicol circuit
      • Replacing 1/96 pt100 temp. sensor of the BUS to confirm maqlfunction (not in place)
  • June 1, 2021
    • - Computer
      • mrt-lx1 maintenance
      • email list problems
      • mrt-lx2 reboot, 41 days up time and too many processes running.
      - Frontend dj
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages and reset.
      • Checked NIKA cooling system.
      • Rechecked NIKA laptop( in Rx cabin) - now works fine.
      • Looked at new water leak with Juan Pedro
      • Two and a half full bottles of LqNe are in Rx cabin. 5 in garage.

      - Telescope
      • replacement of the motor encoder Az2-M1 because noisier
      • revision of the AC system of Rx room
      • antenna axes encoder reset (because four encoders frozen)
      • annual maintenance of diesel group by Rivera with battery replacement
      • diesel group test with load including low voltage switches activation
      • measurement and adjustment of water chlorination
  • May 25, 2021
    • - Computer
      • MRT Isilon Upgrade
      • MRT VM rebalancing
      • mrt-lx1 log rotation
      • proj-rsync VM turned back on, Netdata installed
      • LDAP server reboots
      - Fronted (SN)
      • Prepared remaining parts (calibration mirror, actuator and electro-vanes) for AETHRA project. They will be sent to Grenoble this week.
      • Labelled new EMIR synthesizer box
      • While preparing NIKA2 for observations a strong correlation was found on the noise of all pixels in array 2. Investigation is ongoing.
      - Telescope (JP, MC, IR)
      • Replacement of the antenna motor El2-M2, including the clutch. Activity within the preventive maintenance.
      • Changed the module for the brake control of spindle HOR2. It was identified as failing and should be the reason of the last pointing problems reported by operators.
  • May 18, 2021
    • - Computer
      • Granada Isilon Upgrade
      • Cleaning up mrt-lx1 VM snapshot/close datastore
      • Disk replacement in MRT Isilon
      • MRT NetApp Disk Shelf fault investigation

      • Reconfigure DNS with new "intranet.iram.es" domain
      - Telescope
      • Revision of both AC units in Computer-BE room by Marín to suppress sporadic high-pressure alarm
      • Low Voltage revision including activation of power switches and generator, with M. Castillo
      • General revision of the Telescope
      - Frontend (dj)
      • Hoisted 3 bottles of lqN2 to Rx cabin. 3 full in Rx cabin, 2 full in garage, 2 empty in garage.
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages.
      • Checked NIKA cooling system.
  • May 11, 2021
    • - Computer
      • General maintenance tasks: log cleaning, disk checks etc.
      • Work on containers
      • Changed MTA from exim4 to postfix as a security precaution on prod servers
      • Maintenance bust: mrt-lx1 completely unusable. FP had to recover from backup. Started observing at 20:00 local time.
      - Frontend (SN)
      • Reinstalled Anapico synthesizer box after modifications.
      • Checked NIKA2 water cooling system.
  • May the 4th, 2021 (Star Wars day ;-))
    • - Computer
      • antenna.iram.es and antennads.iram.es tests for bug fixes.
      - Frontend (SN)
      • Dismounted Anapico synthesizer box for final modifications.
      • Checked collision problem between EMIR carousel and NIKA M6 mirror. A new protection switch will be installed
        • during the heavy maintenance.
  • Apr. 27, 2021
    • - Computer
      • MRT VMWare upgrade part III
      • nika2-b checks, reboot, diagnostics
      • antenna.iram.es changed to get the antenna info from rabbitmq instead of the database in mrt-lx1.
      • Add project script to mrt-lx2 PAM session
  • Apr. 20, 2021
    • - Computer
      • Granada VMWare upgrade part II
      • antenna.iram.es changed to run on Python 3.9, several other improvements also carried out to this web page.
      • mrt-lx2 security patches to kernel, Xorg, other packages (+reboot)
      - Frontend (sn)
      • Installed and tested new synthesizer box (Anapico) for PLL reference. At present only one synthesizer can be used.
      • Installation of polycarbonate plates on the RX cabin ceiling ongoing.
  • Apr. 13, 2021
    • - Computer
      • Granada VMWare upgrade part I
      • Cleaning up elevator room
      • vac1 tests
      • nika2-b raid expansion
      - Frontend dj
      • Hoisted 3 bottles of lqN2 to Rx cabin. 3 full in Rx cabin, 2 full in garage, 3 empty in garage.
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages.
      • Checked NIKA cooling system.
  • Apr. 5, 2021
    • - Computer
      • Nothing critical
      - Frontend SN
      • Replaced old ceiling lamp for installation of water protection polycarbonate plates.
      • Polycarbonate plates design was modified but still not installed.
      • Prepared VLBI phase noise setup at 230 GHz.
      • Prepared with J. Pedro the new Anapico synthesizer box. Work in progress.
  • Mar. 30, 2021
    • - Computers
      • NFS Tests
      • Radiolink and fiber failover tests
      • Update of the firmware in the MRT and Granada Mikrotiks that are used as "The Dude" servers (10.0.69.100 and 10.0.69.200).
      - Frontend dj
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages.
      • Checked NIKA cooling system.
      • Hoisted 3 bottles of N2 to rx cabin.
      • Revised water leak situation in Rx cabin with Juan Pedro.
      • Revised 192.....66 pc software.
  • Mar. 23, 2021
    • - Computers
      • General maintenance tasks: cleanup logs, replace failed disks etc. Not intrusive as we have tech. time in two days.
      - Frontend
      • A replacement for NIKA2 array 3 synthesizer has been installed. Test will be done during the next TT on Friday 25th.
      • Phase noise test for 1.3mm VLBI has been done in preparation for the coming test with NOEMA.
      • Refilled NIKA2 water cooling circuit.
  • Mar. 16, 2021
    • - Computers
      • General maintenance tasks: cleanup logs, replace failed disks etc.
      - Frontend dj
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages.
      • Checked NIKA cooling system
      • Investigated 192.....66 machine.
  • Mar. 09, 2021
    • - Computers
      • Move /ncsServer from mrt-lx1 to ncs-nfs (ROLLED BACK)
      • Update wiki containers on mrt-lx0 (patches)
      • Added disk for docker registry to gra-lx1
      • fsck'd all disks on gra-lx1
      • recovered failed disk on nika2-b (/ssd stripe)
      • fsck disks on nika2-a|b
      - Frontend
      • Replaced broken synthesizer on array 3 of NIKA2. Very noisy data on array 3 after that.
      • After a second synthesizer was installed (on March 10th) the array 3 was again operational.
      • Water pressure on NIKA2 He compressors was low. Increased up to 2 bar.
  • Mar. 02, 2021
    • - Computers
      • Backup pruning (the servers are getting full)
      • .dbm NFS bug investigation ongoing
      • Patch docker daemon on gra-lx1, mrt-lx0, www (security)
      • Rebooted www (for security patches and fsck)
      - Frontends. dj
      • Started NIKA warmup with JLS
      • Checked HERA cooling system
      • Checked Emir i2c units for error messages.
      • Removed Anapico synth from E0
      • Investigated problem of communication between pc ...66 and NIKA compressors using a new RS 232 to RS 485 converter (spare NIKA).
        • Unit was tested at home using loop feedback and checked out okay, however problem still present.
      • Investigated problem with an independent pc and converters but problem still remains. Need to review Labview software.
      • Transferred two NIKA adsorber boxes fro garage to cable spiral with the help of Juan Pedro.
  • Feb. 23, 2021
    • - Computers
      • General low-impact maintenance activities
      - Frontends
      • Prepared Anapico synthesizer for TT on Thursday 25. Hardware was checked.
      • Craned 3 N2 bottles to the RX cabin
      • Checked NIKA2 water cooling: Refilled water up to 2 bar pressure
      - Backends
      • Action 1
      • Action 2
      - Telescope
      • Action 1
      • Action 2
  • Feb. 16, 2021
    • - Computers
      • FP: Attempt at eliminating the use of DBM format in the NCS (replace it with .pickle). Some progress but still many files to go (127 total).
      • FP: rotated NCS logs and general maint. on elvin.
      • WR: Patches/reboot on ncs-nfs.iram.es
      - Frontends
      • Checked water leaks in and on top of the receiver cabin
      • Investigated problem on NIKA2 control PC (192.168.1.66) for He compressor control. Still not working remotely
      - Backends
      • Action 1
      • Action 2
      - Telescope
      • Action 1
      • Action 2
  • Feb. 09, 2021
    • - Computers
      • FP: Attempt at eliminating the use of DBM format in the NCS (replace it with .pickle). Reverted for now.
      • All: Support power cut in Granada offices (switching machines off and on, monitor the UPS etc.)
      • PM: Removed /vis2 mount from mrt-lx1 and mrt-lx3, it is not needed anymore according to AS. The /vis2 was not deleted in the local filesystems just in case. The directory was indeed removed from the NetApp partition.

      • WR: Enable Livepatch and unattended-upgrades on ncs-nfs2
      • WR: Changed from autofs to manual /etc/fstab entries on mrt-lx6, also disabled unattended-upgrades
      • PM: Removed old projects accounts from 2010 to 2015 see the list in RemovedOldProjectsAccountsFebruary2021. The old passwd files can be found at /root/remove_old_accounts.

      - Frontends
      • Action 1
      • Action 2
      - Backends
      • Action 1
      • Action 2
      - Telescope
      • Action 1
      • Action 2
  • Mar. 02, 2021
    • - Telescope
    • JP: Training M. Castillo about Servos: movement Az, El and Sub. PLC. Remote access
    • JP: Tracking failure of the telephone line for reading the electricity counters
  • Mar. 09, 2021
    • - Telescope
    • JP, General maintenance in servo area: motors, fans, lights, stowpins. Revising spindles movement in local, cleaning potentiometers.
    • JP, Revising AC Rx cabin, ventilation antenna and tower.
    • JP, Checking pilots information LV. Replacing a blown pilot
  • Mar. 16, 2021
    • - Telescope
    • JP, replacement of an antenna motor (El2,M2) within the preventive maintenance (with M. Castillo)
    • JP, training M. Castillo of remote antenna movement and associated programs (for main axes, M2 and wobbler)
  • Mar. 23, 2021
    • - Telescope
    • JP, MC, greasing the antenna crown Az and El. By Peri.
    • JP, MC, checking UPS autonomy. 15 min with batteries
    • JP, MC, modification of pointing model by dP7 = +12"
    • JP, MC, minor maintenance: glicol level, AC receiver, ventilation antenna, servos, ...
  • Mar. 30, 2021
    • - Telescope
    • JP, MC, testing Az and El brakes measuring the operative sliping torque and the gap between ferodos
  • Apr. 6, 2021
    • - Telescope
    • JP, MC, checking the emergency limit switches in azimuth and elevation
    • JP, MC, identifying failure of ventilator 1 of 5 of the BUS. Replacing motor next week

MaintenanceLog (last edited 2021-06-22 19:45:51 by navarro)