Page MenuHomePhabricator

(Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org
Closed, ResolvedPublic

Description

This task will track the racking, setup, and installation of atlas-codfw.wikimedia.org

Hostname / Racking / Installation Details

The existing atlas has failed: https://netbox.wikimedia.org/dcim/devices/1297/

This new one will need to replace it, using the same hostname, as the old one is completely defunct. This should be done by renaming the old one to asset tag/decommissioning it, then using that hostname on new.

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.
atlas-codfw:

  • - receive in system on procurement task T269046 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - serial setup/testing - device connected to console server, validated although just printing junk right now.
  • - Get USB install image from RIPE Atlas admins
  • - Flash install image to USB key
  • - Power cycle hardware with USB key inserted to start OS install
  • - Validate device comes online / works after OS install
  • - Await confirmation from RIPE that device is online and it shows on public portal (https://atlas.ripe.net/anchors/list/)

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Related Objects

StatusSubtypeAssignedTask
Resolvedcmooney
Resolvedcmooney

Event Timeline

RobH added a parent task: Unknown Object (Task).Jan 27 2021, 9:05 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
Papaul added subscribers: CDanis, Papaul.

@CDanis this all complete on my side. The device is connected on scs1-a1 on port 47. Tested console using Cisco console cable with baud rate of 115200 it works.

@Papaul do you know what the status is with this device? I can confirm there are some characters visible via serial console / port 47 of scs-a1-codfw, so that looks to be working. But it doesn't seem to be much use as is (no menu, readable text or option to interrupt boot etc. appears.)

Do we have the boot image for this device from RIPE? And is it on a USB key installed in the unit?

Is it connected to the network? Netbox shows no interface/connection details, however atlas-codfw_old shows as still being connected to asw-a1 ge-1/0/4? Perhaps you just re-used that port? Right now that port is up, and there is a MAC address learnt on it, but ARP/ND for the IP addresses the anchor should be using fails from the CRs. So looks like the OS is not installed on the device.

I as far as i know I removed the old faulty device, replaced it with this on, connected the console, power and network to the device, that's all

I've moved netbox details (console and ethernet connection, IP addressing) from old device to the replacement device now, reflecting the actual setup in the DC. I also defined a new device_type for it (RIPE Atlas Anchor v3) as it is a different model than the others we have in stock.

I've also contacted RIPE about getting the image to install/configure the device from USB. Will update when I have update back from them.

Can the setup of this device please be noted on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Atlas_Anchor ?

I've never set one of these up, but we'll need to replicate the setup for drmrs.

@RobH no problem I will document the process there once it is fully clear to me.

Update for now is that RIPE have gotten back to me and allowed me to register the new hardware, which I've now done. I expect the next step is they provide a boot image and install instructions for us, which I will document.

RIPE have emailed back to confirm registration and provided image download

From: RIPE Atlas (no reply) [mailto:no-reply@ripe.net]
Sent: Tuesday, August 17, 2021, 10:34 AM
To: Cathal Mooney
Subject: RIPE Atlas Anchor Registration for Wikimedia Foundation (us-dal-as14907)

> Dear Cathal Mooney,
>
> Thank you for registering the technical information for your RIPE Atlas anchor!
>
> You will receive an email shortly with instructions on how to install the anchor software.
>
> If you have any questions, please do not hesitate to contact us at atlas@ripe.net
>
> You registered the following details:
>
> IPv4 address: 208.80.152.244
> IPv4 netmask: 255.255.255.240
> IPv4 gateway: 208.80.152.241
> IPv4 ASN: 14907
>
> IPv6 address: 2620:0:860:201:208:80:152:244
> IPv6 prefix: 64
> IPv6 gateway: 2620:0:860:201::1
> IPv6 ASN: 14907
>
> Kind regards,
>
> RIPE Atlas Team

Image is saved to /srv/firmware/ on apt1001. Will review instructions and work with DC-Ops to get it installed.

@RobH I've updated that Wiki page now with instructions on how to create the USB disk image and begin the install.

https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Atlas_Anchor#Firmware_Image

@Papaul When you have some time please take a look. When you've been able to prep the USB installer we should co-ordinate to update the Anchor OS (I want to view the console during the process to capture the output).

Thanks.

Papaul rebooted yesterday with the USB key present. Things appeared to go ok, on the serial console the device went into a Linux (CentOS) boot sequence after which it seemed to begin the install process:

Starting installer, one moment...
anaconda 21.48.22.93-1 for CentOS Linux 7 started.
 * installation log files are stored in /tmp


                           ....




Installing nss-softokn (62/299)
Installing libnl3-cli (63/299)
Installing libassuan (64/299)
Installing e2fsprogs-libs (65/299)
Installing cyrus-sasl-lib (66/299)
Installing groff-baInstalling ethtool (83/299)

Installing chkconfig (27/299)
Installing nss-util (28/299)
Installing readline (33/299)
Installing elfutils-libelf (35/299)
Installing libcap-ng (36/299)
Installing libgpg-error (39/299)
Installing libattr (41/299)
Installing libgcrypt (44/299)
Installing libxml2 (45/299)
Installing xz (69/299)
Installing libunistring (71/299)
Installing ca-certificates (77/299)
Installing tcp_wrappers-libs (79/299)
Installing keyutils-libs (82/299)
Installing ethtool (83/299)
Installing iptables (86/299)
Installing less (88/299)
Installing nettle (92/299)
Installing shared-mime-info (95/299)
Installing gettext-libs (97/299)
Installing gsettings-desktop-schemas (99/299)
Installing dbus-glib (100/299)
Installing libselinux-utils (109/299)
Installing ncurses (111/299)
Installing libproxy (113/299)
Installing snappy (114/299)
Installing libpcap (115/299)
Installing slang (117/299)
Installing libverto (119/299)
Installing cracklib-dicts (130/299)
Installing gettext (131/299)
Installing libmount (133/299)
Installing libpwquality (135/299)
Installing pam (136/299)
Installing systemd-libs (137/299)
Installing procps-ng (139/299)
Installing plymouth-core-libs (140/299)
Installing python-pyudev (141/299)
Installing python-iniparse (142/299)
Installing libselinux-python (143/299)
Installing python-slip (144/299)
Installing grubby (152/299)
Installing bind-libs-lite (153/299)
Installing nss (154/299)
Installing nss-tools (157/299)
Installing logrotate (158/299)
Installing binutils (159/299)
Installing alsa-lib (160/299)
Installing mariadb-libs (161/299)
Installing curl (164/299)
Installing rpm-libs (165/299)
Installing rpm (166/299)
Installing openldap (167/299)
Installing libuser (168/299)
Installing python-urlgrabber (172/299)
Installing fipscheck-lib (173/299)
Installing rpm-python (179/299)
Installing gpgme (180/299)
Installing pygpgme (181/299)
Installing qrencode-libs (189/299)
Installing libaio (190/299)
Installing device-mapper-persistent-data (191/299)
Installing libutempter (195/299)
Installing util-linux (196/299)
Installing device-mapper-libs (199/299)
Installing cryptsetup-libs (200/299)
Installing dracut (201/299)
Installing kmod (202/299)
Installing systemd (203/299)
Installing dbus (204/299)
Installing device-mapper-event-libs (210/299)
Installing cronie (216/299)
Installing crontabs (217/299)
Installing policycoreutils (218/299)
Installing openssh (219/299)
Installing libpciaccess (221/299)
Installing libdrm (222/299)
Installing device-mapper-event (228/299)
Installing libsoup (236/299)
Installing fxload (238/299)
Installing libestr (242/299)
Installing kbd (249/299)
Installing rsyslog (250/299)
Installing man-db (251/299)
Installing kexec-tools (259/299)
Installing grub2 (260/299)
Installing openssh-clients (264/299)
Installing postfix (265/299)
Installing microcode_ctl (271/299)
Installing btrfs-progs (277/299)
Installing xfsprogs (278/299)
Installing rootfiles (281/299)
Installing iwl6050-firmware (284/299)
Installing iwl5000-firmware (287/299)
Installing iwl5150-firmware (288/299)
Installing iwl6000-firmware (289/299)
Installing iwl3945-firmware (292/299)
Installing iwl135-firmware (295/299)
Installing iwl1000-firmware (297/299)
Performing post-installation setup tasks

@Papaul it seems things went ok, but looking at the instructions I think we need to remove the USB drive and then power cycle again. Whenever you're next out on site can you do that? Thanks.

Seems like we have success :)

Port is now up and MAC address learnt:

cmooney@asw-a-codfw> show ethernet-switching table | match 1/0/4.0  
    sandbox1-a-codfw    00:0d:b9:57:4c:b8   D             -   ge-1/0/4.0

And I can ping both public IPs:

cathal@nbgw:~$ ping 208.80.152.244
PING 208.80.152.244 (208.80.152.244) 56(84) bytes of data.
64 bytes from 208.80.152.244: icmp_seq=1 ttl=50 time=133 ms
64 bytes from 208.80.152.244: icmp_seq=2 ttl=50 time=128 ms
^C
--- 208.80.152.244 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 128.282/130.428/132.575/2.146 ms
cathal@nbgw:~$ ping 2620:0:860:201:208:80:152:244
PING 2620:0:860:201:208:80:152:244(2620:0:860:201:208:80:152:244) 56 data bytes
64 bytes from 2620:0:860:201:208:80:152:244: icmp_seq=1 ttl=56 time=188 ms
64 bytes from 2620:0:860:201:208:80:152:244: icmp_seq=2 ttl=56 time=125 ms
64 bytes from 2620:0:860:201:208:80:152:244: icmp_seq=3 ttl=56 time=127 ms
^C
--- 2620:0:860:201:208:80:152:244 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 125.226/146.700/188.251/29.386 ms

I've logged onto the RIPE Atlas portal and clicked through the 'install software' screen to let RIPE know it's up and running. They'll do their tests and then add it to the live Atlas system. I'll keep this task open until we hear back from them to confirm all looks ok.

And we are live :)

From: RIPE Atlas [mailto:atlas@ripe.net]
Sent: Thursday, September 30, 2021, 2:31 PM
To: Cathal Mooney
Subject: Your RIPE Atlas anchor (us-dal-as14907) is now online

> Dear Cathal Mooney,
>
> We're happy to announce that your RIPE Atlas anchor is functioning properly and is now connected to the RIPE Atlas network.
>
> You can see your anchor when logged in to the RIPE Atlas website.
>
> The direct link to the probe page for the anchor is here:
> https://atlas.ripe.net/probes/7038/
>
> Alternatively, go to the "My Atlas" menu and click on "Anchors".
>
> You can now use your anchor as a probe for your own measurements, and select it as a target under "Measurements".
>
> You can also find a map and complete list of RIPE Atlas anchors at:
> https://atlas.ripe.net/anchors/map/
> https://atlas.ripe.net/anchors/list/
>
> As a RIPE Atlas anchor host, you will earn ten times the number of credits as a regular probe host.
> Learn more about what you can do with your credits at: https://atlas.ripe.net/about/customised-measurements/
>
> Another benefit of hosting a RIPE Atlas anchor is having your logo displayed on the Community page: 
> https://atlas.ripe.net/get-involved/community/#!tab-anchor-sponsors
> If you would like us to include your logo, please provide us with the link to your website and your company logo in SVG or EPS format, or a high-resolution PNG of at least 300x300 pixels.
>
> Thank you for participating in RIPE Atlas!
>
> Kind regards,
>
> RIPE Atlas Team
> atlas@ripe.net