Mentioned in SAL (#wikimedia-cloud) [2021-09-03T15:20:13Z] <bstorm> stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 T290318

Mentioned in SAL (#wikimedia-cloud) [2021-09-03T15:24:15Z] <bstorm> stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 T290318

The commands are hanging to disconnect this server from the cluster, so I have to reboot it in order to break the link. I've downtimed it in icinga until Tuesday and am doing that now.

Mentioned in SAL (#wikimedia-cloud) [2021-09-03T15:34:17Z] <bstorm> rebooting labstore1005 to disconnect the drives from labstore1004 T290318

That was successful:

root@labstore1004:~# drbd-overview
 1:test/0   StandAlone Primary/Unknown UpToDate/DUnknown /srv/test  ext4 9.8G 535M 8.7G 6%
 3:misc/0   StandAlone Primary/Unknown UpToDate/Outdated /srv/misc  ext4 5.0T 1.9T 3.0T 39%
 4:tools/0  StandAlone Primary/Unknown UpToDate/Outdated /srv/tools ext4 8.0T 6.1T 1.5T 81%

With that in "standalone" mode, it won't try to reconnect without manual help.

The urgency is moderate here. The NFS service for the cloud (a core function of Toolforge ) is still up and should function fine. This is the failover and backup host. We will not be able to take weekly backups of Toolforge or any other cloud VPS NFS until labstore1005 is stable again, and failover will be impossible. This is the only warm standby host and backup source for the majority of Cloud-Services NFS.

tldr: We aren't down, but one of our core services to stay up is now a single point of failure with backups stopped.

Change 717948 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] cloud NFS: tighten up the traffic shaping a little

https://gerrit.wikimedia.org/r/717948

Change 717948 merged by Bstorm:

[operations/puppet@production] cloud NFS: tighten up the traffic shaping a little

https://gerrit.wikimedia.org/r/717948

The first thing we will need to do is to try and update the raid controller firmware along with the bios.

Update from sub-task and IRC discussions:

Updating the firmware is the first thing to try. However, the system is currently experiencing transient failures, any troubleshooting could result in repair or complete failure of the controller. Rather than potentially offline an important backups server without any recourse a replacement controller is in the process of being ordered via T290602.

When it arrives, the current controller should have the firmware updated to see if the error clears. It is far easier to just clear the error than swap the controller and import the config.

If the update of firmware doesn't clear the error, then we should do a couple things:

• backup the config of the raid if possible, if not just note it manually
• swap controller, controller may detect 'foreign config' on the disks, if it does, use the foreign config as its the old controllers configuration data written to the raid array and should simply import the config.

If the new controller isn't needed, it is still an H730P that could be used in other systems if needed in the future.

Just a quick update - the replacement part was shipped out on Monday, and should be arriving today. (might be in the loading dock already)

https://www.dell.com/support/order-status/en-us/order-detail?v=POdYeYCNspqtdufquPwQVErv79WCLVapFzmCIJSeulM%3D&src=oc&ref=ocdpidirn

https://wwwapps.ups.com/etracking/tracking.cgi?tracknums_displayed=10&TypeOfInquiryNumber=T&HTMLVersion=4.0&track=Track&InquiryNumber1=1ZX5656V0370774572&requester=ST/trackdetails

Thanks,
Willy

Attempting to flash the raid firmware results in failure to verify package contents on the firmware file. The firmware file is fine (re-downloaded and checked the checksum locally before upload twice, upload attempt 2 times per file download). It seems the idrac is just ancient, so it has to be updated first.

Each of these attempts included a reboot, so I've left it powered off for the idrac flash so we don't have to keep rebooting the OS for no reason.

Unfortunately its 2.41, and when you try to flash to 2.81 (latest) it will flash to 2.61 first, then require reflash of the same file to push to 2.81. This 2.61 introduces a bad SSL self sign generation that breaks firefox and chrome, but in the past works in Safari. So I'm flashing this idrac now and will complete hte flash in safari (ssh tunneling of local proxy) to complete idrac, then reattempt the raid bios.

attempted flash of bios and raid bios, failure to due unable to verify package - upload attempted twice from the same set of downloaded files
reattempted flash of bios and raid bios with new downloads and uploads - failure to due unable to verify package - upload attempted twice from the same set of newly downloaded files
attempted flash of idrac to resolve package verification issues, idrac https locked up and required racreset command via ssh
attempted flash of idrac from 2.30 to 2.81, failed due ot package verification. we're going to have to step this up manually by cutting the versions down one at a time.

tried to cut the difference and go half as old on firmware, didnt work (package verify failed) so started just incrementing one version at a time.

idrac
2.30 to 2.40 successful
2.40 to 2.41 successful
2.41 to 2.52 successful

Mentioned in SAL (#wikimedia-cloud) [2021-09-16T15:56:03Z] <bstorm> removing downtime for labstore1005 so we'll know if it has another issue T290318

So far so good:

[bstorm@labstore1005]:~ $ sudo journalctl -S "2021-09-15" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
[bstorm@labstore1005]:~ $

robh@labstore1005:~$ sudo journalctl -S "2021-09-15" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$ sudo journalctl -S "2021-09-16" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$ sudo journalctl -S "2021-09-17" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$

still going strong