Page MenuHomePhabricator

db2075 failed to boot kernel 2/3 tries, please upgrade firmware/BIOS to mitigate
Closed, ResolvedPublic

Description

The parent task affected host by only occasionally failing to boot. db2075 failed twice in a row before being able to fully boot.

@Papaul, could we get it upgraded (firmware/BIOS) so we can reliably reboot it.

The host is in service, please tell us when available so we can stop it in advance (should normally take just 5 minutes to stop it).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:28:12Z] <jynus@cumin2001> dbctl commit (dc=all): 'depool db2075 for mw maintenance T254139', diff saved to https://phabricator.wikimedia.org/P11411 and previous config saved to /var/cache/conftool/dbconfig/20200608-152811-jynus.json

Before

 BIOS Version 	
2.4.3
Firmware Version 	
2.40.40.40
IP Address(es) 	
10.193.1.55
iDRAC MAC Address 	
84:7B:EB:F6:97:56
DNS Domain Name 	
Lifecycle Controller Firmware 	
2.40.40.40

After

 BIOS Version 	
2.11.0
Firmware Version 	
2.70.70.70
IP Address(es) 	
10.193.1.55
iDRAC MAC Address 	
84:7B:EB:F6:97:56
DNS Domain Name 	
Lifecycle Controller Firmware 	
2.70.70.70

@jcrespo firmware upgrade complete

Thank you for the help, putting the services back up.

@Marostegui there seems to be a bug on 10.1.45-MariaDB installed locally, as the systemd unit doesn't notify the start (despite actually getting started). Probably you already know about this- so I will not research further.

Yeah, I was testing the new version on that host with the new package and then I got into lots of others things.
If you have some time to kill and don't mind taking a look, that'd be appreciated, otherwise I will try to get to it in next week or something

Mentioned in SAL (#wikimedia-operations) [2020-06-11T11:54:31Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2075 T254139', diff saved to https://phabricator.wikimedia.org/P11469 and previous config saved to /var/cache/conftool/dbconfig/20200611-115430-marostegui.json