While trying to reimage db1173 (old s6 master), the host won't get back up.
I have tried several things (including a hard reset, poweroff and then power back up) etc but even though the ILO says the power is on, I don't see the server booting up or anything.
Could you take a look? If possible this week as this is the host we'd failover to in case of a primary master failure in s6.
If it cannot be done this week, let me know so I can prepare another host as a stand by master
Thanks!
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db1173: Disable notifications | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Marostegui | T298585 Upgrade WMF database-and-backup-related hosts to bullseye | |||
| Resolved | Marostegui | T299479 Upgrade s6 to Bullseye | |||
| Resolved | • Cmjohnson | T310595 db1173 won't boot up |
@Marostegui at first glance the server was hanging up during the boot process at memory configuration, I did not get any hardware errors, I put the server down to minimum post requirements (1 CPU and 1DIMM) and put everything back from there, on the last reboot DIMM A7 presented as having failed. A ticket has been placed with DELL.. Part may be here tomorrow or Friday. I replaced it for now with a spare from a decommissioned server so you do not have to set up another backup server.
Thank you so much @Cmjohnson! I can indeed access the host now and I have reimaged it sucessfully. Decreasing the priority since the initial issue was triaged (so fast!). So once the part is there just let me know one day/hour and I can leave it off for you.
@Cmjohnson did the part arrive? I don't know if you want to place that new part on this host or if you prefer to leave the DIMM that you already placed? Thank you!
Excellent, can you let me know a day and time that works for you to replace it? I can leave the host offline for you
Mentioned in SAL (#wikimedia-operations) [2022-06-29T09:01:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1173 for on-site maintenance T310595', diff saved to https://phabricator.wikimedia.org/P30603 and previous config saved to /var/cache/conftool/dbconfig/20220629-090120-root.json
Change 809540 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1173: Disable notifications
@Jclark-ctr host offline, you can proceed whenever you want. Once you are done, please power it back on and I will take it from there.
Thanks a lot!
Change 809540 merged by Marostegui:
[operations/puppet@production] db1173: Disable notifications