Upgrade firmware on db1136
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Mar 10 2021, 7:16 AM

Description

While working on the reboots for the stretch kernel upgrade we ran into a known issue with some firmwares: T216240.
This affected db2073 (T276909) and by upgrading its firmware we got it back.

We need to switchover current s7 primary master to a candidate master, db1136 (T274336) but before doing so, we'd like to upgrade the kernel on that host, which would need a reboot.
@wiki_willy can we get sometime scheduled with eqiad dcops so the host can get upgraded?.

We'd like this to have some high priority as we need to replace s7 primary master soon (scheduled for 23rd March), as it is a host that might run into BBU issues (T258386) and could crash anytime.

Let us know which day/time could work so we can have the host with mysql stopped and powered off.

Thanks!

Details

	Subject	Repo	Branch	Lines +/-
	db1136: Enable notifications	operations/puppet	production	+0 -1
	db1136: Disable notifications	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Resolved	Legoktm	T281515 June 2021 Datacenter switchover
Resolved	Marostegui	T285139 DB maintenance work to do while eqiad is passive (June 2021)
Resolved	Marostegui	T266483 Enable report_host for mariadb
		Restricted Task
		Restricted Task
		Unknown Object (Task)
Resolved	Marostegui	T258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
Declined	None	T258386 db1080-95 batch possibly suffering BBU issues
Resolved	Marostegui	T274336 Switchover s7 from db1086 to db1136
Resolved	sgrabarczuk	T276899 Read only time window needed for s7
Resolved	Marostegui	T276150 Schema change to make rc_id unsigned and rc_timestamp BINARY
Declined	None	T216240 Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092
Resolved	• Cmjohnson	T277007 Upgrade firmware on db1136

Event Timeline

Marostegui triaged this task as High priority.Mar 10 2021, 7:16 AM

Marostegui created this task.

Marostegui moved this task from Triage to Blocked on the DBA board.

Marostegui added parent tasks: T258386: db1080-95 batch possibly suffering BBU issues, T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092.

Maintenance_bot added a project: SRE.Mar 10 2021, 7:45 AM

LSobanski subscribed.Mar 10 2021, 10:21 AM

wiki_willy reassigned this task from wiki_willy to • Cmjohnson.Mar 10 2021, 4:53 PM

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Mar 11 2021, 3:34 PM

@Marostegui can we schedule this for Monday next week? 1500/1600UTC timeframe please? Thanks

Sounds good @Cmjohnson - I will leave the host off beforehand so you can proceed as you wish. Once you are done, just power it back on and I will take it from there.
Thank you

Change 672337 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1136: Disable notifications

https://gerrit.wikimedia.org/r/672337

gerritbot added a project: Patch-For-Review.Mar 15 2021, 8:38 AM

Change 672337 merged by Marostegui:
[operations/puppet@production] db1136: Disable notifications

https://gerrit.wikimedia.org/r/672337

Mentioned in SAL (#wikimedia-operations) [2021-03-15T08:54:10Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1136 T277007', diff saved to https://phabricator.wikimedia.org/P14829 and previous config saved to /var/cache/conftool/dbconfig/20210315-085409-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-03-15T08:54:49Z] <marostegui> Stop MySQL on db1136 T277007

@Cmjohnson db1136 is now off, you can proceed as needed

Marostegui moved this task from Blocked to In progress on the DBA board.Mar 15 2021, 8:58 AM

Maintenance_bot removed a project: Patch-For-Review.Mar 15 2021, 9:10 AM

@Marostegui updated the BIOS firmware

This host booted from PXE boot, and attempted to reimage itself.
Luckily the partman recipe we have didn't delete its data. Did the BIOS upgrade change the default boot method?