Re-enable clouddb1020 wikireplica (analytics s5 and s8)
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	• Bstorm
	Oct 8 2021, 3:11 PM

Description

Since the hardware has been repaired, we now have to put clouddb1020 back in service.

Ensure the database is sound and replication is restored on s5 and s8 (Data Persistence)
Update the views for T291806 and T292043 to match the rest (WMCS)
Repool the host (T292043)

Status	Subtype	Assigned	Task
Duplicate		None	T291961 clouddb1020 crash
Resolved	Request	• Cmjohnson	T291963 hw troubleshooting: crash (with thermal event) for clouddb1020.eqiad.wmnet
Resolved		None	T292850 Re-enable clouddb1020 wikireplica (analytics s5 and s8)

• Bstorm triaged this task as Medium priority.Oct 8 2021, 3:11 PM

• Bstorm created this task.

Current status: host is up, mariadb is not yet

I've brought mariadb@s5 up without starting replication, and i'm running mariadb-check --all-databases against it now.

s5 check completed successfully after 6h50m. I've enabled replication, and it's catching up now.

Doing the same procedure for s8, check running now.

@Bstorm: clouddb1020:s5 has caught up on replication, and is released into your loving care :)

s8 is still running mariadb-check, so probably won't have replication re-enabled until tomorrow.

s8's check finished successfully, i've started replication on it now.

@mdipietro per ^ looks like clouddb1020 can be repooled

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Completed in T292043 closing.

rook closed this task as Resolved.Jan 19 2023, 2:01 PM