For later:

Jul 01 00:46:43 es1039 mysqld[5895]: 2026-07-01  0:46:43 6 [Warning] Detected table cache mutex contention at instance 1: 26% waits. Additional table cache instance cannot be activated: consider raising table_open_cache_instances. Numb>
Jul 01 00:46:52 es1039 mysqld[5895]: 2026-07-01  0:46:52 9 [Note] Start semi-sync binlog_dump to slave (server_id: 172008451), pos(./es1039-bin.005358, 4)
Jul 01 00:46:52 es1039 mysqld[5895]: 2026-07-01  0:46:52 12 [Note] Start semi-sync binlog_dump to slave (server_id: 171970719), pos(./es1039-bin.005358, 4)
Jul 01 00:46:52 es1039 mysqld[5895]: 2026-07-01  0:46:52 10 [Note] Start semi-sync binlog_dump to slave (server_id: 172005380), pos(./es1039-bin.005358, 4)

For the checklist on the switchover steps:

< Amir1> we follow the checklist outlined in the ticket
< Amir1> https://phabricator.wikimedia.org/T430765
< Amir1> the checklist is produced by switchmaster (https://switchmaster.toolforge.org/

The switchover is done, the cluster is RW now:
https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=2026-07-01T00:12:34.145Z&to=2026-07-01T01:09:38.332Z&timezone=utc&var-site=eqiad&var-group=core&var-shard=es7&var-role=$__all&viewPanel=panel-19

I leave es1039 depooled for HW inspection and what is wrong. Repool when needed.

In T430764#12074260, @ssingh wrote:

For the checklist on the switchover steps:

< Amir1> we follow the checklist outlined in the ticket
< Amir1> https://phabricator.wikimedia.org/T430765
< Amir1> the checklist is produced by switchmaster (https://switchmaster.toolforge.org/

When you do it in emergencies, there are some intricacies. The biggest is this: When the master is rebooted, it comes back online as read only, you need to explicitly set it to RW in MySQL (set global read_only = 0; on the host). You can also do the switchover while the master is RO with --read-only master option when running db-switchover commands but I didn't go that route since the whole section was RO anyway while it'd be better to do it that way for core sections.

Also it's important to check mariadb logs (the systemd service logs) to make sure things are not firework-y. The crash recovery mechanism of MariaDB is quite robust these days but you never know.

And heartbeat needs a restart after crash (the pt-heartbeat-wikimedia systemd service)

Icinga downtime and Alertmanager silence (ID=f965c095-744c-4fe7-997d-d637cbb7a225) set by slyngshede@cumin1003 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Hardware crash

es1039.eqiad.wmnet

Host is stilled depooled. Adding downtime, for 1 day.

for the record: the host crashed again and triggered multiple p.ages.

jelto@es1039:~$ uptime 
 03:35:02 up 2 min,  1 user,  load average: 0.13, 0.05, 0.01

Thank you all, I will follow up from here.

In T430764#12074209, @CDanis wrote:

00:42:36 <Amir1> okay, the user impact should be gone now
00:43:11 <Amir1> it's removed from writes

Followups: documentation, documentation, documentation.

Yeah unfortunately it wasn't clear from the doc alert.
We do have the cookbook script which depools pcX and msX even if a master is down, we should adapt it for esX too. I will create a task for it.

Nothing on HW logs that I can see, last entry from 2025.

Change #1306847 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1039: Disable notifications

https://gerrit.wikimedia.org/r/1306847

Change #1306847 merged by Marostegui:

[operations/puppet@production] es1039: Disable notifications

https://gerrit.wikimedia.org/r/1306847

As soon as I put some load on es1039 it crashed again unfortunately no HW logs produced again.
@VRiley-WMF @Jclark-ctr can you check from your side if you are able to see any other logs?

The last entry is from 2025, which is very strange.
It is clear that something is very wrong with this host, it is not able to hold any load (not even mysql replication load, which should be minimal).
I'll select another candidate master in the meantime.

@CDanis @Ladsgroup could you help replacing the description with a summary of the incident?

In T430764#12074594, @MLechvien-WMF wrote:

@CDanis @Ladsgroup could you help replacing the description with a summary of the incident?

I will take care of that.

Thanks, could you also put the Severity Phabricator tag ?

I've not been able to make the host crash again even doing some stress tests. I will keep researching.

this paged again this morning

In T430764#12078767, @ABran-WMF wrote:

this paged again this morning

I am not sure why this paged again:

• the host has notifications disabled
• the issue from yesterday should've recovered yesterday too when I brought it up and started mariadb, which caught up so at all effects the host is recovered.

[07:16:53]  <marostegui> !incidents
[07:16:56]  <+sirenbot> 8120 (ACKED, 25h 41m old)  es1039 (paged)/MariaDB Replica SQL: es7 (paged)
[07:16:56]  <+sirenbot> 8121 (ACKED, 25h 41m old)  es1039 (paged)/MariaDB Replica IO: es7 (paged)

That's strange because the host is fully up, so no idea why it didn't auto resolve.

For the record:

[07:19:13]  <marostegui> !resolve 8120
[07:19:16]  <+sirenbot> 8120 (RESOLVED, 25h 43m old)  es1039 (paged)/MariaDB Replica SQL: es7 (paged)
[07:19:16]  <marostegui> !resolve 8121
[07:19:19]  <+sirenbot> 8121 (RESOLVED, 25h 43m old)  es1039 (paged)/MariaDB Replica IO: es7 (paged)

I've been unable to make this host crash again so on Monday I will pool it back and close this ticket.

Change #1307678 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1039: Enable notifications

https://gerrit.wikimedia.org/r/1307678

@VRiley-WMF this host crashed a few days ago and has been working fine ever since. However I just tried a normal reboot before placing it back in production and it never came back.
I've attached to the console but it is showing nothing. Could you double check?
Would you also be able to check if idrac/bios/firwmare/anything else needs any update?

Thank you!

@Marostegui Of course, I will take a look at this and update it.

Hey @Marostegui BIOS and iDRAC have been updated and it seems like it boots just fine now. Should it be okay to close this out?

Can you assign it back to me becasue there're a few things required from my side.
Thanks!

Upon looking at a similar issue ticket to this one that was resolved, The backplane firmware was needed for an update as well. I have just updated the backplan firmware with I believe may have fixed the other issue. Would you mind testing it again? @Marostegui

Scratch that, sorry. I just logged back into it and it seems to throw the same error. Let me make a few more changes on my end.

Dell had me update the firmware for the drives and the backplan as well to be sure. The system is currently showing as healthy, but I did pull another set of logs from the system and sent them over to Dell. If you would like to test it out @Marostegui please go for it. I'm currently waiting to head back from Dell.

Thanks Valerie! Starting some more tests now.

Change #1307678 merged by Marostegui:

[operations/puppet@production] es1039: Enable notifications

https://gerrit.wikimedia.org/r/1307678

Starting pool of es1039 by marostegui@cumin1003: Repooling after testing

I am slowly repooling this host after a few successful tests.

Completed pooling of es1039 by marostegui@cumin1003: Repooling after testing

@Marostegui Hopefully everything is looking good. I will close this ticket for now. Feel free to open it if there is another issue.

Thank you for your help!