To note, this host was rebooted and that matches the time that this failed:

cwilliams@db2248:~$ last -1000 | grep -Fm1 reboot
reboot   system boot  6.12.95+deb13-am Tue Jul 28 14:59 - still running

Checking the status of x4, there is traffic but it the ticket (T404715) suggests still WIP.
For now, added 4 days of silence to the SystemdUnitFailed alert, given that it is otherwise showing good health.

The plan is to depool, do a switchover and then debug the host; the switchmaster tool doesn't seem to like x4 and the cluster is not in orchestrator, so the steps will need to be adjusted slightly.
Once depooled and free of replicas:

• systemd reset-failed ifup@eno12399np0.service to check that works OK
• reboot the host to see if it reappears on the next reboot

NetOps confirmed that there is nothing wrong per se, given that the address is already active. It is likely to be a race condition, which for VMs has already been noted (modules/toil/manifests/ganeti_ifupdown.pp and modules/profile/manifests/base/production.pp)

[15:08] <topranks> seems odd though, we have a vanilla /etc/network/interfaces here, and we've thousands of hosts like that and not seen this before
[15:11] <topranks> looking at the logs it does seem to have failed first time, so perhaps there is some race condition we've not seen before

Some notes about the section:

[15:19] <Amir1> so it's pooled under s4 and x4
[15:19] <Amir1> with the difference being x4 is not getting any
[15:20] <jynus> I see, so it would need depool from both, and I guess a master switch too
[15:20] <Amir1> yeah
[15:20] <Amir1> they connect to the same port, same mariadb instance. It's just pretending to be two different things (until we split them apart)

In T433610#12171657, @CWilliams-WMF wrote:

Checking the status of x4, there is traffic but it the ticket (T404715) suggests still WIP.
For now, added 4 days of silence to the SystemdUnitFailed alert, given that it is otherwise showing good health.

Yeah, it is a bit confusing.
x4 will be a split from s4. But for now those hosts are serving in s4 but with the tag of x4 (and placed on both sections). There's still MW work to do before we can proceed with the split.

The plan is to depool, do a switchover and then debug the host; the switchmaster tool doesn't seem to like x4 and the cluster is not in orchestrator, so the steps will need to be adjusted slightly.
Once depooled and free of replicas:

• systemd reset-failed ifup@eno12399np0.service to check that works OK
• reboot the host to see if it reappears on the next reboot

NetOps confirmed that there is nothing wrong per se, given that the address is already active. It is likely to be a race condition, which for VMs has already been noted (modules/toil/manifests/ganeti_ifupdown.pp and modules/profile/manifests/base/production.pp)

[15:08] <topranks> seems odd though, we have a vanilla /etc/network/interfaces here, and we've thousands of hosts like that and not seen this before
[15:11] <topranks> looking at the logs it does seem to have failed first time, so perhaps there is some race condition we've not seen before

Some notes about the section:

[15:19] <Amir1> so it's pooled under s4 and x4
[15:19] <Amir1> with the difference being x4 is not getting any
[15:20] <jynus> I see, so it would need depool from both, and I guess a master switch too
[15:20] <Amir1> yeah
[15:20] <Amir1> they connect to the same port, same mariadb instance. It's just pretending to be two different things (until we split them apart)

Mentioned in SAL (#wikimedia-operations) [2026-08-03T10:35:36Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db2248 from s4 T433610', diff saved to https://phabricator.wikimedia.org/P95854 and previous config saved to /var/cache/conftool/dbconfig/20260803-103535-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2026-08-03T10:36:53Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db2245, db2246 and db2247 T433610', diff saved to https://phabricator.wikimedia.org/P95855 and previous config saved to /var/cache/conftool/dbconfig/20260803-103652-marostegui.json

@CWilliams-WMF I've depooled the slaves too, you can silence them: db2245, db2246 and db2247 (as well as db2248 the master) and go ahead and reboot it.

In T433610#12178223, @Marostegui wrote:

@CWilliams-WMF I've depooled the slaves too, you can silence them: db2245, db2246 and db2247 (as well as db2248 the master) and go ahead and reboot it.

If you want me to do it, I can also to that :)

I've depooled the slaves too, you can silence them: db2245, db2246 and db2247 (as well as db2248 the master) and go ahead and reboot it.

@Marostegui thanks, I will do that now.

Icinga downtime and Alertmanager silence (ID=2685c559-3f2e-4204-b523-b15e420b421a) set by cwilliams@cumin1003 for 4:00:00 on 4 host(s) and their services with reason: Checking network

db[2245-2248].codfw.wmnet

A reset of the service cleared the alert.

root@db2248:~# systemctl status --lines=0 ifup@eno12399np0.service
× ifup@eno12399np0.service - ifup for eno12399np0
     Loaded: loaded (/usr/lib/systemd/system/ifup@.service; static)
     Active: failed (Result: exit-code) since Tue 2026-07-28 14:59:26 UTC; 5 days ago
   Duration: 154ms
 Invocation: 68c0a7aa8fd14647a58942b10e2ca3f4
    Process: 1138 ExecStart=/usr/sbin/ifup --allow=hotplug eno12399np0 (code=exited, status=1/FAILURE)
   Main PID: 1138 (code=exited, status=1/FAILURE)
   Mem peak: 2.5M
        CPU: 72ms

root@db2248:~# systemctl reset-failed ifup@eno12399np0.service

root@db2248:~# systemctl status --lines=0 ifup@eno12399np0.service
○ ifup@eno12399np0.service - ifup for eno12399np0
     Loaded: loaded (/usr/lib/systemd/system/ifup@.service; static)
     Active: inactive (dead)

After that, the host was rebooted and the service is showing as OK:

root@db2248:~# uptime
 11:31:29 up 1 min,  2 users,  load average: 0.45, 0.18, 0.06
 
root@db2248:~# systemctl status --lines=0 ifup@eno12399np0.service
● ifup@eno12399np0.service - ifup for eno12399np0
     Loaded: loaded (/usr/lib/systemd/system/ifup@.service; static)
     Active: active (exited) since Mon 2026-08-03 11:30:38 UTC; 39s ago
 Invocation: 870a479aa92a4aed935e2750a2550db4
    Process: 1134 ExecStart=/usr/sbin/ifup --allow=hotplug eno12399np0 (code=exited, status=0/SUCCESS)
   Main PID: 1134 (code=exited, status=0/SUCCESS)
   Mem peak: 4.6M
        CPU: 136ms

Good news!
Let me know when you want me to repool the replicas via dbctl and not the cookbook.

Let me know when you want me to repool the replicas via dbctl and not the cookbook.

The 3 replicas need their kernel upgrading:

(3) db[2245-2247].codfw.wmnet                                                                                                                                                                                                                                                                                                        
----- OUTPUT for command #1: 'uname -r' -----                                                                                                                                                                                                                                                                                        
6.12.94+deb13-amd64

I don't think that the rolling restart script covers these, shall I try the multiinstance cookbook as that doesn't use Zarcillo?

@Marostegui they are all rebooted and green, so ready for you to repool.

For reference, done with

sudo cookbook sre.mysql.multiinstance_reboot 'db[2245-2247].codfw.wmnet' --reason=Maintenance

Mentioned in SAL (#wikimedia-operations) [2026-08-03T15:12:13Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool db2245, db2246, db2247 and db2248 T433610', diff saved to https://phabricator.wikimedia.org/P95857 and previous config saved to /var/cache/conftool/dbconfig/20260803-151212-marostegui.json

Repooled!