Day 13 Update:

Day 12 Update (in progress, will edit as day progresses):

This migration was completed just know with no issues. Thanks to both @Jclark-ctr and @herron for the on-site part and the icinga handling!

Day 11 Update:

• 8 hosts moved, 5 remain out of 308 total hosts.
• John did all the moves today working with Andrew.
• Migrated 6 of the 8 WMCS hosts found and added to T411025, only clouddumps1002 and cloudelastic1010 remain from WMCS.
• lvs10[19,20] hosts pending migration scheduling with Cathal and Brett.
• alert1002 scheduled for migration on 2025-12-03 @ 13:00 GMT

New host count:

I prefer we not wait for the entire refresh of pending Q2 hosts but instead migrate all these hosts during the first or second week of December.

So the disk hasn't failed out of md0, just md1 and md2. I'd attempt to rebuild manually and if that doesn't work then RMA the drive since it shows no errors in smartctl.

Both kafka-main100[89] moved, last one to move is wikikube-ctrl1003

In T407897#11399303, @Marostegui wrote:

Thanks Rob, I think the confusion was whether we ordered the right HW or not. Doing 1G is fine for this host, 10G would be ideal, but we are not expecting all hosts to be doing 10G anyway (as far as I know, we are (or were) very limited in 10G switches).
So whatever works best for DCOps, we can live with 1G or 10G.

I've chatted with @brouberol via IRC:

Day 9 Update:

• 9 hosts moved, 10 remain - 300 hosts total at start of migration
• John worked with Ben directly to migrate the (8) Data Platform hosts this AM.
• The last (4) Data Platform hosts are scheduled for migration tomorrow.
• Myself and John worked with Scott to get conf1009 migrated today.
• (2) lvs hosts will be moved after Cathal coordinates with Brett for patch submission this week
• alert1002 will migrate on 2025-12-03 @ 13:00 GMT T405946
• (3) ServiceOps hosts remain: wikikube-ctrl1003, kafka-main100[89], feedback pending on sub-task T405950#11402388.

IRC Echo Update (chatting with Scott in irc about this just echoing to task for history):

In T405950#11402180, @RobH wrote:

conf1009 migrated,

@brouberol: Please provide feedback on migration of wikikube-ctrl1003 and kafka-main1008 as these are the last serviceops hosts to migrate:

In T405950#11389672, @RobH wrote:

@brouberol, you were tagged into this task by T405950#11236474 but I don't have any feedback on the migration details for kafka-main1009 other than: "Coordinate with ServiceOps, some badly written clients may need a kick"

We would like to migrate this on either Thursday, Nov 20 or next week Nov 24-26, earlier in the week preferred. Would you detail what needs to be done for this to happen?

If we need to run any depool commands or if you can depool it from use for one of those days, please let us know!

conf1009 migrated,

Can we also set a date/time for moving the other three hosts remaining?

IRC Update:

Day 8 Update:

Sent an email to @BTullis to ensure he is aware of these 12 hosts pending his feedback, subject line: Need IF feedback for 12 remaining hosts since November 12th

Clarification Questions and statements:

In T405950#11393903, @Scott_French wrote:

@RobH - Confirming conf1009 for 2025-11-24, but the SRE staff meeting runs from 17:00 - 18:00 UTC. I'd suggest starting no later than 16:30 (and pausing by 17:00) or starting at / after 18:00.

All Infrastructure-Foundations hosts in eqiad c/d rows migrated to the new switch stacks.

Day 8 Update:

Please note all wikikube workers have been migrated and we're now down to only 4 hosts left with serviceops to migrate:

@Ladsgroup had other things going on and wasn't able to do this today but did link me to the directions on how to depool: https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting

I've set a gcal event for 2025-12003 @ 10AM EST / 15:00 GMT for the alert1002 migration.

I totally messed up the dates on your comment: 2025-11-20, 2025-11-24, 2025-11-25 2025-12-03, 2025-12-04

Please note all hosts listed on this task have been migrated.

Please note we didn't get to these two today, will do tomorrow!

In T405950#11238805, @Scott_French wrote:

conf1009 is (1) a member of eqiad main-etcd cluster, so clients will attempt to issue writes to it, (2) the upstream source for etcd-mirror replication to codfw, and (3) not a PyBal config host. Also, it sounds like the connectivity disruption should be very brief, similar to previous migrations (i.e., switch-side of the cable just moves from adjacent old device to new).

Given that, the simplest reasonable sequence would be:

• Silence EtcdReplicationDown
• Migrate to the new switch
• On "done" from DC-Ops, in parallel:
  • Test conf1009 (e.g., check cluster membership on peers, probe conf1009 with a quorum-read)
  • Check the health of etcd-mirror on conf2005 and restart if needed
• Delete silence
• Restart eqiad-associated confds and navtiming, verify Liberica control-plane daemons are healthy

Rationale:

• This is only a single cluster member (i.e., the cluster will maintain quorum, though there may be an election), so there's no strong justification to temporarily point eqiad-associated clients at codfw, particularly given the amount of effort and disruption (e.g., read-only periods) involved.
• We could temporarily point etcd-mirror at a different member, but I don't think that's worth the effort either. The timed-watch strategy it uses tends to be remarkably robust to "temporarily pull the network cable" (more so than to a proper upstream connection close). Also, the "check and restart if needed" approach worked well when the recent round of nginx updates reached conf1009.
• There is some risk that if replication is disrupted and a large influx of writes pushes us out of the 1000 event log window, then recovery is a bit involved. We could rule this out by making etcd read-only during the migration. However, that turns the "oops, your client hit a transient error attempting to communicate with conf1009" into "full write unavailability," which again seems rather disruptive for a low-likelihood issue.

In any case, I'll give some thought to timing and follow up in the sheet. It's only the last point that I'm still on the fence about (i.e., whether to bracket the migration with read-only), but (1) that's technically easy to do if we choose to do it and (2) should not have much of an impact on scheduling (it mostly just requires additional upfront comms).

We've migrated 9 of the 10 observability hosts. We're now only left with alert1002 which the notes detail will require scheduling. In addition to picking a scheduled date, can you also provide any steps and details I'll need to do the date of the migration?

We're now down to 44 hosts overall to migrate, and 12 of those belong to your team.

Migration Update:
Only 3 Data-Persistence hosts remain for migration: pc101[678].

@brouberol, you were tagged into this task by T405950#11236474 but I don't have any feedback on the migration details for kafka-main1009 other than: "Coordinate with ServiceOps, some badly written clients may need a kick"

Day 7 Update:

• 33 hosts moved today, 44 remain
• all row c wikikube migrated, some of row D wikikube migrated
  • 23 wikikube hosts remain out of the 44 left to move
• (2) pc hosts will be moved next week on 24, 25, or 26th. Manuel is out this Thursday-Friday.
• (2) lvs hosts will be moved whenever @cmooney would like to schedule being around for it as they are a bit touchy. lvs1020 is backup to 1019, both must move so 1020 will move first.
• other hosts left on list need details submitted (they've been asked on sub tasks this week) or scheduling.

Going to depool wikikube in rack eqiad D1 for port migrations.

sudo cookbook sre.k8s.pool-depool-node -t T405950 -r 'network migration' --k8s-cluster wikikube-eqiad pool wikikube-worker126[0-9].eqiad.wmnet
sudo cookbook sre.k8s.pool-depool-node -t T405950 -r 'network migration' --k8s-cluster wikikube-eqiad pool wikikube-worker10[36,51,52,54,54,55,83].eqiad.wmnet

Depooling wikikube in rack C6:

In T405942#11388802, @Ladsgroup wrote:

Please ping me before moving of pc1014 so I depool pc4 cluster from rotation.

• backup1006, backup1007, ms-backup1002 moved yesterday.
• db1189 was moved yesterday by accident sorry about that!
• The only data persistence hosts left to move are:
  • moss-be1002 - no directions provided on moving this, please advise
  • pc1014 - scheduled to move today
  • pc1016 - not yet scheduled for migration
  • pc1017 - not yet scheduled for migration
  • pc1018

I think it'll be ok when we move things tomorrow, since I know exactly the mistake I made I don't think I'll make it again for a few months minimum ; D

In T405950#11386181, @Scott_French wrote:

@RLazarus and I were looking into verifying workloads returning to the migrated workers, and ran into a few surprises.

Going by what's been marked complete in the sheet from today, there appears to be a difference between the what was depooled vs. what was ostensibly migrated.

Specifically, it looks like the following, where - indicates "in the sheet, not depooled" and + indicates "not in the sheet, depooled" (no mark indicates agreement):

+wikikube-worker1016
 wikikube-worker1063
-wikikube-worker1135
-wikikube-worker1136
-wikikube-worker1137
-wikikube-worker1138
-wikikube-worker1139
-wikikube-worker1154
-wikikube-worker1155
-wikikube-worker1156
 wikikube-worker1157
+wikikube-worker1254
+wikikube-worker1255
+wikikube-worker1256
 wikikube-worker1305
 wikikube-worker1306
 wikikube-worker1313

So, there appear to be 4 workers depooled that weren't in-scope for migration, and 8 workers migrated that weren't depooled (of which 3 look like maybe a typo in the depool command - i.e., wikikube-worker[1254-1256] vs wikikube-worker[1154-1156]).

In any case, as far as we can tell, workloads on the 8 workers appear to have weathered it the disruption.

@RobH - Were the 8 that weren't depooled, but appear to be checked off actually deferred to tomorrow? If not (i.e., they were indeed migrated), is there some way we can help prepare the depool commands ahead of time to make sure it's aligned with the planned migrations?

Thank you for the update, we'll likely move these two hosts tomorrow!

dupe of T408510

dupe of T410456

IRC Discussion Update:

optic swap by john fixed it.

This is indeed detecting slow:

Day 6 Update:

Awesome! We're also moving dns1006 at the same time (we'll move it first while k8 hosts drain) and then we'll move onto moving these! I'll ping you in about 70 minutes for the start of the window at 17:00 GMT. Thanks!

All machine learning hosts have been migrated, resolving this task.

In T405945#11372555, @RobH wrote:

@LSobanski,

The only two Infrastructure-Foundations hosts left to migrate are

In T405945#11250904, @LSobanski wrote:

• aux-k8s-worker100[67]: can be drained at any time, we have a cookbook

However, this doesn't quite make it clear if I should run the cookbook and move whenever works for me, or if your team should run the cookbook and you want to set a date/time to work with us or have us ping you via irc?

If it is something I can run the cookbook, can you advise exactly what cookbook and flags I'd run to migrate each of these two hosts?

Please note we now only have 12 data platform hosts remaining for migration. I still need clarification for

@Marostegui and/or @jcrespo,

Is it possible that I could send the commands for this or do we need someone in your team? If we need someone in your team, could we schedule an hour or so for this tomorrow (Tuesday, Nov 18thy) at 18:00GMT start time? (So 10:00 AM Pacific?)

@BCornwall and @ssingh: We chatted about this last week, can we schedule this work to move dns1006 tomorrow, Tuesday November 18th at 9AM Pacific / 5PM GMT?

Sorry about that, we've now migrated all of the non scheduled migration hosts (except k8) so we can schedule this for next week. Would Tuesday, Nov 18th @ 915 Pacific / 17:15GMT work for migrating the two gitlab hosts?

The only two Infrastructure-Foundations hosts left to migrate are

Day 5 Update:

All ganeti hosts migrated to their new switch ports in eqiad rows c/d

ganeti1028
ganeti1047
ganeti1048
ganeti1037

IRC Update from chat with Tobias:

IRC Update:

For dns1006, since the downtime of the host is around 5-12 seconds (missing about 5-12 seq numbers via ping) I'm not sure it even has to be fully depooled as long as we don't move it during an active dns sync.

All cp hosts in rows C/D have been migrated as of today (last ones done) and all that is left in Traffic realm for migration is dns1006 and lvs1020 via T405602.