Note that @KOfori is out, this should be directed to @Kappakayala in the meantime.

@RobH - do you want me to add the plan in the task description or in comments like:

dbproxy1024	C6 Can be done anytime - just needs downtime
dbproxy1025	D6 Can be done anytime - just needs downtime
dbproxy1029	C3 Can be done anytime - just needs downtime

Provided that the moves happen one at a time (probably goes without saying), then the Cassandra hosts can be done at any time, and without coordination. The Cassandra hosts here are: aqs*, restbase*, & sessionstore*

For backup* and ms-backup* hosts, I would prefer to stop the backup process (mediabackups). Nothing would be lost if they fail (they will be retried again), but it takes so little effort to do it for me beforehand that is better if I stop and restart it after the maintenance.

es1031 C3
es1032 C3
es1033 D8
es1034 D8

Can be ignored, will be decommissioned in a couple of weeks (T406690)

es1045 C5
es1046 D6
es1051 D1
es1052 D3
es1053 D6
es1057 C3

Can be done anytime, just need downtime

Updating https://docs.google.com/spreadsheets/d/13ow4JxrsQdz8KSsdBBNwvlrAuGKo8OHWcnR4RhXTYc0/edit?usp=sharing with the details of the move is likely easiest, but I'm copying all the above comments into that sheet.

In T405942#11268506, @Eevans wrote:

Provided that the moves happen one at a time (probably goes without saying), then the Cassandra hosts can be done at any time, and without coordination. The Cassandra hosts here are: aqs*, restbase*, & sessionstore*

aqs*, restbase*, & sessionstore can be done anytime without coordination. @Eevans: So no icinga notice and just move the network port without further interactions with the OS or services? If so, that is by far the easiest. Do they require any time between hosts? That is 6 aqs hosts, 5 restbase, and 2 sessionstore, so less than 6 business days.

So if we need to do anything other than move the port, please let us know.

@marosgui: So the hosts you list as dbproxy and es with "Can be done anytime, just need downtime", do you mean we can just downtime in icinga, or do we need to depool the host somehow from services? We'd likey then just treat them similar to the aqs, restbase, and sessionstore host and just plan to move one es and one dbproxy per day.

In T405942#11273802, @RobH wrote:

Updating https://docs.google.com/spreadsheets/d/13ow4JxrsQdz8KSsdBBNwvlrAuGKo8OHWcnR4RhXTYc0/edit?usp=sharing with the details of the move is likely easiest, but I'm copying all the above comments into that sheet.

Will do it that way then

@marosgui: So the hosts you list as dbproxy and es with "Can be done anytime, just need downtime", do you mean we can just downtime in icinga, or do we need to depool the host somehow from services? We'd likey then just treat them similar to the aqs, restbase, and sessionstore host and just plan to move one es and one dbproxy per day.

Correct, you can just downtime them and move - just let us know it was done so we can double check and in the dbproxy's case just reload haproxy (to clarify: we will do the reload)

So, the swift & Ceph nodes:

• ms-be* please do 1 at a time (and check pingable again before moving onto the next)
• ms-fe* please EITHER do one at a time and depool before move and pool afterwards OR depool swift.discovery from eqiad entirely, do all of them and then repool
• thanos-fe1007 please depool before move, pool afterwards
• apus-fe* / moss-fe* please do 1 at a time and depool before move and pool afterwards
• moss-be1002 can just be done (but I'd like to know when!)

In T405942#11273802, @RobH wrote:

[ ... ]

In T405942#11268506, @Eevans wrote:

Provided that the moves happen one at a time (probably goes without saying), then the Cassandra hosts can be done at any time, and without coordination. The Cassandra hosts here are: aqs*, restbase*, & sessionstore*

aqs*, restbase*, & sessionstore can be done anytime without coordination. @Eevans: So no icinga notice and just move the network port without further interactions with the OS or services? If so, that is by far the easiest. Do they require any time between hosts? That is 6 aqs hosts, 5 restbase, and 2 sessionstore, so less than 6 business days.

So if we need to do anything other than move the port, please let us know.

Sorry, I should have been clearer here. We should definitely downtime them to avoid noise (the downtime cookbook), and verify that they're back up before proceeding (though once they're back up it should be fine to proceed directly with the next one).

Please note this migration has shifted from Oct 15th start date to Nov 1 start date.

In T405942#11276918, @MatthewVernon wrote:

So, the swift & Ceph nodes:

• ms-be* please do 1 at a time (and check pingable again before moving onto the next)
• ms-fe* please EITHER do one at a time and depool before move and pool afterwards OR depool swift.discovery from eqiad entirely, do all of them and then repool

Is this just a depool command from its cli?

• thanos-fe1007 please depool before move, pool afterwards

Is this just a depool command from its cli?

• apus-fe* / moss-fe* please do 1 at a time and depool before move and pool afterwards

Is this just a depool command from its cli?

• moss-be1002 can just be done (but I'd like to know when!)

Yes, for each of those hosts you can depool by running depool from the host in question (and then pool afterwards). Thanks!

Mentioned in SAL (#wikimedia-operations) [2025-11-06T17:21:48Z] <robh> multiple moves from C/D per T405942

@Marostegui and/or @jcrespo,

Please note we've now migrated all Data-Persistence hosts that do not require direct scheduling with you via the notes. We now only have the remaining hosts:

backup1006
backup1007
db1153
db1167
db1181
db1184
db1189
db1221
db1222
es1033
moss-be1002
ms-backup1002
pc1013
pc1014
pc1016
pc1017
pc1018

All of these hosts have notes to have one or the other (some list Jaime, some list Manuel) to coordinate with you for these hosts. Some need depool, some will likely need master failovers, etc.

Would you please review the remainder of the list and if possible, shuffle master/redundant host architecture around and let us know how / when / who will work with us to move the remaining 17 Data Persistence hosts? Anything that can be staged for us to move on our own (John and myself) great please update and tell us, otherwise please pick a date/time with overlap to Eastern Timezone working hours for John to work with you on migrating these hosts.

@RobH thanks for working on this!
For backup* and ms-backup1002 @jcrespo would be able to tell you better than I do
For moss-be1002 @MatthewVernon

As per the rest I can prepare most of those ahead (as you mentioned, failovers, special depooling etc), this is a brief list of things that need (mostly for my own)

db1153 - depooling ms3 and downtime codfw
db1167 - depooling + downtime wikireplicas
db1181 - switchover
db1184 - switchover
db1189 - switchover
db1221 - depooling + downtime wikireplicas
es1033 - depooling
pc1013 - depooling + downtime codfw
pc1014 - depooling + downtime codfw
pc1016 - depooling + downtime codfw
pc1017 - depooling + downtime codfw
pc1018 - depooling + downtime codfw

@Jclark-ctr I can leave the following hosts ready for you to do anytime tomorrow when you get to the DC and you can proceed anytime you want during your day without having me to be present:

db1153
db1167
db1221
es1033
pc1013
db1181
db1184
pc1013

For Wed I can have the following hosts ready for you when you get to the DC:

db1189
pc1014

Would the list of hosts for Tuesday and Wednesday work for you?

And that would be it for this week as I am out Thu half day and Friday as on-call compensation.
The following week we could do the pending ones: pc1016 pc1017 and pc1018

@Marostegui Thanks, that works for me!
Was es1033 going to be decommissioned? Is that still the case?
Also, will any cookbooks need to be run for these, or will I just be able to run the NetBox script and move them?

In T405942#11379665, @Jclark-ctr wrote:

@Marostegui Thanks, that works for me!
Was es1033 going to be decommissioned? Is that still the case?

Yeah, but we need it for a bit longer.

Also, will any cookbooks need to be run for these, or will I just be able to run the NetBox script and move them?

I will leave them fully ready for you to do your side of things (like you've done with the other databases) - I will even downtime them myself.

@Jclark-ctr Would me stopping backups tomorrow, Tuesday 18 before your TZ (e.g. before 11 am UTC/6am Eastern Timezone) and then those host can be done at any time during your day (ideally not stopped > 24 hours) work for you?

backup1006
backup1007
ms-backup1002

If you say that works for you, I will get them ready for tomorrow. If not, propose any other day, and I will have it prepared for you that day in advance.

Stress on any time,please don't wake up early for those, backups can be stopped for hours, but ideally not for more than 1 day.

@jcrespo That works for me Thanks!

Mentioned in SAL (#wikimedia-operations) [2025-11-18T06:30:10Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depool pc1 T405942', diff saved to https://phabricator.wikimedia.org/P85355 and previous config saved to /var/cache/conftool/dbconfig/20251118-063010-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-11-18T06:30:48Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depool ms3 T405942', diff saved to https://phabricator.wikimedia.org/P85356 and previous config saved to /var/cache/conftool/dbconfig/20251118-063048-marostegui.json

@Jclark-ctr the following hosts are ready for you to proceed. No special cookbooks or downtime are required:
db1153
db1167
db1121
es1033
pc1013
db1181
db1184
pc1013

Icinga downtime and Alertmanager silence (ID=8cc3d2b8-5b7e-411e-aa85-6a4983c97ec1) set by jynus@cumin1003 for 1 day, 0:00:00 on 4 host(s) and their services with reason: Network maintenance

backup[1006-1007].eqiad.wmnet,ms-backup[1001-1002].eqiad.wmnet

Media backups processing on eqiad is stopped and the following hosts have been downtimed for 24 hours from now:

backup1006
backup1007
ms-backup1002

any work on them can proceed at any time.

@RobH / @Jclark-ctr as I noted above, moss-be1002 can be done whenever, I'd just like to be told when you're going to do it, please.

Mentioned in SAL (#wikimedia-operations) [2025-11-19T06:25:10Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool ms3 T405942', diff saved to https://phabricator.wikimedia.org/P85372 and previous config saved to /var/cache/conftool/dbconfig/20251119-062509-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-11-19T06:26:34Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool ms3 T405942', diff saved to https://phabricator.wikimedia.org/P85373 and previous config saved to /var/cache/conftool/dbconfig/20251119-062634-marostegui.json

@Jclark-ctr
db1189
pc1014

Those can be moved anytime when you get to the DC

@Jclark-ctr
db1189
pc1014

Those can be moved anytime when you get to the DC

@Jclark-ctr I think we scheduled db1189 for today but it was done yesterday? The spreadsheet marks it as done and also I can see:

[Tue Nov 18 17:39:15 2025] tg3 0000:04:00.0 eno1: Link is down
[Tue Nov 18 17:39:21 2025] tg3 0000:04:00.0 eno1: Link is up at 1000 Mbps, full duplex

If that's the case, no problems happened so that's good.

Based on the spreedsheet, no more interruptions are expected on

backup1006
backup1007
ms-backup1002

So I will restart eqiad media backups now.

Update:

• backup1006, backup1007, ms-backup1002 moved yesterday.
• db1189 was moved yesterday by accident sorry about that!
• The only data persistence hosts left to move are:
  • moss-be1002 - no directions provided on moving this, please advise
  • pc1014 - scheduled to move today
  • pc1016 - not yet scheduled for migration - need a good date/time from you
  • pc1017 - not yet scheduled for migration - need a good date/time from you
  • pc1018 - not yet scheduled for migration - need a good date/time from you

Please ping me before moving of pc1014 so I depool pc4 cluster from rotation.

In T405942#11388798, @RobH wrote:

• moss-be1002 - no directions provided on moving this, please advise

@RobH, not mine, but please note the comment above https://phabricator.wikimedia.org/T405942#11384678

In T405942#11388802, @Ladsgroup wrote:

Please ping me before moving of pc1014 so I depool pc4 cluster from rotation.

Will do, it won't be until after 17:00 GMT today.

In T405942#11388807, @jcrespo wrote:

In T405942#11388798, @RobH wrote:

• moss-be1002 - no directions provided on moving this, please advise

@RobH, not mine, but please note the comment above https://phabricator.wikimedia.org/T405942#11384678

Indeed, sorry about that, google sheet updated!

In T405942#11388802, @Ladsgroup wrote:

Please ping me before moving of pc1014 so I depool pc4 cluster from rotation.

pc4 was depooled earlier today. Nothing to be done.

Mentioned in SAL (#wikimedia-operations) [2025-11-19T17:16:23Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool pc4 T405942', diff saved to https://phabricator.wikimedia.org/P85395 and previous config saved to /var/cache/conftool/dbconfig/20251119-171622-marostegui.json

Repooled pc4 as Rob confirmed pc1014 has been moved.

Migration Update:
Only 3 Data-Persistence hosts remain for migration: pc101[678].

Chatted with @marosgui earlier in IRC and he'll be out Thursday and Friday of this week, so we won't move these three hosts until next week. I'd prefer if we stick to moving items on Monday and Tuesday, leaving Wednesday as a potential 'fix it' day in case anything migrates and has issues. WMF Holidays for next week's Thursday and Friday: 2015-11-2[78].

@Marostegui: In irc you mentioned pc101[56] but pc1015 is moved, I suspect it was just a typo. If you could review and depool any or all of pc101[678], then myself and either John or Valerie will get them migrated to the new switch stack on Monday or Tuesday (whichever day you prefer). I'm not sure how many pc hosts can depool in a given day, so if we have to carry any of these over to the week of December 1st, then we shall.

Please let us know if a move on 2025-11-24 or 2025-11-25 is possible and for which hosts, thank you!

If you want to, I'll be around Thursday and Friday of this week and I can depool them for you. I can also do the 10G switch too (but that comes later I think?). When he is around, he'll be the responsible person but I can do it to free him a bit from work. Note that PC hosts must be depooled serially (so depool a section, move them, etc., repool, and then depool the next one). Just ping me in IRC tomorrow or the day after and we can get it done.

@RobH as @Ladsgroup mentions, pc* hosts can only be done one at the time. I am out half today and Friday as oncall compensation. If @Ladsgroup is around you could coordinate with him to get those three moved this week (yes, pc1015 was a typo).
Pending hosts could be done:

pc1016 - Thursday with @Ladsgroup
pc1017 - Friday with @Ladsgroup
pc1018 - Monday with @Marostegui

Let us know if this would work.

Update:

@Ladsgroup had other things going on and wasn't able to do this today but did link me to the directions on how to depool: https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting

However, the command I ran seems to not understand where to auto failover the server:

robh@cumin2002:~$ sudo cookbook sre.mysql.parsercache -t T405942 -r portmigration pc1016 depool --downtime-hours 1
Acquired lock for key /spicerack/locks/cookbooks/sre.mysql.parsercache: {'concurrency': 20, 'created': '2025-11-20 17:58:30.682902', 'owner': 'robh@cumin2002 [1242476]', 'ttl': 1800}
START - Cookbook sre.mysql.parsercache
Exception raised while executing cookbook sre.mysql.parsercache:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 61, in execute
    hosts = self._query_default_backend(query_string)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 97, in _query_default_backend
    return query.execute(query_string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
    self._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/puppetdb.py", line 309, in _build
    super()._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 76, in _build
    parsed = self.grammar.parseString(query_string.strip(), parseAll=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected end of text, found ':'  (at char 1), (line:1, col:2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 364, in query
    hosts = query.Query(self._config).execute(query_string)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 64, in execute
    hosts = super().execute(query_string)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
    self._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 111, in _build
    super()._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 79, in _build
    self._parse_token(token)
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 116, in _parse_token
    if self._replace_alias(token_dict):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 156, in _replace_alias
    raise InvalidQueryError("Unable to find alias replacement for '{alias}' in the configuration".format(
cumin.backends.InvalidQueryError: Unable to find alias replacement for 'db-section-pc1016' in the configuration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 19, in run
    return self._run(self.args, self.spicerack)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/mysql/parsercache.py", line 179, in run
    hosts: RemoteHosts = spicerack.remote().query(query)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 366, in query
    raise RemoteError("Failed to execute Cumin query") from e
spicerack.remote.RemoteError: Failed to execute Cumin query
Released lock for key /spicerack/locks/cookbooks/sre.mysql.parsercache: {'concurrency': 20, 'created': '2025-11-20 17:58:30.682902', 'owner': 'robh@cumin2002 [1242476]', 'ttl': 1800}
END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99)
robh@cumin2002:~$

The main error seems to be: cumin.backends.InvalidQueryError: Unable to find alias replacement for 'db-section-pc1016' in the configuration

So we'll need someone on your team to review, correct my command mistakes if any, or explain whats happening.

Suggested schedule:

pc1016 - Friday
pc1017 - Monday
pc1018 - Monday

Would this work to clear up the last 3 Data-Persistence hosts for migration?

@RobH I'm out and not near a keyboard but you have to replace pc1016 with pc6

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:03:28Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool pc6 (T405942)', diff saved to https://phabricator.wikimedia.org/P85435 and previous config saved to /var/cache/conftool/dbconfig/20251121-140327-ladsgroup.json

pc1016 has been moved with @Ladsgroup Thanks for your help this morning

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:09:04Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repool pc6 (T405942)', diff saved to https://phabricator.wikimedia.org/P85436 and previous config saved to /var/cache/conftool/dbconfig/20251121-140903-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:13:45Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool pc7 (T405942)', diff saved to https://phabricator.wikimedia.org/P85437 and previous config saved to /var/cache/conftool/dbconfig/20251121-141345-ladsgroup.json

Since the depool time was quite short, the latency immediately recovered so we are moving forward to pc7 and pc8 too.

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:17:47Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repool pc7 (T405942)', diff saved to https://phabricator.wikimedia.org/P85438 and previous config saved to /var/cache/conftool/dbconfig/20251121-141747-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:21:00Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool pc8 (T405942)', diff saved to https://phabricator.wikimedia.org/P85439 and previous config saved to /var/cache/conftool/dbconfig/20251121-142059-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2025-11-21T14:25:01Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repool pc8 (T405942)', diff saved to https://phabricator.wikimedia.org/P85440 and previous config saved to /var/cache/conftool/dbconfig/20251121-142500-ladsgroup.json

https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=2025-11-21T13:37:29.200Z&to=2025-11-21T14:28:00.361Z&timezone=utc&var-site=$__all&var-deployment=mw-web&var-method=GET&var-code=200&var-handler=php&var-service=mediawiki&refresh=1m&viewPanel=panel-9