Thanks for the heads-up!

@Raine this deletion can happen anytime or you still have to do things on your end?

Right, given that it is everywhere, I'll simply use sX.dblist and issue a DROP TABLE IF EXISTS categorylinks_icu72

Rebuilding:

				Raw Size: 1.746 TB [0xdf8fe2b0 Sectors]
				Firmware state: =====> Rebuild <=====

Thank you!

In T409557#11833394, @fnegri wrote:

@Marostegui for s6, let's proceed with announcing a 24h maintenance window. Let me know when you have time to do it and I'll send out the announcement.

In T419635#11832765, @FCeratto-WMF wrote:

The run on s5 had an error that needs further investigation: https://phabricator.wikimedia.org/P90336 .

I think it may take around 5-6 hours. But to be on the safe side in case of issues, I'd suggest we announce a possible 24h times degradation stating that it is likely to take way less.
I'd also be doing one at the time, so we reduce the impact to one section at the time.
Let me know what you think

Any used disk that we can use for this host?
Thanks!

I can do that but I'll have to stop that host to clone the new one, would that be okay for the service?

This is due to T419635 and should be fixed with T422459

In T423151#11828091, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts: clouddb1019.eqiad.wmnet

• clouddb1019.eqiad.wmnet (FAIL)
  • Host not found on Icinga, unable to downtime it
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Failed to power off, manual intervention required: Remote IPMI for clouddb1019.mgmt.eqiad.wmnet failed (exit=1): b''
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

ERROR: some step on some host failed, check the bolded items above

This is ready for DC-Ops

This has been fixed upstream and will be released on 10.11.17 https://jira.mariadb.org/browse/MDEV-38072
https://github.com/MariaDB/server/pull/4928

In T327300#11817241, @Ladsgroup wrote:

Databases now have a centralized depool and repool cookbook that encapsulates all the different ways you need to depool and repool db hosts (for different clusters). I wonder we can integrate that. I dream of a day that netops or dcops depool and repool most db hosts and do maintenance without even needing to notify DBAs (with exception of master of core clusters but that's like 20 hosts in total). What do you think @Marostegui ?

In T422777#11817870, @Personal.luckymail2002 wrote:

$ export HTTP_PROXY=http://proxy_user_id:proxy_password@proxy_server:proxy_port
$ export HTTPS_PROXY=http://proxy_user_id:proxy_password@proxy_server:proxy_port

Thanks! If you are happy with it, we can close this task.

sounds good from my side yes!

Thanks John for trying swapping many parts - unfortunately it didn't work so I am going to close this task and open a new one to decommission this host.
This host is dead.

Wilco! Thanks!

Blocked on finishing dborch1003 and the completion of this task: T317179

@ssingh we'd need support from your team to move dborch1003 behind the CDN - is it possible to allocate some time for this with @FCeratto-WMF? Thanks!

@jcrespo how do you feel about getting db2160 and db1217 migrated to Trixie? MariaDB version won't change.
The issue is that these hosts have the backups for the rest of sections (which are not yet scheduled to get migrated)

Thank you - let me know if I can help

Great!

I'm on call next week so we can force a lag page

@FCeratto-WMF has the parent's schema change progressed in any section that'd require recreating the views here?

@Ladsgroup what's the status of this? Should I start scheduling switchovers?

In T415165#11804490, @fnegri wrote:

Change of plans, @Marostegui will reimage clouddb1019 instead, as that one needs to be reimaged anyway because of a hardware issue: T422813: clouddb1019 down.

@Jclark-ctr I am not able to reimage the host, it is not rebooting, can you check onsite what's on the screen? I've tried several times to reboot it manually but there's no output at all.

Thanks John - let me reimage it now

Thank you @Jclark-ctr - let us know when we can take over.
Thankfully its replacement will arrive soon (famous last words) (T405296)

ops-eqiad can you check on site? The above errors seem HW related.

We can probably start by running a pt-show-grants on each host, compare them and make sure they are all the same. Once done that, we can start fixing redundant stuff and escaping things.
One thing I've always noticed was:
GRANT ALL PRIVILEGES ON '%\_p'.* already covers every _p database, making these three specific ones redundant:

• heartbeat\_p — subsumed by %\_p
• meta\_p — subsumed by %\_p
• centralauth\_p — subsumed by %\_p
• %wik%\_p — also subsumed by %\_p

This is basically all done and waiting for the next HW to arrive (see subtasks) to get installed directly as Trixie

That works for me @jcrespo - thank you.
Also keep in mind those hosts aren't blocking us in anyway as the version is the same.

Thanks @fnegri all clear now. I will check the task and the patch and see if I can think of something.
Thank you!

Not sure what this is for now, so protecting

If possible clouddb1015 would be nice T422777

@fnegri can we resume this and reimage one host with Trixie?

I can take a look from the DB side of things, but remember that being active-active we make no changes at all on the DB layer during switchovers (apart from circular replication, which only affects master for a few days).

Done and repooled

I think this is fixed. The new patched version (10.11.16u3) has been uploaded to bookworm and trixie repos. For more tracking of the bug itself: https://jira.mariadb.org/browse/MDEV-39209
@fnegri you may continue your updates

In T396816#11793443, @FCeratto-WMF wrote:

It's been a month and there has been no warnings from it the replication lag monitoring. We can discuss the next steps in the next team meeting.

Works for me!