Stalling until db1164 is back in service: T307198: db1164 fails to POST/boot/etc

I've set the host to 'failed' in netbox: https://netbox.wikimedia.org/dcim/devices/2999/

On the web console, i can see it get as far as this before resetting:

Success.

In T303603#7841068, @Milimetric wrote:

As @Zabe says, T233004 is stalled on this. Is there anything I can help with? This might be more of a "teach Dan how to do this" slowdown than a help, but gets you an extra person to help with these kinds of changes in the future? Anyway, I'm game.

The script dropped the connection after 2h15m.

As an example, i depooled a host (db1120) so i could reboot it. 1.5h later, i'm still waiting for refreshLinkRecommendations to drop it's connection so that i can actually do the reboot. I have no way of knowing when that will happen. I also don't know how bad/disruptive/etc it would be to just reboot it anyway.

In T305469#7843940, @Marostegui wrote:

For what is worth, es2029 and es2030 are scheduled to be done 14th, which is a bank holiday for me, so someone else would need to bring mysql up once that is done.

In T301315#7831250, @jcrespo wrote:

The only thing left is to check (by DBAs) that the backup available at (e.g.) : dbprov1002:/srv/backups/dumps/latest/dump.db_inventory.2022-04-05--03-13-31 looks as expected (content-wise).

In T301315#7821645, @jcrespo wrote:

One last thing- does orchestrator + zarcillo server have a name? Backups (jobs) have a completely arbitrary identifier for monitoring purposes- any suggestion (even if it is orchestrator OR zarcillo containing both)? Alternatively, it is possible to setup separate backup jobs, even for the same server.

In T301315#7821493, @jcrespo wrote:

@Kormat I can help setting up backups, but someone has to make clear first what to backup (what databases?), from which servers/dcs, and how frequently they are needed.

@jcrespo: Can i hand this task over to you? I think the dumps grants probably need some cleanup, if you can have a look at that, too.

In T301315#7811237, @Marostegui wrote:

Regarding the plan I would include the following steps before dropping the database in db2093:

• Check current grants on db2093 to make sure they are the same on db1115

In T276589#7756918, @Kormat wrote:

Granted removed:

Granted removed:

• es1
• es2
• es3
• es4
• es5
• m1
• m2
• m3
• m5
• s1
• s2
• s3
• s4
• s5
• s6
• s7
• s8
• db_inventory
• x1
• x2
• pc1
• pc2
• pc3
• mediabackupstemp
• A:db-core-test

I'm afraid we have to cancel this maintenance for now. Hopefully we'll be able to do in the next couple of weeks, but i'll open a new task for that when i have a firm date. Sorry for the hassle <3

In T276589#7756067, @Marostegui wrote:

In T276589#7755980, @gerritbot wrote:

Change 768657 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin2001 from mysql root clients and related grants

https://gerrit.wikimedia.org/r/768657

Apart from merging this, we also need to remove the grants from production (@Kormat would you be able to take care of that?)

In T276589#7741816, @Kormat wrote:

I'm working on the wmfdb + wmfmariadbpy sides of this for data-persistence. wmfdb is good to go, i'm currently doing some testing with wmfmariadbpy in pontoon in preparation for doing a new release in the next day or so.

Deployment complete.

In T276589#7727280, @Joe wrote:

Any update on this? This upgrade is blocking serviceops who needs bullseye for the kubernetes python libraries and cookbooks.