Hi. Do we have an idea for when these hosts could be available? We have ongoing issues with parsercache (see T282761) that we hope moving to the new HW will partially mitigate.

Notes from the meeting today with @Kormat and @Krinkle

• Code changes to be made:
  • Recording number of iterations since the last percentage print (purge script feature to be added).
  • Running on multiple servers (adding a --server parameter) requires changes to both the maintenance function in SqlBagOStuff and the script.
• Sleep interval deduction (making sleep variable instead of static) requires further data gathering.
• Filtering deletes from replication and running on both DCs would result in spare PC not getting purged but this won't be a problem once the script is host-aware.
• We can test the new hosts, they would have to be introduced as replicas first.

>>>! In T282761#7169546, @LSobanski wrote:

Do you happen to know if there is a specific reason to do the deletes in the above way?

It's hard to be sure since the code is so old, but based on today's best practices I would retroactively say that for production code our policy is generally to only perform writes or deletes by primary key. This is for performance, but also to be determistic and safe under various other conditions. I suppose it also helps with row-based replication. I understand parser caches could be exempt from this, as they are special in various ways, but from a code perspective we currently treat it like we treat anything else in production.

Does this mean it's not an option or can we consider an exception for parser cache? It looks like a promising way forward.

@Krinkle @aaron do you happen to know if there is a specific reason to do the deletes in the above way?

We're back to 67 hours today.

Thanks, let us know when the database is created, so we can sanitize it.

@Marostegui @Kormat @jcrespo I took the liberty of subscribing you as it seems like the easiest way to keep track of documentation changes in one place.

https://wikitech.wikimedia.org/wiki/SRE/Data_Persistence/Documentation_guidelines documents the approach I've been taking so far.

MariaDB start and stop section is now a separate page: https://wikitech.wikimedia.org/wiki/MariaDB/Start_and_stop

Thanks, let us know when the database is created, so we can sanitize it.

In T275784#7133984, @Marostegui wrote:

https://github.com/openark/orchestrator/releases/tag/v3.2.5

This version does include our patch.
I think we've never upgraded orchestrator since it was installed, it would be a good practice to upgrade and document the upgrade process.

Thanks, let us know when the database is created, so we can sanitize it.

I updated the Wikitech instructions page to clarify the requirements (diff here: https://wikitech.wikimedia.org/w/index.php?title=Creating_new_tables&type=revision&diff=1913898&oldid=1913356). @DLynch I would appreciate a comment on whether it's clear enough.

A stub document capturing this is at https://wikitech.wikimedia.org/wiki/MariaDB/Rebooting_a_host.

For the "metadata dashboard" usage, something like https://github.com/nocodb/nocodb could be interesting to evaluate.

I think we're all ok with the current state, resolving.

This turned into a much bigger scope than my original intention but it's a good discussion to have. I am now thinking that my ask was not well defined and needs to be adjusted, especially given my dislike of using email for operational notification :)

Assigning to Stevie to confirm if this can go into Ready.

The remaining work may be addressed by T283580

@Kormat How do you see the urgency of this (and the probability we'll work on it anytime soon)?

Assigning to Stevie to confirm if this can go into Ready.

In T252528#7100826, @Kormat wrote:

This does mean that pt-heartbeat-wikimedia needs to be started manually after a boot, however.

Blocked at least until we get a clear picture from T283261.

My understanding based on recent conversations was that a decision is yet to be made about what approach to take with HDFS backup / redundancy. Is that still the case and is this a discovery task or do we have an answer and this is an implementation task? I'm trying to figure out what is the expectation of involvement from our side.

Having seen this in a while and it's not super clear what the original problem was. Resolving.

Timing wise, this should happen before DC switchover happens (likely the week of June 21st) as we'll have our hands full during that time. This makes things tricky as the three weeks before that date happen to be the All Hands and the two Percona trainings. In other words, if the host can be ready for data transfer bu mid next week we're significantly limiting the risk of it having issues,.

Bonus question, is there an option for some traffic shaping / QoS to remediate the above automatically?