This is an amazing proof-of-concept, thanks so much @TK-999 !!!

I checked my shell history on deploy1002 and all I've done there recently is scap backport 1026628.

Unfortunately subdivision-level mapping didn't help in PE -- there are many regions where magru is both better and worse than eqiad.

magru is a clear win for:
UY, CL, AR, BR, PY

Oh, and I think magru is a win for SV as well.

In T356412#9766840, @MatthewVernon wrote:

I think I have two questions:

1. Where is it defined what should and shouldn't get its own intermediate? (e.g. I see cassandra has one)

magru is a clear win for:
UY, CL, AR, BR, PY

That sounds good to me @elukey . I don't think a new intermediate is needed.

FYI this happened for me again, despite the above patch

19:48:44 /usr/bin/sudo /usr/local/sbin/mediawiki-image-download 2024-05-02-194555-publish (ran as mwdeploy@mw2382.codfw.wmnet) returned [255]: ssh: connect to host mw2382.codfw.wmnet port 22: Connection timed out

+1, omit_replicas_in_mwconfig seems like the right way to begin implementing this.

In T363407#9752049, @JMeybohm wrote:

Ok, understood. The only thing I'm really worrying about is that metrics change/get less intuitive with this. For example in here it's pretty clear what the filter means (selecting "local_service"). I think we will loose clarity here if local_service changes to mw-web.eqiad.main. Maybe adding local as suffix/prefix would help here (and you could strip that out again in OTTL?

I like the idea! A few questions

BTW in case it was not clear, my intentions here are basically:

• deploy something ASAP (like next week) that everyone is reasonably happy with for the interim
• don't do anything to get in the way of the badly-needed Envoy upgrade
• don't break anything else

In T363407#9743785, @JMeybohm wrote:

Thanks for the write-up!
What is not very clear to me is what part of the work would need to be done anyways (in case we'd have a envoy version >= 1.24). The reason I'm asking this is that envoy 1.23 is EOL since a year or so, so we need to look at an upgrade anyways.

Anyway I think that all that is needed to unblock VLAN migrations has been done or documented on this ticket? Optimistically closing but please re-open if you disagree.

In T360029#9725627, @Volans wrote:

As for the commit I advocate to add dbctl support in Spicerack but IIRC that requires changes in dbctl as most of its logic is in its CLI part and not exposed as a library, but to be checked.

I largely agree with Arzhel's assessment. At a cursory glance, Uruguay or Paraguay look ideal as first candidates.

I think you should be able to use the existing spicerack interface to confctl to do the set/host_ip=... action -- that should be equivalent to a ConftoolEntity.update call.

@Marostegui As it turns out, plain old confctl can be used to do this already.

In T360029#9722005, @Ladsgroup wrote:

In T360029#9658042, @CDanis wrote:

Actually the idea is that dbctl should not contain the IPs at all. It should look up the IP via DNS, we should store FQDN instead.

This has been fixed with this patch, which I forgot to associate with this bug.

Just to make sure I understand, the request here is an easy-to-automate way of dbctl to change the instance IP address?

Should this ticket really be "deprecate cergen"? :)

This would best be fixed by extending the haproxy bwlim work done in T317799 -- we've talked about having per-ASN limits in addition to the existing and partially-deployed per-file-URI limits.

Sent upstream as https://github.com/jaegertracing/helm-charts/pull/541

As it turns out, this required a change to the upstream chart:

I've verified that oauth2-proxy will silently just serve plain HTTP if you specify https_address but don't provide it with TLS key material. So I think I've provided it with such in this patch?

All pods on k8s-aux-eqiad restarted, thanks @akosiaris for the script.

Per docs, Thanos supports logging when a query is received but before it begins execution:

@Fabfur just wanted to make sure you've seen this task, it is decent documentation of the existing mechanism and probably helpful for doing T353910

The script was added to the wmf-sre-laptop package in May 2023 with this commit