@EChetty Hi from the SLO project! Thanks for this -- Will asked me to have a look at the schema, from the POV of where we're going next with the edit save SLO. Broadly I think this is going to have everything we need, just a couple of clarification questions.

Great, thanks!

Hmm, also: As a group access change, this should be reviewed and approved in the Infrastructure-Foundations team meeting.

We're past the European work day, so I don't expect a response from Lukasz (who's OOO) or Alex before Monday. I'm sure next week's SRE clinician will pick this up once it's signed off.

In T308308#7926592, @elukey wrote:

I may have created this task too soon, some discussion on T305729 is still happening, let's wait before proceeding.

Hi @Dmantena, you should be all set now:

This should be all set!

Added to ldap/nda:

Thanks both! Proceeding.

After consulting with SRE colleagues, I stand corrected -- the email address on the account is fine, and we'll just use the wikimedia.org address in our own records. Going ahead!

@RoccoMo Hi from the SRE team! Thanks for the request, we'll get you sorted out shortly.

I think she already has a wikitech account: https://wikitech.wikimedia.org/wiki/User:Esther_Akinloose

Thanks both @Joe and @thcipriani for the ping -- agreed clinic duty is as good a route for this as any.

You're both in the wmf group already, so nothing to do there:

Just picking up SRE clinic duty for the week -- I'm so sorry this has been sitting for so long! I'll ask around and try to find out what happened here.

Grabbing this from @jhathaway as I've taken over SRE clinic duty for this week. This is actually the right template for the use case, since Superset access is via LDAP, not SSH (even though PII access is controlled with a posix group). We can go right ahead with it.

Yeah -- I can do the implementation but I'm not sure if we've settled on what we want it to look like.

@MoritzMuehlenhoff Checking in -- have you had any time to take a look at this?

Draft incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-03-31_api_errors

Another way I'd like to improve this is to deal with Puppet skew on the two hosts.

Found another example of this, in case the extra data helps -- thanks @MSantos and @ECohen_WMDE for pointing me to the right task. (Moved here from T228612.)

In T228612#7809798, @ECohen_WMDE wrote:

@RLazarus - I feel like your screenshots are actually an example of the bug in this task T228812: Place names get cut off and unreadable on tile boundary

Hmm, the 1.21.1 build didn't work out of the box. Running build-envoy-deb buster future got me this:

As in T300324#7752134, I've rolled out all the k8s services where Envoy version was the only diff. We're now up to 1.18 everywhere, except for k8s services with other undeployed changes, and I'll follow up with those at the end.

From the time sliders it looks like the issue is that all or part of the pad gets deleted and replaced by a character, at these revisions respectively:

Oh, yep, it's strip_matching_host_port in the HTTP connection manager: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto

I just upgraded thanos-fe to envoy 1.18.3, but out of the box I see the same behavior:

1.15.4 is still running in a few places on k8s -- after bumping the default version, I rolled out all services where that was the only diff. Some services had some undeployed changes from who knows how long ago, so I left them untouched (T265979 for that problem in general).

To take a step back, the varnish slo dashboard linked in the description didn't actually originate from a template. Presumably this one was a manual fork of the original etcd slo example dashboard that's been manually adjusted.

Thanks for letting us know! We did indeed have this issue again for a few minutes earlier (intermittently between 02:36 and 03:00 UTC) but things are back to normal now. Sorry for the inconvenience, and more permanent solutions are in progress to keep this from happening again.

This came up again in T301507.

Done, thanks!

Found another example of this, in case the extra data helps -- thanks @MSantos for pointing me to the right task.