Thanks. At present the controller monitors all namespaces, but ignores pods other than in mw-script. So if I were estimating memory usage I'd base it on the total number of pod events in the cluster, not just in the namespace.

That sounds reasonable! Note for the future that helm diff has a --suppress-output-line-regex which does exactly what you'd like it to do, but it's not available in the version we're currently running.

If we're really worried about that race condition, is it plausible to do this?

Clinic duty SRE here -- I/F, can you start investigating this at the MTA end? Triaging this to High in case it's widespread, but feel free to decrease if it turns out it's not.

@AndyRussG Welcome back!

Yep, we're following closely -- but we don't use Debian unstable, so we're not directly affected. Thanks for checking!

Clinic duty SRE here, thanks @karapayneWMDE for the ticket. I merged https://gerrit.wikimedia.org/r/1015995 (thanks @Dzahn!) and followed up with

In T360867#9664888, @Clement_Goubert wrote:

The reason we're catching it now, and not with the regular httpbb checks being done on the cumin nodes is that the scap httpbb checks are being run without --retry_on_timeout.

Curious: As @Clement_Goubert and I discussed, both the directory (via puppet file) and the database file (via puppet exec of imagecatalog init) have the right user, imagecatalog. There's nothing in Puppet (like a recurse) to ensure ownership on the database file, but it still ought to come out correct, as far as I can tell. Claime reports the file was owned by mwbuilder, which runs the release tools, but I think imagecatalog init should still have run first as the imagecatalog user.

FYI: I redid a dry run and live test for 01-stop-maintenance.py after https://gerrit.wikimedia.org/r/1008583 and it's good to go.

This is good to go for the March 2024 switchover, so removing it as a subtask.

I don't see anything obviously hardware-broken in logs. I notice it was just repooled yesterday after maintenance for T352010, but nothing jumps out as an obvious cause. Over to the DBAs from here, enjoy. :)

Thanks @bd808!

That sounds like a perfect solution, except that I'm not in Trusted-Contributors. Which is admittedly pretty funny.

Confirming @jasmine_ is an intern on my team. If she needs any vouching, please consider her vouched! :)

Will do, thanks for the pointer.

Restarted mailman3 at 00:43, icinga alerts are cleared, and the graph in T358020#9565952 is trending down again. Thanks @JJMC89 for the report and thanks @Legoktm for the ping.

the intention was probably for this to match something a bit more restrictive (e.g., matching ^/wiki(/.*)?$)

Sorry yeah, I was using the term broadly. The goal is to edit the Apache config, but that hieradata file is how you'd do it. :)

Hi from Service Ops SRE!

Surfacing @JMeybohm's reasonable concern from https://gerrit.wikimedia.org/r/c/988851/comments/3827b6cd_15427748:

This week's clinic duty SRE is @Arnoldokoth.

@Joe @JMeybohm That's a lot of code review at once, across two tasks -- I posted it all for context, but no expectation you'll have time to look at all of it immediately. (It's all tested together from my homedir on deploy2002, and works.) Here's the suggested reviewing order:

Okay, let me know if https://gerrit.wikimedia.org/r/983963 plus the most recent iteration of https://gitlab.wikimedia.org/repos/sre/k8s-controller-sidecars/-/merge_requests/1 is what you had in mind...

Oh, I misunderstood what you meant by "enable the controller on a per namespace level" above! I thought deploying one instance per namespace was what you had in mind.

Yeah, as foretold:

In T348284#9395546, @Joe wrote:

I reckon this technique will be useful for all charts that need a Job object.

Super helpful explanation, thank you! https://gerrit.wikimedia.org/r/981703 should do the above, and https://gerrit.wikimedia.org/r/981704 adds the binding for the mw-script namespace. I'll deploy those like wikitech:Kubernetes/Add_a_new_service#Deploy_changes_to_helmfile.d/admin_ng and that will let me finish experimenting with the helm charts for both the sidecar controller and the actual jobs.

@JMeybohm Can you help with an RBAC issue?

Yep, sounds like a similar fit.

Doh. Okay, that's a good reason not to go that route. :) I'll give the "primary container" label a try, thanks.

Both good points, thanks.

Done:

• Added you to the wmf LDAP group.
• Added you to the WMF-NDA Phabricator project.
• Created your shell user ahoelzl and added it to the analytics-privatedata-users POSIX group.
• Created your Kerberos principal.

It seems like this fell through the cracks between last week's SRE clinic duty (mine) and this week's. Let me finish it up for you, thanks for your patience.

This sounds right to me -- thanks @elukey for getting it rolling. Early on, we had talked about autogenerating links for different calendar quarters and adding them to the text panel on top, but my recollection is we decided to spend that energy on Pyrra instead.

I don't have edit access to acl*security.

@thcipriani Sorry for the back-and-forth, but just because it isn't 100% explicit from reading this task -- did you want @Mabualruz to get deployer training before being added to the group? Or do we have your approval to add him, so that he can do the training hands-on?

In T345959#9170888, @Ahoelzl wrote:

@RLazarus Here is the new production (non-shared) public key:
[...]

We have some plans for SLO-based alerting in the pipeline, but nothing implemented yet.

No, I tagged it private when we asked for PII, so that it would already be private when that stuff was posted. Since it never appeared, I'm fine with opening it up.

Hi @Ahoelzl, welcome to the Foundation! SRE here, I'll be able to set you up with production access.

Hi @joanna_borun -- does this need Infrastructure Foundations approval?

In T345868#9151520, @Clement_Goubert wrote:

I propose using a _shellbox_common_ directory like we have a _aqs2-common_ and a _mediawiki-common_ directory in helmfile.d/services and symlink from there.

In T343377#9105446, @SLyngshede-WMF wrote:

If it's just a matter of managing a LDAP group, then that's perfectly within scope of the IDM. It's one of the features that already have a first-attempt implementation.

In T343377#9102852, @jhathaway wrote:

Email doesn't seem like a great way to communicate for page worthy incidents, would it be possible to insist on IRC and have them input their handle as part of the Klaxon flow, since we already recommend they chat with us in #wikimedia-sre?

In T343377#9102295, @jhathaway wrote:

One issue that I raised, but perhaps was not captured anywhere is adding some guidance to the documentation on how the folks being paged can communicate with person who used Klaxon. For instance should I assume the person using Klaxon is on IRC and their is a channel we can chat in about the incident?

Only two blockers were raised at the August 7 SRE meeting:

Those numbers don't immediately raise alarm bells for me -- "storage" doesn't mean anything persistent, only ephemeral data that can disappear when the script exits, right? As long as that's the case (and assuming you're using ~1 CPU), you should be fine. I'm tagging in @akosiaris to confirm the resource request is sensible.