sounds good, thanks @Volans

@Volans I think there is a possibility that the puppet 7 upgrade fixed this issue, I am unable to find any puppet 7 hosts with similar failures.

In my experience commit messages are more frequently read via git log and friends then in the browser, so I would rather optimize for that view and also keep them compatible with git interpret-trailers. One of the benefits of opensource is of course contributing upstream. I would expect us to be willing to contribute bug reports, feature request, and even code to the primary tool which we use to manage our source code. Also git trailers themselves are pretty ubiquitous, so I would assume gitlab would be interested in improving their use with their product.

apologies for the breakage, this is caused by moving away from the rsync frags in /etc/rsync.d, to concat, would it be possible to move the secrets to a different path, pehaps, /etc/rsync-doc-auth-secrets then update the secrets_file param to point to the new path?

Thanks for reopening @jbond I'll take a look at those.

In my testing this has been resolved by @fgiunchedi's patch

Updated link, https://puppet.atlassian.net/browse/PUP-6631

I think this can be resolved, the only deprecated option we are using at present is:

The rsync one is probably still an issue, since I don't use the rsync profile for the dcl puppetserver.

I took a brief look around to try to understand what other folks are doing, but documentation is surprisingly sparse.

The crun bug with linux kernel 6.1.55 has been patched in 1.8.1-1+deb12u1 and tested, thanks @faidon!

Shellcheck no longer reports any POSIX violations in our repo, which is equivalent to, but more robust than the checkbashisms command:

https://github.com/goccy/go-yaml project does a much better job of retaining formatting when decoding yaml documents. I am going to investigate reworking my formatting tool to use this library and compare the results with https://github.com/go-yaml/yaml

Package is now in proposed updates, https://www.debian.org/releases/proposed-updates

Great news, @faidon uploaded a patched version of 1.8.1 so this may be fixed on bookworm soon, https://bugs.debian.org/1055241

In T350302#9302409, @MatthewVernon wrote:

because my laptop is running sysvinit as PID 1, not systemd.

In T350302#9302352, @MatthewVernon wrote:

[FWIW, my WMF laptop doesn't run systemd-resolved and it would be difficult to make it do so]

@jbond thanks for opening the task. The current setup will break under Puppet 8 as strict_variables are enabled by default. I agree that a simple renaming of realm.pp would break to many profiles, but hopefully we can whittle down realm.pp until it is gone, per your new task T350008. I spent some time investigating what it would take to move to strict_variables and realm.pp is the main initial blocker, so I am happy to help with the effort.

Thanks @Quiddity I'll take a look at those links

Would it be possible to add email headers with an example, with all personal information removed?

@ayounsi thanks for detailed replied and the linked blog posts. Given that additional data, I am substantially less concerned about using MSS clamping.

In T348837#9254127, @BBlack wrote:

One potential issue with relying solely on MSS reduction is that, obviously, it only affects TCP. For now this is fine, as long as we're only using LVS (or future liberica) for TCP traffic (I think that's currently the case for LVS anyways!), but we could add UDP-based things in the future (e.g. DNS and QUIC/HTTP3), at which point we'll have to solve these problems differently.

In T348837#9253425, @cmooney wrote:

Either way, katran/liberica uses IPIP, so having the option for GUE in IPVS doesn't solve that problem if we hit it. I think we can probably stick with IPIP for that reason.

basic support completed, https://gitlab.wikimedia.org/jhathaway/dcl/-/commit/5c7d0babb1fc388580e8478975c49b69c91effed

fixed, in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/c3efc055349c1f8814b414a2584312eecd86a0ca

@Vgutierrez thanks for opening this ticket and investigating ipip support in ipvs. Another alternative would be GUE encapsulation, which is also supported by Katran. Evidently UDP encapsulation may have performance benefits because routers are tuned to support it, the patch for foo over udp which is similar to GUE posted some performance numbers, https://lwn.net/Articles/614433/.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

In T346607#9175200, @LSobanski wrote:

@jhathaway do we have a standard process for adding DKIM? Do you have any other thoughts on this?

In T331699#9136475, @MoritzMuehlenhoff wrote:

One other option would be to simply start with a fresh, parallel setup and skip Bullseye entirely:

In T344607#9114588, @BCornwall wrote:

Not sure if this fits as a new ticket.

I'm happy to be added on the North American side of the pond.

In T343377#9102709, @RLazarus wrote:

In T343377#9102295, @jhathaway wrote:

One issue that I raised, but perhaps was not captured anywhere is adding some guidance to the documentation on how the folks being paged can communicate with person who used Klaxon. For instance should I assume the person using Klaxon is on IRC and their is a channel we can chat in about the incident?

Oops, sorry for missing that. The form does say "include how to best get in touch with you," and it's probably already going to be on the user's mind too -- they're there because they want to hear from you, pretty urgently!

But if they forget, their LDAP email address is automatically included in the page, so you could get hold of them that way. (Depending on how we set this up, "LDAP email address" might be replaced with something else, but your point is a good one, and we should make sure to replace it with some sort of direct contact information, not just a username.)

One issue that I raised, but perhaps was not captured anywhere is adding some guidance to the documentation on how the folks being paged can communicate with person who used Klaxon. For instance should I assume the person using Klaxon is on IRC and their is a channel we can chat in about the incident?

no strong opinion from me either, txt files are such a wild west format that I feel like we will hit this issue with other ones, on the other hand standardizing on .data seems reasonable as well. @bking would using .data be difficult in this specific case?

Ready for initial feedback

manually pushed, kicking automated pushing to version one

https://gitlab.wikimedia.org/jhathaway/dcl/-/blob/main/README.md

https://gitlab.wikimedia.org/jhathaway/dcl/-/commit/8591391340dc7786121a96fe29f8519ba0a982e6

sent note to SRE list, waiting on feedback