thanks for the overview @Volans, very helpful

We worked around this by using the ip, so closing.

Yeah, I agree with @Eevans that there is value in iterating on the pieces we can test in staging.

Sampling schedule looks good to me.

Review is complete, everything is satisfied for the technical pilot. There are still outstanding items for the soft launch phase, which I marked as "not done". I did not review the production pieces.

never mind, I see the page now, https://wikitech.wikimedia.org/wiki/EditCheckHeadless

I would prefer at least a stub page on wikitech, that has a link to the scroll, specifying why the page is just a stub

If we are comfortable with only HMAC signing for k8s, that I would be comfortable with only HMAC signing for our servers.

I agree with your suggestion to move signing to pki2002.codfw.wmnet. That alleviates the time pressure and noise during the operation.

Debian's Rsyslog now has support for open telemetry, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1130542

In T431285#12296583, @DMburugu wrote:

In the observability section, I've filled in most of the SLO, Monitoring and Alerting sections as N/A for now since we're not yet on any production wiki. We plan on setting this up once the extension is running on a couple of pilot wikis since we'll have a better sense of what those sections will entail.

In T431285#12296561, @DMburugu wrote:

I've marked N/A under the release confidence section of the scroll since the extension rollout will be similar to PersonalDashboard which is already in production. Let me know if I should have used "Done" instead.

In T431285#12275748, @DMburugu wrote:

Hey @jhathaway, I'm filling the Service Summary section and wanted to ask if it's okay to mark "Is your service or component present on the Service Catalogue?" as done when I add it to the Google Sheet instead of the wikitech page which doesn't exist at this time.

In T353511#12276334, @elukey wrote:

Hey folks! I just noticed this task, I don't mean to stop the work that is already half way through but please next time reach out to the I/F team before such a big rollout. Given the fact that memcached clients and servers may get the same PKI certs, it would have been nice to think about a separate security realm like a memcached intermediate. Not at all a blocker, but we'd need to be mindful in adding too may things to discovery in the future :)

I am indeed the scroll bearer, please let me know how I can help

I booted the box into SystemRescue running 12.02 running Linux kernel 6.12.43-1-lts, and I can't detect any performance issues, I ran stress and stress-ng and they both returned nominal results. I did not test the NIC since all debugging was done over the serial console. I ran ipmitool sel list which returned the same list of errors as "Health Event Log" from the BMC web interface. I rebooted and tested again and obtained the same results. The system is still running on the SystemRescue image if folks what to run any tests over the serial console. Perhaps there was a regression with the kernel update? Would the re-image use the exact same kernel?

the edits were correct, so I re-ran with the force flag.

I went ahead and merged the patch, without the config enforcement in place. @LSobanski and I agreed that it was better to reduce spam now and cut a separate task for the config enforcement.

patch out for review: https://gerrit.wikimedia.org/r/c/operations/dns/+/1319154

In T383047#12165505, @Scott_French wrote:

Very nice find, @jhathaway. Reverting the to the original 1s timeout (leaving ability to override it intact) sounds reasonable to me, though not a strong opinion. Depending on how things go tomorrow, I may have time to do that.

In T383047#12165518, @Perryprog wrote: Out of curiosity, how does the byte threshold come into play with this?

The 4s delay was do to Rspamd’s fuzzy_check module timing out when contacting its configuration servers. We did not intend for fuzzy_check to check email from trusted networks. Unlike other Rspamd modules it does not skip checking based on local_addrs or the email being marked as accept in a prefilter. This module has a separate IP whitelist field, which is now being used to disable checking.

There is some very strange behavior with regard to the message size, if I send a message of 343 bytes or less, there is no delay in queuing the message, however if the message size is 344 bytes or larger, there is about a four second delay:

since we went the route of rewriting From headers:

In T431726#12109228, @Esanders wrote:

I'm also now looking into using lightpanda.io. Early experiments are showing a 80-90% drop in CPU/memory usage which is very promising.

This issue arose again at 2026-07-10 03:00 UTC, https://grafana.wikimedia.org/goto/bfrpyuspzo3r4a?orgId=default

while investigating whether this task could be resolved, I found...

I think that makes sense, "Internal error!" is the helpful message that the args to the script were wrong. However, the whole dual python and shell script setup is a mess. Touching the file directly does have a small risk of breaking, if the script changes, but I think that is acceptable.

@MoritzMuehlenhoff what do you think of the patch? Or do you want to find a way to retain the colors?

The patch sounds great, let's take that route. There's no need for colour output for the command to disable Puppet merges.

In T429129#12030077, @Volans wrote:

My understanding is that it's set to dumb when not in a PTY:

mylaptop $ ssh cumin1003.eqiad.wmnet 'echo $TERM'
dumb
mylaptop $ ssh -t cumin1003.eqiad.wmnet 'echo $TERM'
xterm-256color
Connection to cumin1003.eqiad.wmnet closed.

In T429129#12029827, @Volans wrote:

One quick fix should be to just remove sudo AFAIUI.
That said, if in the longer term we still aim to go towards unprivileged cumin that will use sudo when needed this is probably something that needs a more proper general fix.

@MoritzMuehlenhoff what do you think of the patch? Or do you want to find a way to retain the colors?

Since we don't have a PTY, I would disable colors. Patch sent!

In T426180#11948244, @elukey wrote:

What is your idea about the last point? Do you want to change the wmfroot password to be different from the current root's?

@Dzahn and @A_smart_kitten, good points, I think this is a policy question for WMCS. Two basic options are:

Thanks @elukey, given this task is necessary, due to the upstream change in OpenBMC to reject the use of the root user, I think we should try to obtain some added benefits out of this work. Namely reduction and auditing of our machine accounts and automated password rolling. Perhaps altering the plan to be?:

The issue is that on mx-in{1001,2001}.wikimedia.org we have wmcloud.org setup as a relay domain in postfix, but at present we have no relay recipients configured. So even though phabricator-no-reply@wmcloud.org is the sender, Postfix is rejecting the mail since it is not a valid recipient for that domain. I can add a recipient map and add the test phabricator address?

Someone from Yahoo was kind enough to reach out to me directly and modify the IP reputation, so emails are flowing again!

All mail to yahoo.com is currently being deferred, the current message is:

@MoritzMuehlenhoff you are correct, at present it is used only in development environments. Happy to fix however, but I'm a bit confused as this call is inside a profile, so does it need to change?

In T423282#11844960, @MoritzMuehlenhoff wrote:

Poking at this further I also noticed one other discrepancy actually: For some reason puppetserver1002 has the jdk variant of OpenJDK installed. This predates the current retention time of the apt logs on that server, so this is probably something that has been like that since the first setup of that host (debdeploy operates on source packages, so it simply upgrades what it installed on every host). A fresh installation only installs that flavour, so I suppose this was doen manually at some point by John when debugging something.

@MoritzMuehlenhoff I tried to reproduce the issue on Friday afternoon, but I was unable to trigger it with simulated loads via cumin. I ratcheted up the concurrency, but once I went over around 25 simultaneous puppet runs, the throughput decreased significantly. However, even when the runs were taking a very long time no similar errors appeared in the logs. Perhaps we need to do more live debugging on Monday.

Fix has been applied and rolled out to both krb hosts. After rebooting krb2002 the sysctl for net.netfilter.nf_conntrack_max was properly updated.

ah, makes sense, thanks for the context

plan looks good, I have a mild preference for 1003 to be the primary, as that would match my expectations.

@MLechvien-WMF you can find the original rationale in this merge request. The idea was that the incident phabricator task would serve as an umbrella task over any incident follow-ups. Our intention was that a human would resolve the task once the follow-ups had been completed or declined. From a brief look at the phabricator library we use, gonduit, it does appear to support moving between workboard columns, so that may be an option.

I don't think buffering the output is always wanted, as you may want to see the first error as quickly as possible, so declining.

@Peachey88 awesome, thanks

instances have been deleted.

added, a few alternatives, after discussing with other folks, happy to hear of others.

In T419906#11720988, @taavi wrote:

In T419906#11713781, @jhathaway wrote:

good catch, should be that a developer account is required, amending

This is still not correct, access to Logstash requires membership in the nda, logstash-access or ops groups (see T376790).

In T419906#11710005, @taavi wrote:

If this data is too sensitive for our normal Logstash instance, why are we fine shipping them to a yet another third party?

Okay, the posting flow is a bit different than I understood it at first glance. A user can only use the web form if they are logged in with an account. Otherwise the buttons appear, but they are mailto links which open the users email client.

Ok we have two options now:

In T386559#11683933, @A_smart_kitten wrote:

In T386559#11683912, @jhathaway wrote:

if /message/new is the correct route [...]

FWIW, if I'm understanding correctly -- from the <!-- Reply form --> in the source of this Hyperkitty page, it seems like /hyperkitty/list/[mailing-list]/message/[id]/reply might be a route that's used for replies via the web UI.

if /message/new is the correct route, here is the count of usage from 03-05:

In T386559#11683076, @A_smart_kitten wrote:

In T386559#11683041, @jhathaway wrote:

As briefly discussed, I think the only users of the web UI for posting are spammers […]

FWIW this is not true, I have used the web UI each time I posted a message to wikitech-l.

@bd808 based on the User-Agent, User-Agent: HyperKitty on https://lists.wikimedia.org/, and looking in the logs, this message appears to have been posted from the web UI. Messages posted from the web UI are sent via SMTP to exim4, but the source IP is localhost, so our exim4 config skips spam checking. As briefly discussed, I think the only users of the web UI for posting are spammers, https://gitlab.com/mailman/hyperkitty/-/issues/264, so let's disable it.

great, thanks again for bringing this to my attention @Xaosflux

@Xaosflux I have added a DKIM key and tested that Microsoft is able to verify the key correctly. If you could resend the failed email and confirm that it is sent successfully, that would be appreciated.

sounds good, please let me know if you need help in any way

@Mbch331 thanks for reporting this issue. What is the expected mail flow? Does appel.wikimedia.nl originate the emails?

@Xaosflux thanks for reporting this issue. I did some tests with the info-en queue, but I wasn't able to reproduce the issue.