@Peachey88 awesome, thanks

instances have been deleted.

added, a few alternatives, after discussing with other folks, happy to hear of others.

In T419906#11720988, @taavi wrote:

In T419906#11713781, @jhathaway wrote:

good catch, should be that a developer account is required, amending

This is still not correct, access to Logstash requires membership in the nda, logstash-access or ops groups (see T376790).

In T419906#11710005, @taavi wrote:

If this data is too sensitive for our normal Logstash instance, why are we fine shipping them to a yet another third party?

Okay, the posting flow is a bit different than I understood it at first glance. A user can only use the web form if they are logged in with an account. Otherwise the buttons appear, but they are mailto links which open the users email client.

Ok we have two options now:

In T386559#11683933, @A_smart_kitten wrote:

In T386559#11683912, @jhathaway wrote:

if /message/new is the correct route [...]

FWIW, if I'm understanding correctly -- from the <!-- Reply form --> in the source of this Hyperkitty page, it seems like /hyperkitty/list/[mailing-list]/message/[id]/reply might be a route that's used for replies via the web UI.

if /message/new is the correct route, here is the count of usage from 03-05:

In T386559#11683076, @A_smart_kitten wrote:

In T386559#11683041, @jhathaway wrote:

As briefly discussed, I think the only users of the web UI for posting are spammers […]

FWIW this is not true, I have used the web UI each time I posted a message to wikitech-l.

@bd808 based on the User-Agent, User-Agent: HyperKitty on https://lists.wikimedia.org/, and looking in the logs, this message appears to have been posted from the web UI. Messages posted from the web UI are sent via SMTP to exim4, but the source IP is localhost, so our exim4 config skips spam checking. As briefly discussed, I think the only users of the web UI for posting are spammers, https://gitlab.com/mailman/hyperkitty/-/issues/264, so let's disable it.

great, thanks again for bringing this to my attention @Xaosflux

@Xaosflux I have added a DKIM key and tested that Microsoft is able to verify the key correctly. If you could resend the failed email and confirm that it is sent successfully, that would be appreciated.

sounds good, please let me know if you need help in any way

@Mbch331 thanks for reporting this issue. What is the expected mail flow? Does appel.wikimedia.nl originate the emails?

@Xaosflux thanks for reporting this issue. I did some tests with the info-en queue, but I wasn't able to reproduce the issue.

@DamianZaremba I tried with a couple of my test accounts, but I was unable to duplicate your results. The list manager confirmation emails all passed dkim. Would it be possible to forward me the entire email, ideally as an eml attachment? jhathaway@wikimedia.org

@Elli though this is technically possible, outside of some existing corner cases, I am not sure the added confusion of having multiple domains is worth the occasional bounced email.

In T417941#11636764, @Dzahn wrote:

@Jgreen I removed the dmarc@donate.wikimedia.org line from that alias.

I did not find any "dmarc-ruf@" alias in that postfix alias file though.

It is being mentioned in T167337 and I guess it means this is in Google. But to be sure let me add @jhathaway

We only run rspamd in combination with postfix now

@jcrespo is this still ongoing, or are you okay with closing?

Thanks @taavi

great, I'll resolve then, let me know if new problems arise.

@Nardog Microsoft appears to have resolved the issue, are you receiving ticket updates now?

@Nardog this appears to be an issue with Microsoft's email platform, they are throttling our mail server. I have opened a ticket with the support desk to try and get the issue resolved.

In T415189#11544940, @MatthewVernon wrote:

@jhathaway I did ms-be2077 today, and see the same failure mode - it failed entirely to DHCP. I killed the cookbook, re-ran it ( sudo cookbook sre.hosts.reimage --os bullseye -t T354872 --move-vlan --new ms-be2077) and it reimaged just fine. Obviously the re-run doesn't make any DNS changes because the first run does that, which does make me think the --move-vlan bit is what's going wrong.

@Nardog we changed our DMARC policy on wikimedia.org to quarantine on 2026-01-20, so that is probably the cause of the deliverability issue. I'll look into why the emails are getting rejected and revert the change if necessary.

Strangely I re-imaged both servers from cumin2002 and ran into no issues. Perhaps when you ran the first re-images @MatthewVernon, though they failed, they setup the conditions for the following re-images to succeed? Was there any interesting output from the move-vlan cookbook? I ran the following commands:

@MatthewVernon looking...

In T412451#11506961, @elukey wrote:

@jhathaway yes please! You can use wdqs1029 or 1030 :)

In T412451#11504344, @elukey wrote:

I tried to reimage wdqs1029 today trying to test https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1214488, but the second try led me to:

Booting from Embedded SATA Port Disk B: debian
error: disk `mduuid/2884f27b943ef5a81ae46a251fc7960c' not found.
grub rescue>

@jhathaway this is really weird, it seems like T404356 but I can't pin point the issue. Any ideas?

In T367399#11502051, @hashar wrote:

Something I forgot, the operations-puppet-catalog-compiler-test job (Puppet 5) was tied to the Jenkins label puppet5-compiler-node. The label is applied to three agents:

• pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud
• pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud
• pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud

The Jenkins agents do not any other labels and I am thus guessing they can be removed from Jenkins and the underlying WMCS instances can be decommissioned? They were setup by @jhathaway :)

In T411027#11420456, @JKelsoteel-WMF wrote:

@jhathaway I did create a group called "noreply@wikimedia.org" to see if messages to that address were also dropped, and it seems they are not. Noah is ok with using the noreply@ address instead of no-reply@ - do you see any problems with this? Thanks for your help!

@JKelsoteel-WMF the addresses no-reply or noreply are used to indicate that the sender does not expect replies to be sent to that address, and any replies will be discarded. Why is using no-reply@wikimedia.org necessary for this use case?

In T408967#11345061, @Krd wrote:

I think I have changed the setting and deployed it, but it still shows the old value, and now cannot be edited. Please check what is wrong.
Please change it to: volunteers-vrt@wikimedia.org

I'm not sure how to remedy this issue. I see we switched to StaticDB in T355979, perhaps we need to rebuild the StaticDB index?

In T408967#11341265, @Krd wrote:

I just heard from one user that password recovery still doesn't work for them.

@Krd we are still receiving bounces for that user as their email rate is still too high. Do they need to subscribe to the 77 remaining queues? Could we perhaps unsubscribe them from all, and pop them a note to resubscribe?

In T408632#11341245, @Krd wrote:

I have unsubscribed the mentioned user. This appears to be the only one, and I will monitor this from now on. It makes no sense to get copies of thousands of spams messages each day.

Is this still occurring?

In T408632#11340504, @Krd wrote:

Please provide in private who that is and how you found the information.

In T408632#11339222, @Krd wrote:

If I checked correctly, nobody is subscribed to the Junk queue, so no notifications for that should have been created.
If there were any, I think that should be investigated further.

After some analysis today, I think the cause of the bounces were as follows:

@Xaosflux the outbound queue has now been cleared of all backscatter bounce emails, so delivery times should be back to normal.

@Xaosflux I assume it is related, but I have not been able to confirm it yet.

@Krd I see the junk mail queue is now at 600k, how can I help clear it out, I saw some of the scheduled jobs were run, but that does not seem to be enough. Also feel free to contact me on IRC for some real time triaging.

@Krd how else can I help?

In T408632#11321073, @Krd wrote:

The 219.240.37.89 looks like a common factor. Can we block this source IP for SMTP as a first measure?

@Krd thanks, I'm investigating, not sure of the cause either.

From a brief look, most of these conntrack entries are from an-coord1003.eqiad.wmnet, along with log entries of the form:

deploy2002 is running bullseye, which has ssh 1:8.4p1-5+deb11u5, so it does not have any of the post quantum algorithms that were first added in 9.0.

In T371416#11241235, @jcrespo wrote:

The https interface has a terrible GUI a bit hidden between submenus.

In T371416#11241205, @jcrespo wrote:

what steps did you take to re-image it

I had to redo the HW RAID setup, which was missing from the configuration of the host (it was there before) through the mgmt interface. Then I did a manual partitioning (which may not have been needed, but I did based on your feedback that the partitioning was correct). All of that was quite painful.

In T371416#11240391, @jcrespo wrote:

It is normal that the recipe was not working, not only the logical configuration was destroyed, the RAID was not setup, too, so it had no virtual RAID configuration either.

@jcrespo it took me a bit of time to coerce the box back into bios mode. I then tried reimaging with bookworm, but the raid step failed, due to the existence of the raid6 volume. After trying a couple of efforts, which failed, I booted off a rescue image and removed the raid6 volume with storcli.

As @jcrespo pointed out on IRC, there is also a quite a bit of puppet 5 documentation which needs to be removed or updated as part of this task.

1. Working hacks!

Thanks @CDanis I happened upon that post as well, I don't think their approach is unreasonable. I think there are different trade offs between complexity and adherence to spec. My preference is to try the sync script, but if that fails I'm happy to look at their approach.

In T404356#11184299, @elukey wrote:

The host doesn't PXE/HTTP boot for some reason, I reopened the provision task in T394357#11184292.

In T404356#11209467, @MatthewVernon wrote:

Does /boot even need to be on a separate partition for UEFI booting?

Would it be acceptable to store the data from the parsed DMARC reports in OpenSearch? My initial estimate is that it would require about 50MiB of data per day.

I was able to reproduce on dse-k8s-worker1014 by flipping VT on that re-running the cookbook, how does this patch look, https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1186619, it tests for me okay on dse-k8s-worker1014.

because of the error on line 22?