In T411027#11420456, @JKelsoteel-WMF wrote:

@jhathaway I did create a group called "noreply@wikimedia.org" to see if messages to that address were also dropped, and it seems they are not. Noah is ok with using the noreply@ address instead of no-reply@ - do you see any problems with this? Thanks for your help!

@JKelsoteel-WMF the addresses no-reply or noreply are used to indicate that the sender does not expect replies to be sent to that address, and any replies will be discarded. Why is using no-reply@wikimedia.org necessary for this use case?

In T408967#11345061, @Krd wrote:

I think I have changed the setting and deployed it, but it still shows the old value, and now cannot be edited. Please check what is wrong.
Please change it to: volunteers-vrt@wikimedia.org

I'm not sure how to remedy this issue. I see we switched to StaticDB in T355979, perhaps we need to rebuild the StaticDB index?

In T408967#11341265, @Krd wrote:

I just heard from one user that password recovery still doesn't work for them.

@Krd we are still receiving bounces for that user as their email rate is still too high. Do they need to subscribe to the 77 remaining queues? Could we perhaps unsubscribe them from all, and pop them a note to resubscribe?

In T408632#11341245, @Krd wrote:

I have unsubscribed the mentioned user. This appears to be the only one, and I will monitor this from now on. It makes no sense to get copies of thousands of spams messages each day.

Is this still occurring?

In T408632#11340504, @Krd wrote:

Please provide in private who that is and how you found the information.

In T408632#11339222, @Krd wrote:

If I checked correctly, nobody is subscribed to the Junk queue, so no notifications for that should have been created.
If there were any, I think that should be investigated further.

After some analysis today, I think the cause of the bounces were as follows:

@Xaosflux the outbound queue has now been cleared of all backscatter bounce emails, so delivery times should be back to normal.

@Xaosflux I assume it is related, but I have not been able to confirm it yet.

@Krd I see the junk mail queue is now at 600k, how can I help clear it out, I saw some of the scheduled jobs were run, but that does not seem to be enough. Also feel free to contact me on IRC for some real time triaging.

@Krd how else can I help?

In T408632#11321073, @Krd wrote:

The 219.240.37.89 looks like a common factor. Can we block this source IP for SMTP as a first measure?

@Krd thanks, I'm investigating, not sure of the cause either.

From a brief look, most of these conntrack entries are from an-coord1003.eqiad.wmnet, along with log entries of the form:

deploy2002 is running bullseye, which has ssh 1:8.4p1-5+deb11u5, so it does not have any of the post quantum algorithms that were first added in 9.0.

In T371416#11241235, @jcrespo wrote:

The https interface has a terrible GUI a bit hidden between submenus.

In T371416#11241205, @jcrespo wrote:

what steps did you take to re-image it

I had to redo the HW RAID setup, which was missing from the configuration of the host (it was there before) through the mgmt interface. Then I did a manual partitioning (which may not have been needed, but I did based on your feedback that the partitioning was correct). All of that was quite painful.

In T371416#11240391, @jcrespo wrote:

It is normal that the recipe was not working, not only the logical configuration was destroyed, the RAID was not setup, too, so it had no virtual RAID configuration either.

@jcrespo it took me a bit of time to coerce the box back into bios mode. I then tried reimaging with bookworm, but the raid step failed, due to the existence of the raid6 volume. After trying a couple of efforts, which failed, I booted off a rescue image and removed the raid6 volume with storcli.

As @jcrespo pointed out on IRC, there is also a quite a bit of puppet 5 documentation which needs to be removed or updated as part of this task.

1. Working hacks!

Thanks @CDanis I happened upon that post as well, I don't think their approach is unreasonable. I think there are different trade offs between complexity and adherence to spec. My preference is to try the sync script, but if that fails I'm happy to look at their approach.

In T404356#11184299, @elukey wrote:

The host doesn't PXE/HTTP boot for some reason, I reopened the provision task in T394357#11184292.

In T404356#11209467, @MatthewVernon wrote:

Does /boot even need to be on a separate partition for UEFI booting?

Would it be acceptable to store the data from the parsed DMARC reports in OpenSearch? My initial estimate is that it would require about 50MiB of data per day.

I was able to reproduce on dse-k8s-worker1014 by flipping VT on that re-running the cookbook, how does this patch look, https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1186619, it tests for me okay on dse-k8s-worker1014.

because of the error on line 22?

On the mx-in servers you can obtain routing information via sendmail -bv, however it is a bit more annoying to work with compared to exim -bt

Thanks @CDanis that is also worth looking into as a redundancy option.

In T378331#11158026, @Volans wrote:

It works fine for me:

We get the correct exit codes now, which is fabulous. However, what is less fabulous as @Tgr notes is that the majority of the error codes are 74, which is used by msmtp to indicate either a disk I/O error or a network I/O error, both of which are broad categories. So, we really need more debug info, how do we obtain it?

It looks like puppet 5 is failing, while puppet 7 is running successfully?

It appears there were two issues:

@TAndic I ran an initial test and our setup appears to be working correctly. Could we perhaps jump on a call and review the configuration together and send some test emails?

great, please re-open if you have any issues

@ttaylor access should be setup, you should receive an email about setting up your kerberos credentials. Please try everything out and report back.

In T400288#11033037, @HCoplin-WMF wrote:

I was indeed able to log in and request it that way! Thank you :)

Apologies for the double request here, if that's y'all's preferred method. You might want to update the SRE team page to include info about the IDM tool. Requesting the access there was super easy and more people should know about it!

Supermicro indicated that the debug output from the supplied BIOS is only outputted to COM2. From briefly looking at the manual COM2 is only available as a header on board, "One serial port on the rear I/O panel (COM1) and one onboard header (COM2)." Which, from my read, indicates we will have to purchase the PCIE bracket and cable.

@ttaylor when you have a moment, please review and sign the L3 server access document.

@HCoplin-WMF happy to help grant you access. You may be able to request access through our new IDM tool, https://idm.wikimedia.org. Can you try logging in and requesting access?

@Miriam would you kindly approve @diego's request to be added to analytics-research-admins?

@KFrancis would you kindly confirm that @Novem_Linguae has signed the NDA?

In T400211#11031851, @Jhancock.wm wrote:

I'm not sure for the wording and want to clarify. Are you ordering the items needed? Or do we need to start a procurement task?

In T400211#11031335, @Jhancock.wm wrote:

@jhathaway i can take care of this connection today. Can you remind me what the hostname of this server is?

@nisrael I sent you an invite, let me know if you can get in.

Based on our discussion with Supermicro on the call today, my understanding is that we will need to support model specific quirks.

In T394788#11029549, @nisrael wrote:

Great thank you Jesse! Just want to confirm, am I safe toinstruct our rep at DMarcian to restart our free trial?

In T394788#11017745, @nisrael wrote:

Hi SRE team,

Checking in on this task. Do you have an approximate timeline when we'd be able to configure the mail routing to DMarcian? Our goal is to restart our trial with them.

ysuu9wx7@ag.us.dmarcian.com has been added to the rua record, and should start receiving aggregate reports

Closing for now, as the testing of the intel card is complete for the moment

In T378028#10995390, @Arnoldokoth wrote:

Thanks @MoritzMuehlenhoff We'll consider that... But I'm doubtful we "strictly" need to test this on hardware. A VM will just work as well me thinks. @jhathaway is out on vacation so we can wait and hear is thoughts when he's back.