Presumably caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219137
@Muehlenhoff @Jelto

Why on earth does deployment-prep need an MX host? We could revert the original patch, but having cleanups held back by random unmaintained cloud VPS nodes is an unhealthy pattern.

This kind of thing is just a pain for all parties involved :/

How about this:

We add a lookup for the package name with a default value of spamd. And then we override it in web-Hiera on Horizon on this instance.

That way we don't have to revert it all and can get puppet fixed.

Change #1219180 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mx/spamassassin: allow overriding sa daemon package name in Hiera

https://gerrit.wikimedia.org/r/1219180

like this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180

That would mean no more OS / distro version check while still allowing to override it.

In T412975#11469602, @Dzahn wrote:

like this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180

That would mean no more OS / distro version check while still allowing to override it.

This adds complexity for no purpose. There is no need for an MX reference in deployment-prep to begin with, but if anyone wants to see one it should follow to what is actually used in prod. The MXes in prod are on Bookworm for a long time and are also running a totally different that than the abandoned deployment-mx03 (we're on Postfix instead of Exim for a long time now).

I was trying to be pragmatic and just fix it without getting into the deeper problems with beta ownership.

This adds complexity for no purpose.

The purpose would have been to be able to move on without having to revert the OS check.

no need for an MX reference in deployment-prep to begin with,
if anyone wants to see one

No idea. Not going to fix ownership issues. Explicitly tried to avoid this.

Change #1219180 abandoned by Dzahn:

[operations/puppet@production] mx/spamassassin: allow overriding sa daemon package name in Hiera

Reason:

https://phabricator.wikimedia.org/T412975#11473246

https://gerrit.wikimedia.org/r/1219180

@dancy do you have any clues as to the need for this MX server? Also cc'ing @taavi as it was suggested to me that you may have knowledge on the topic. Knowing this would help to fix it the right way.

In T412975#11477379, @LSobanski wrote:

@dancy do you have any clues as to the need for this MX server?

Only vaguely. Codesearch indicates that it is referenced in 3 places:

• https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/825c3f3b6858eca9a1c16b7c7e5aec5fddf0b7db/deployment-prep/deployment-acme-chief.yaml#13
• https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/7e99b6bbe2b4ad8618af8d772236f6b9d5f987b6/wmf-config/CommonSettings-labs.php#219 (BounceHandler config)
• https://gerrit.wikimedia.org/g/operations/puppet/+/6c3a27773a97125440b611dba92dd47990651d4c/hieradata/cloud/eqiad1/deployment-prep/common.yaml#343

There's been an MX in beta for over a decade and it's there because beta strives to be "production-like" for users. deployment-mx03 is not abandoned—Developer Experience are the stewards for beta (since Nov 2024) so we strive to at least maintain status quo—which is why we filed this task.

We definitely don't/can't hold details for every service running in prod, but when we see a puppet failure it's a good signal something is wrong—sounds like there's more wrong here than the puppet failure (which is, unfortunately, not uncommon for Beta).

@MoritzMuehlenhoff if @Dzahn 's workaround is less than ideal to you as someone who knows how the MX hosts work in prod, then how should we proceed to get beta back on its feet?

In T412975#11477568, @thcipriani wrote:

@MoritzMuehlenhoff if @Dzahn 's workaround is less than ideal to you as someone who knows how the MX hosts work in prod, then how should we proceed to get beta back on its feet?

It should be as simple as re-creating a new node with bookworm as deployment-mx04. The old stack on Bullseye used Exim, while we've migrated to Postfix on the current mx* nodes in prod.

Even though it may be around for a decade, I would still argue however that an MX is not really in scope for deployment-prep: if the purpose is to be a testground for MX-related changes, then it fails that objective since running an MX in cloud VPS is simply too different in terms of external connectivy to be useful. For the purpose of sending out mail it for beta it would be better to simply use the default MXes for Cloud VPS.

If anyone wants to merge Daniel's patch as workaround over the holiday period, fine with me. But the underlying issue is that mx03 is fundamentally outdated and needs to be replaced.

Change #1219180 restored by Dzahn:

[operations/puppet@production] mx/spamassassin: allow overriding sa daemon package name in Hiera

https://gerrit.wikimedia.org/r/1219180

In T412975#11477609, @MoritzMuehlenhoff wrote:

In T412975#11477568, @thcipriani wrote:

@MoritzMuehlenhoff if @Dzahn 's workaround is less than ideal to you as someone who knows how the MX hosts work in prod, then how should we proceed to get beta back on its feet?

It should be as simple as re-creating a new node with bookworm as deployment-mx04. The old stack on Bullseye used Exim, while we've migrated to Postfix on the current mx* nodes in prod.

Okie doke, there is now a deployment-mx04 running bullseye. After fighting puppet ssl cert signing and making a small regex tweak, puppet stopped yelling at me.

Using mail on the command line works. Are there other bits I should check here?

Even though it may be around for a decade, I would still argue however that an MX is not really in scope for deployment-prep: if the purpose is to be a testground for MX-related changes, then it fails that objective since running an MX in cloud VPS is simply too different in terms of external connectivy to be useful. For the purpose of sending out mail it for beta it would be better to simply use the default MXes for Cloud VPS.

That's fair—it's possible others know better than I do about why it was done this way. But I could also believe it's here because prod has one and therefore beta has one. In practice, it seems to be used for https://www.mediawiki.org/wiki/Extension:BounceHandler and as the smart_host (guessing for password resets and the like). Why not use the cloud MX? Dunno.

Change #1219939 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[operations/mediawiki-config@master] Beta: update mx host ip

https://gerrit.wikimedia.org/r/1219939

In T412975#11478073, @thcipriani wrote:

Why not use the cloud MX? Dunno.

I'm pretty sure that having an MX in deployment-prep predates having a shared MX for all of Cloud VPS. And the answer to an expected follow up of "why not use a prod MX then?" is that Faidon would have skinned me alive for doing so.

Should I abandon or keep https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180 now?

In T412975#11498683, @Dzahn wrote:

Should I abandon or keep https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180 now?

You can abandon it. Thanks again for stepping in quickly to try to help.

Change #1219939 merged by jenkins-bot:

[operations/mediawiki-config@master] Beta: update mx host ip

https://gerrit.wikimedia.org/r/1219939

Change #1225620 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] /home/dancy/src/wmf/operations/puppet/hieradata/cloud/eqiad1/deployment-prep/common.yaml

https://gerrit.wikimedia.org/r/1225620

Change #1219180 abandoned by Dzahn:

[operations/puppet@production] mx/spamassassin: allow overriding sa daemon package name in Hiera

Reason:

https://phabricator.wikimedia.org/T412975#11513241

https://gerrit.wikimedia.org/r/1219180

Change #1225620 merged by JHathaway:

[operations/puppet@production] deployment-prep common.yaml: Update mediawiki_smarthosts

https://gerrit.wikimedia.org/r/1225620

Mentioned in SAL (#wikimedia-releng) [2026-01-16T16:33:16Z] <dancy> Deleting deployment-mx03.deployment-prep (T412975)

I confirmed that I did receive an email when I created a beta wiki account.