Very likely related to T285527.

Fun question: why is that being sent out from cloudinfra mx:es instead of deployment-prep's own mx boxes?

In T291679#7376088, @Majavah wrote:

Fun question: why is that being sent out from cloudinfra mx:es instead of deployment-prep's own mx boxes?

I have no idea, but either way I don't think that wouldn't fix this specific problem.

Trying to get another shot at verifying my email and still no luck, although the hostname that the IP resolves to has changed to mx-out03.wmcloud.org (maybe related to T294194, see below for my SMTP server's logs).

postfix/smtpd[2237128]: connect from mx-out03.wmcloud.org[185.15.56.18]
postfix/smtpd[2237128]: Anonymous TLS connection established from mx-out03.wmcloud.org[185.15.56.18]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256
postfix/smtpd[2237128]: NOQUEUE: reject: RCPT from mx-out03.wmcloud.org[185.15.56.18]: 450 4.1.8 <wiki@wikimedia.beta.wmflabs.org>: Sender address rejected: Domain not found; from=<wiki@wikimedia.beta.wmflabs.org> to=<wiki@chlod.net> proto=ESMTP helo=<mx-out03.wmcloud.org>
postfix/smtpd[2237128]: disconnect from mx-out03.wmcloud.org[185.15.56.18] ehlo=2 starttls=1 mail=1 rcpt=0/1 bdat=0/1 quit=1 commands=5/7

Been a while since this last updated so I just took a wildshot and logged into Horizon myself (after assuming that the beta cluster is also managed here, considering its wmflabs.org domain) to see if this is quickly solvable, and it does seem like it. An MX record set can be created (likely under the beta.wmflabs.org zone) and be set to point towards any MX server. Setting the MX record for wikimedia.beta.wmflabs.org to point to mx-out03.wmcloud.org will likely resolve this specific issue, as the from address's domain will finally resolve to a value. Of course, this only might let mail through assuming that SPF and other DMARC checks aren't failing.

I don't have access to the Beta cluster on Horizon (if such is even managed there), so this isn't something I can do myself. Would be extremely useful if someone can get this patched up, since I'm not able to test out OAuth features as advertised in mw:OAuth/For Developers#Setting up a development environment without validating my email address. I don't want to be forced to make OAuth test applications on actual Meta.

I can see the existing record sets in the beta.wmflabs.org zone in deployment-prep horizon, but I can’t create a new record set myself, probably because I’m only a user but not an admin of the project. (In the tools project, on the other hand, I have a “create record set” button available.) But I agree that adding that DNS record would probably help with this issue – my mail server logs look pretty similar, and mx-out03.wmcloud.org still seems to be the right domain (or instance-mx-out03.cloudinfra.wmflabs.org – both have the same IP address).

This popped up recently on the Wikimedia Discord and I decided to check if it still holds up. Still does, it seems. The above digs still don't return MX records. Here's postfix log entries for an attempted email delivery earlier, which didn't succeed.

Jun 12 05:02:05 mailman postfix/postscreen[2881289]: CONNECT from [185.15.56.18]:24739 to [172.18.0.2]:25
Jun 12 05:02:11 mailman postfix/postscreen[2881289]: PASS OLD [185.15.56.18]:24739
Jun 12 05:02:12 mailman postfix/smtpd[2934976]: connect from instance-mx-out03.cloudinfra.wmflabs.org[185.15.56.18]
Jun 12 05:02:12 mailman postfix/smtpd[2934976]: Anonymous TLS connection established from instance-mx-out03.cloudinfra.wmflabs.org[185.15.56.18]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256
Jun 12 05:02:13 mailman postfix/smtpd[2934976]: NOQUEUE: reject: RCPT from instance-mx-out03.cloudinfra.wmflabs.org[185.15.56.18]: 450 4.1.8 <wiki@wikimedia.beta.wmflabs.org>: Sender address rejected: Domain not found; from=<wiki@wikimedia.beta.wmflabs.org> to=<wiki@chlod.net> proto=ESMTP helo=<mx-out03.wmcloud.org>
Jun 12 05:02:13 mailman postfix/smtpd[2934976]: disconnect from instance-mx-out03.cloudinfra.wmflabs.org[185.15.56.18] ehlo=2 starttls=1 mail=1 rcpt=0/1 bdat=0/1 quit=1 commands=5/7

This seems to happen for the confirmation/registration email (for accounts created by existing users on Special:CreateAccount), but not for other emails. Notably, I received a user right change email following this log entry, albeit it was marked as spam.

Hi, I found this task, as I am also unable to confirm my email address on the beta cluster.

In T291679#7474212, @Chlod wrote:

Been a while since this last updated so I just took a wildshot and logged into Horizon myself (after assuming that the beta cluster is also managed here, considering its wmflabs.org domain) to see if this is quickly solvable, and it does seem like it. An MX record set can be created (likely under the beta.wmflabs.org zone) and be set to point towards any MX server. Setting the MX record for wikimedia.beta.wmflabs.org to point to mx-out03.wmcloud.org will likely resolve this specific issue, as the from address's domain will finally resolve to a value. Of course, this only might let mail through assuming that SPF and other DMARC checks aren't failing.

I don't have access to the Beta cluster on Horizon (if such is even managed there), so this isn't something I can do myself. Would be extremely useful if someone can get this patched up, since I'm not able to test out OAuth features as advertised in mw:OAuth/For Developers#Setting up a development environment without validating my email address. I don't want to be forced to make OAuth test applications on actual Meta.

I know you posted this almost 5 years ago, but I tried doing it anyway. To be clear, I have no idea what I'm doing.

I created this thing, I hope I did it correctly: https://horizon.wikimedia.org/ngdetails/OS::Designate::RecordSet/4482e09c-3d25-447f-b8e2-5aa2a105b60e/99124be1-c471-4c74-b2f9-b4288ae3447c

Unfortunately it did not seem to have helped, at least for my email.

If anyone is willing to try debugging this, I would be happy to be your arms. Otherwise I guess I'll delete that in a few days, to avoid confusing people in the future.

Well, I just tried sending an email to myself via Special:EmailUser and it worked fine. So from my side this seems to have worked :)

It depends on the recipient address. I can receive the emails at an @gmail.com address (and I could receive them earlier today as well, before the changes), but not at an @fastmail.com one. (I'm not cool enough to use my own domain or run my own mail server.)

I also started a support thread with Fastmail, maybe they can say why it's not coming through in this case.

In T291679#11642995, @matmarex wrote:

It depends on the recipient address. I can receive the emails at an @gmail.com address (and I could receive them earlier today as well, before the changes), but not at an @fastmail.com one. (I'm not cool enough to use my own domain or run my own mail server.)

This is an unrelated problem to this task. The deployment-prep mail server (deployment-mx04) does not have a floating IPv4 assigned to it, instead outbound v4 traffic from it will be NATed to the shared egress NAT address. As documented on https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS, outbound mail cannot be sent that way - only instances with floating IP addresses can send mail directly to the internet. Gmail and Lucas's mail server have IPv6 support, which does not have this problem, but Fastmail does not and so the beta mail server cannot reach it.

Thanks @taavi. I've also had a conversation about this on #wikimedia-releng on IRC with bd808 and hashar yesterday, who said something similar:

https://wm-bot.wmcloud.org/browser/index.php?start=02/23/2026&end=02/23/2026&display=%23wikimedia-releng

<bd808> There is a floating IP setup in the Deployment-prep Cloud VPS project with the description "For MX server".
<bd808> huh. not mapped to any instance and the Horizon UI doesn't give me any logging about the history of that object
<hashar> bd808: Beta got used to validate/pretest an extension that deals with emails and or for a mx migration maybe?
<hashar> Puppet has a bunch of history for ""deployment-mx""
<hashar> I think the deployment-mx was made because some wanted to reproduced every single bit of infra we had
<hashar> or maybe there was a use case for testing something involving the mx

I went digging into the aforementioned Puppet history, and found https://gerrit.wikimedia.org/r/c/operations/puppet/+/169194 from 2014 by Tony Thomas, when deployment-mx was set up. I found something more recent in T412975, which hints that it's still needed – for testing BounceHandler (checks out, since Tony Thomas worked on that).

I also found T343925, where you and Ladsgroup solved seemingly the same problem as reported in this task, but for deployment-mx03 ("Sender address rejected: Domain not found", assigned a floating IP address).