I am tagging Collaboration-Services for your information only, feel free to triage as you see fit :)

Seems like mx-in*.wikimedia.org do not like these emails for whatever reason:

2026-04-07 19:39:51 1wACH9-00BqE5-1C ** phabricator-no-reply@wmcloud.org R=dnslookup_unsigned T=remote_smtp_unsigned H=mx-in2001.wikimedia.org [208.80.153.75] I=[172.16.2.248] X=TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes DN="CN=mx-in1001.wikimedia.org": SMTP error from remote mail server after RCPT TO:<phabricator-no-reply@wmcloud.org>: 550 5.1.1 <phabricator-no-reply@wmcloud.org>: Recipient address rejected: User unknown in relay recipient table DT=0s

In T422559#11796709, @taavi wrote:

Seems like mx-in*.wikimedia.org do not like these emails for whatever reason

my first guess/interpretation without digging too hard is the destination MX is sending a new SMTP connection back to the no-reply address (maybe because envelope-from or from) and the wmcloud.org MX is saying it doesn't exist so then the original connection is rejected too.

see https://dev.to/sofiatarhonska/how-to-verify-email-address-without-sending-an-email-10jp#email-box-pinging

Yeah, I guess it seems like this might potentially be being caused by Wikimedia's production mail infra somewhere.

FWIW, these are some sample headers from an email the test-instance might send (as extracted from an email it sent to me the other day). [If SREs would like to examine a fuller version of an email's headers, let me know.]

Date: Sun, 5 Apr 2026 07:24:23 +0000
To: Phabricator <phabricator-no-reply@wmcloud.org>
From: a_smart_kitten__test <phabricator-no-reply@wmcloud.org>
Cc: [my email address]
Reply-to: a_smart_kitten__test
 <T178+public+a082557452deb877@phabricator.wmcloud.org>
Subject: [Maniphest] [Commented On] T178: test task with other assignee set

[Having the To: header be the no-reply email address itself seems interesting; but IIUC this is the same as what production-Phabricator does, and it seems like it's (somewhat) documented in upstream's docs.]

From doing some searching, I found the patch 'postfix: verify recipients when possible' (for T325406) from July 2024, which seems like it might be what's causing Wikimedia's MX servers to reject these emails from the test Phab instance. cc @jhathaway FYI / in case you have any thoughts.

I guess one way of solving this might be to add wmcloud.org to profile::postfix::mx::unverifiable_domains (& maybe also wmflabs.org, for completeness). But I don't know enough about this configuration to currently know e.g. whether that'd be a good way of solving this.
(It might be worth noting that phabricator.wikimedia.org is already listed under that Hiera key.)

From upstream's docs (my emphasis):

[…] the [To:] address should ideally correspond to a valid, deliverable mailbox that accepts the mail and then simply discards it. If the address is not valid, some outbound mail will bounce, and users will receive bounces when they "Reply All" even if the other recipients for the message are valid. In contrast, if the address is a real user address, that user will receive a lot of mail they probably don't want.

Hmm. I'm not sure whether this sort of mailbox - ie., one that can be delivered to, but that discards all mail, & that ends with wmcloud.org - exists (and/or, whether it'd currently be possible for this sort of mailbox to exist) for/within Cloud VPS.

I suppose, if fixing this issue by way of a change to Wikimedia's production mail config would be undesirable, it might potentially be possible to try and modify how the Phab/Phorge software behaves in this regard (as one random idea, maybe a new config setting that moves recipients in the Cc: header to the To: header, for emails where there wouldn't otherwise be any valid recipients in the To: header?). But I guess that would might depend on upstream's willingness to accept that sort of change.
[adding Phabricator for visibility, given that one possible option here is a Phab/Phorge software change]

If possible, I would be interested to hear what the folks that maintain Wikimedia's production mailservers (Infrastructure-Foundations?) think about what's been written in this task so far, though (& whether a fix for this should be on the Wikimedia mailserver side of things, and/or the Phab side of things, and/or somewhere else) :)

Removing Phabricator tag for now unless an upstream change is the ultimate way to go

The issue is that on mx-in{1001,2001}.wikimedia.org we have wmcloud.org setup as a relay domain in postfix, but at present we have no relay recipients configured. So even though phabricator-no-reply@wmcloud.org is the sender, Postfix is rejecting the mail since it is not a valid recipient for that domain. I can add a recipient map and add the test phabricator address?

In T422559#11938383, @jhathaway wrote:

I can add a recipient map and add the test phabricator address?

To make sure I understand this option correctly, would this option result in anything happening (and/or needing to happen) on the Test-Phabricator side of things (ie., the devtools VPS project)?

Thanks for the root cause @jhathaway!

Well, I guess we should take a moment to think about it before starting to maintain a new manual list that might then grow further to other tools.

If we are just that one special use case, alright, let's do that. But others will likely follow with requests to be added.

Maybe a policy about it first would be good here.

Or are we just supposed to use a different sender address? It's not like we care as long as the mail arrives.

In T422559#11942353, @Dzahn wrote:

Or are we just supposed to use a different sender address? It's not like we care as long as the mail arrives.

IIUC, any sender address that we could (currently) use from Cloud VPS might have the same problem here -- IIUC, to send email from Cloud VPS, the sender-address has to end in @wmcloud.org (T388022#11233132); however, it seems like Wikimedia's inbound MX servers might currently reject any emails that include a To: header that points to a @wmcloud.org address (as it seems like some emails from the test Phab instance would do).

@Dzahn and @A_smart_kitten, good points, I think this is a policy question for WMCS. Two basic options are:

1. mx-in: accept all addresses from wmcloud.org
2. mx-in: accept a static list of addresses
3. mx-in: call out to a wmcloud.org mx server to verify the address.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Just ran into this again the other day. A new Phab test instance was setup - we wanted to test - could not reset password - then remembered this ticket.

In T422559#11945766, @jhathaway wrote:

1. mx-in: accept all addresses from wmcloud.org

This seems the best option to me.

(And since mx-in are also the inbound MXes for wmcloud.org, imo it would be smart for them to just silently drop mail for anything not bound to the already defined well-known addresses.)