The mail config appears to be:

$mail_config = [
    {   
        'key'      => 'wikimedia-smtp',  
        'type'     => 'smtp',
        'options'  => {
            'host' => 'localhost',
            'port' => 25,
        }
    }
]

So mostly this question would be "is it even possible to deliver mail from cloud VPS instances to the outside world" or "why does mail to localhost not get delivered on a default cloud VPS instance" or something like that. Which would be more a general question to ask wmcs team first.

Then after that we could see if and which part of the config would have to be adjusted in cloud vs prod while keeping the difference as minimal as possible.

Also Phabricator (Phorge) config is split between puppet (SRE) and scap (releng) so additionally it depends what a task is about which team really should look. That being said, SRE collab and releng typically meet for Phabricator deploys anyways, so this should be doable.

Finally, no "VPS project Phabricator" exists, at least not anymore. There is only a VPS project called "devtools" nowadays with some test instances of some services.

So probably herald rule to add both collab and releng and/or retire that tag or replace it.

So mostly this question would be "is it even possible to deliver mail from cloud VPS instances to the outside world" or "why does mail to localhost not get delivered on a default cloud VPS instance" or something like that. Which would be more a general question to ask wmcs team first.

It is possible, see Help:Email in Cloud VPS. Is the from address @wmcloud.org?

@JJMC89 Thank you! The part "in most cases you can set the SMTP host to localhost and port to 25" is perfect because then our existing code should just work without changes.

We might indeed just need Sending_emails_from_a_non-Cloud_VPS_domain but have to debug some more what it is actually trying.

Random thought: if fixing the email-sending issue is proving to be a challenge, maybe we could mitigate this by (temporarily?) setting https://phabricator.wmcloud.org/config/edit/auth.require-email-verification/ to false — ie., no longer requiring email verification in order for accounts to be able to login to the test Phabricator instance?
(Of course, I guess one possible downside to this idea is the potential for an increased risk of automated spambots using the test Phab instance :/)

As another potential idea… we could set auth.require-email-verification to false, but set auth.require-approval to true, and ask people to file a task in VPS-project-Phabricator (using a prefilled form like this) for an admin on the test instance to approve their account. These sorts of approvals would hopefully require less time/effort than the de facto current process of a collaboration-services SRE SSHing to the test instance to manually activate a user's email address (xref T377236, T397280), and would hopefully make the test instance available for more people to use.

If either of those ideas sound good, we could then slightly reframe this task to just be something like "Phabricator test instance can't send email" (or resolve this one and split that issue out into a new task).

@Dzahn @Aklapper (and/or anyone else), what do you think?

a task to request account activation is a bit of a hurdle to get into the test instance [and requires SRE time

To be honest the SRE time required to manually activate some phab users on request is orders of magnitude smaller than resolving this ticket or dealing with spam bots.

Your suggestions sound good and are really thoughtful but there is also no simple way to have different configuration settings on test vs prod instance.

an admin on the test instance to approve their account. These sorts of approvals would hopefully require less time/effort than the de facto current process of a collaboration-services SRE SSHing to the test instance

To me personally SSHing to the instance and running a single command is less effort than logging in on a web UI and finding where to approve a user in there. The new suggestion would require a list of admins to exist that can be contacted or a ticket can be assigned to somehow, which wouldn't be obvious to me how.

There are two separate issues being conflated here:

• Issue 1: Should some human have to manually approve accounts on the test instance (in some way or another)
• Issue 2: If the answer to Issue 1 is yes, what should that process be.

You seem to be thinking the answer to issue 1 is yes. I would think it is no; it's illogical for the test instance to be more locked-down than the production instance here.

The production instance is connected to other systems handling the user sign-up. MediaWiki/SUL and LDAP/developer accounts with their own mechanisms to prevent abuse. The test instance doesn't have these. So I am not sure about the "more locked down" part here.

It seems very likely that just removing the approval step will result in spam users.

In T388022#11230217, @Pppery wrote:

There are two separate issues being conflated here:

• Issue 1: Should some human have to manually approve accounts on the test instance (in some way or another)
• Issue 2: If the answer to Issue 1 is yes, what should that process be.

You seem to be thinking the answer to issue 1 is yes. I would think it is no; it's illogical for the test instance to be more locked-down than the production instance here.

I feel like there might be some inadvertent crossed wires here. (I apologise if I could have worded my last comment better!)

To be clear, I was suggesting that - if it's proving difficult to get the test Phab instance to successfully send emails - the "must have a verified email address" config requirement could be disabled for the test instance. (Judging by @Dzahn's comment re. there not being a simple way for the test instance to have different config to the prod instance, though, I guess this might not be possible in the first place.)

My second idea of requiring account approval was in case folks would (e.g.) be worried about increased spam on the test instance if the email verification requirement was removed; and I was putting forward the idea of account approval to try to pre-emptively offer a solution to that potential problem. I don't necessarily per se think that a human should have to manually approve accounts on the test instance — I intended to just offer it as an idea, for if folks thought that removing the email verification requirement would be unworkable on its own :)

In T388022#11230175, @Dzahn wrote:

[…] The new suggestion would require a list of admins to exist that can be contacted or a ticket can be assigned to somehow, which wouldn't be obvious to me how.

FWIW, I watch the VPS-project-Phabricator tag & I imagine I'd've been be happy to approve accounts if/when I saw a ticket requesting account approval. However, I get that this would introduce a bus factor of 1, so I guess my suggestion would've been for a prefilled form to also tag Release-Engineering-Team (if that would've been okay with them), in a similar way to the GitLab account activation form. (To be fair though, I don't know how amenable RelEng would've been to this idea, which on its own might've made it unworkable 😅)

In T388022#11230175, @Dzahn wrote:

To be honest the SRE time required to manually activate some phab users on request is orders of magnitude smaller than resolving this ticket or dealing with spam bots.

Fair enough :) In which case, do you mind if I add a note (e.g.) under https://www.mediawiki.org/wiki/Phabricator#Get_started (& maybe also on the homepage of https://phabricator.wmcloud.org?) to inform folks that they can file a task to request that collab-services manually verifies their test instance account? That wouldn't fully mitigate the additional hurdle to using the test instance, but would hopefully at least make people aware of how they can request account activation (and that they can request it).
[genuine question]

I don't mind if you do that. That would give us an idea how common it is (or will become).

Taking a step back though: I just did a quick test and sent an email from the shell of the phab test host to myself.. and I .. got that email.

Seems like this is only in the phabricator config itself after all.. or something was fixed meanwhile since this ticket was created.

It hasn't been fixed. I asked phabricator.wmcloud.org to resend me a verification email and it didn't arrive.

In T388022#11231303, @Dzahn wrote:

I don't mind if you do that. That would give us an idea how common it is (or will become).

Thanks :) I've added some text to https://phabricator.wmcloud.org/W1 so far, and I'll probably add something to the wiki pages once I think of what to write on them. Wording improvement suggestions welcome!

Fwiw, while debugging an unrelated issue I spotted this in the Cloud VPS outbound mail server logs that looks highly relevant:

2025-10-01 13:06:08 H=phabricator-bullseye.devtools.eqiad1.wikimedia.cloud [172.16.4.197]:12115 I=[172.16.2.248]:25 F=<no-reply@phabricator.wmcloud.org> rejected RCPT <[redacted]>: Mail sent from Cloud VPS using non-supported domain phabricator.wmcloud.org

As noted in https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS#Sending_emails_from_a_Cloud_VPS_domain, subdomains of wmcloud.org cannot send mail by default (since DKIM signing mail from arbitrary subdomains is non-trivial).

How hard would it be to support sending mail from phabricator.wmcloud.org on the cloud-vps side? That seems like it would solve this problem by itself.

In T388022#11231344, @Pppery wrote:

It hasn't been fixed. I asked phabricator.wmcloud.org to resend me a verification email and it didn't arrive.

What I meant was the general sending of mail from a cloud VPS project shell to the world. I wanted to see if it's a general problem or specific to Phabricator. "fixed" in this context referred to delivering mail in general (by using the mail command on the shell).

Also I would like to add some general comments about this ticket. We do like it if people get something out of this test instance and want to use it.. we really do. But it was not planned for this. The main reason this was created was that we have something where we can test puppet and phab config changes before merging in production. So please bear with us when things like sending email or user auth wasn't a consideration before we got actual requests like this.

In T388022#11234003, @Dzahn wrote:

Also I would like to add some general comments about this ticket. We do like it if people get something out of this test instance and want to use it.. we really do. But it was not planned for this. The main reason this was created was that we have something where we can test puppet and phab config changes before merging in production. So please bear with us when things like sending email or user auth wasn't a consideration before we got actual requests like this.

The default Phabricator home page says:

• To test and experiment with Phabricator, go to phabricator.wmcloud.org instead.

Similarly both https://www.mediawiki.org/wiki/Phabricator and https://www.mediawiki.org/wiki/Phabricator/Help prominently advertise that instance. Should those be changed?

In T388022#11234012, @taavi wrote:

Should those be changed?

IMO it would be a shame if these were changed - it's just my personal opinion, but I think there's real value in having a test Phabricator instance available to people, to experiment around with things in. I (for one) have definitely found it very helpful since I've been able to access it :) I also probably wouldn't've known about the possibility of using it, if it wasn't for the advertisements of it that @taavi mentions.
(To be clear, I'm very grateful to collab-services/RelEng and everyone else for the work that's put into keeping the test instance running!)

On a slight side note: for what it's worth, it looks like the invitation on [[mw:Phabricator]] to experiment in (what I'm assuming was then called) the Phabricator Labs instance may date back to around 2014.

In T388022#11233681, Pppery wrote:

How hard would it be to support sending mail from phabricator.wmcloud.org on the cloud-vps side? That seems like it would solve this problem by itself.

I assume this question would welcome input from cloud-services-team.
Is it feasible? Or out of scope? Or something in-between?
Thanks in advance for sharing some additional setup/config knowledge.

Setting up the needed DKIM signing keys and other mail-related config is a non-trivial chunk of manual work, and I'm not sure off-hand how the additional DNS records play with the novaproxy logic that manages the proxy-related DNS records. So first I'd like to hear why Phorge can not use a differently configured sender address like every other piece of software that sends mail in Cloud VPS before deciding to support that configuration.

Judging by https://we.phorge.it/book/phorge/article/configuring_outbound_email/#outbound-quot-from-quot-and-quot, I wonder if this might be able to be resolved by setting https://phabricator.wmcloud.org/config/edit/metamta.default-address/ to something like no-reply@wmcloud.org (or maybe no-reply-phabricator@wmcloud.org)?

(disclaimer: this might not work & I have no way of testing it, but it seemed worth mentioning just in case :))

@A_smart_kitten Meanwhile I have written docs how to auth a user manually, put them on wikitech/linked on mediawiki.org and told my team. So if there are requests it won't be blocked on me personally. Cheers.

(let's see how common this actually is and only worry about it if it becomes a regular thing)

在T388022#11485293中，@Pppery写道：

What?

What? See the homepage of testing instance, we already made a form for request account

The homepage of the testing instance acknowledges the existence of this bug. That doesn't change the fact that the bug exists or make it an intended feature.

在T388022#11485297中，@Pppery写道：

The homepage of the testing instance acknowledges the existence of this bug. That doesn't change the fact that the bug exists or make it an intended feature.

This is a workaround, and this talk looks is trying to select a workaround, may need a new task for fix the original problem, but this task is not for this

This task is requesting the original problem be fixed. I have no idea what you are trying to say.

在T388022#11485299中，@Pppery写道：

This task is requesting the original problem be fixed. I have no idea what you are trying to say.

But actually, everybody ignored original problem and trying to find a workaround :)

... hence this task is, and should be, still open because it's not fixed yet

So all that needs to be done to fix this to set the metamta.default-address config setting to noreply@wmcloud.org and make sure that metamta.can-send-as-user is false, right?

@Dzahn: Hmmm how does this affect the production instance? Or what did you have in mind by adding the Phabricator project tag? :)

@Aklapper The last question above was about a configuration change. Configuration changes affect all instances. We also usually deploy changes to test instance and production. Feel free to remove again.

In T388022#11715419, @Dzahn wrote:

@Aklapper The last question above was about a configuration change. Configuration changes affect all instances.

I believe that this config change would need to be one that would only be applied to the test instance (or at least, that it'd have to be a no-op on the prod instance), otherwise I guess the prod instance would begin trying to send emails from a @wmcloud.org domain.

Disclaimer that my understanding of Puppet code (and how it interacts with Cloud VPS) is very limited, and I might be completely barking up the wrong tree here... but possibly:

• Currently, modules/profile/manifests/phabricator/main.pp seems like it might be setting metamta.default-address to no-reply@${domain} [0]. In turn, ${domain} seems like it might be coming from a phabricator_domain setting.
• Within hieradata/cloud/eqiad1/devtools/common.yaml, it seems like - for the test instance (half-assuming that that file is where the test instance's Puppet config lives) - the phabricator_domain setting is being set to phabricator.wmcloud.org.
• So, to me, that then makes sense as to why - with the current config - Phabricator is trying to send emails as no-reply@phabricator.wmcloud.org.
• Maybe one way of fixing this might be to introduce some additional code within phabricator/main.pp that makes it possible to override the value it sets for metamta.default-address, and that defaults back to no-reply@${domain} when no such override is provided. But figuring out how to do this sort of thing would then be going beyond the bounds of my personal Puppet knowledge :)

[0] Actually, looking again on this a few days later (2026-03-20), it seems like the metamta.default-address config setting might actually be being read from a mail_default_address setting (link) within config_deploy_vars. Huh. I'm not sure why (what seems to be) the same config setting is defined twice within phabricator/main.pp ¯\_(ツ)_/¯

Change #1256301 had a related patch set uploaded (by A smart kitten; author: A smart kitten):

[operations/puppet@production] phabricator: Set a custom default-mail-address for the test instance

https://gerrit.wikimedia.org/r/1256301

Change #1256301 merged by Dzahn:

[operations/puppet@production] phabricator: Set a custom default-mail-address for the test instance

https://gerrit.wikimedia.org/r/1256301

Thanks @A_smart_kitten I merged and deployed the change and have confirmed there was no change or error on the production host.

Waiting for the local project puppetserver to sync with that.

Noticed the puppet change was not applied on the test instance.

Found out the local puppetmaster (puppetmaster-1003.devtools) that is still used for this instance (could be challenged why not just use the default master) was not updated and stuck at a previous commit from quite some time ago.

Reset the local git repo on the master. Then could confirm after another puppet agent run your change was actually applied as in:

grep mail_default /etc/phabricator/config.yaml 
    mail_default_address: phabricator-no-reply@wmcloud.org

As we already talked about on IRC, this probably needs scap now to merge the config in from /etc/phabricator/

I tried to do the scap deploy:

debug1: Server host key: ...
Host key verification failed.

17:42:34 connection to phabricator-bullseye.devtools.eqiad1.wikimedia.cloud failed and future stages will not be attempted for this target
17:42:34 phabricator/deployment: fetch stage(s): 100% (in-flight: 0; ok: 0; fail: 1; left: 0) 
17:42:34 1 targets had deploy errors
17:42:34 1 targets failed
17:42:34 1 of 1 default targets failed, exceeding limit
**Rollback all deployed groups? [Y/n]: Y
Invalid choice: 'Y'**

@brennen Could you do a scap deploy to the test instance? See above how I tried to Rollback but "Y" was an invalid choice.

@brennen Could you do a scap deploy to the test instance? See above how I tried to Rollback but "Y" was an invalid choice.

Yep, running now. One moment...

Thank you, @brennen

I see "mail_default_address: phabricator-no-reply@wmcloud.org" in /etc/phabricator/config.yaml and scap has deployed.

I have not received test email.

Thank you for reviewing/merging (& fixing the puppetmaster issue) @Dzahn, & thank you for doing the scap deploy @brennen :)

Looking at https://phabricator.wmcloud.org/config/edit/metamta.default-address/ now, it now says that the 'Local Config' value is phabricator-no-reply@wmcloud.org. So it seems like it is definitely a success at least in terms of getting the test-instance Phabricator to have this updated config.

To check whether the test-instance can now send emails (to me at least), I created https://phabricator.wmcloud.org/T278 just now. (I have 'Self Action Mail' enabled in my test-instance 'Email Delivery' preferences, so it should send me an email about a task I create).
And..... it sent me an email for it!

In T388022#11783814, @Dzahn wrote:

I tried to do the scap deploy:

debug1: Server host key: ...
Host key verification failed.

17:42:34 connection to phabricator-bullseye.devtools.eqiad1.wikimedia.cloud failed and future stages will not be attempted for this target
17:42:34 phabricator/deployment: fetch stage(s): 100% (in-flight: 0; ok: 0; fail: 1; left: 0) 
17:42:34 1 targets had deploy errors
17:42:34 1 targets failed
17:42:34 1 of 1 default targets failed, exceeding limit
**Rollback all deployed groups? [Y/n]: Y
Invalid choice: 'Y'**

Interesting. What host did you run scap on? What does scap version report there?

In T388022#11784516, @dancy wrote:

Interesting. What host did you run scap on? What does scap version report there?

I ran this on deploy-1006.devtools.eqiad1.wikimedia.cloud as scap deploy -v -l 'phabricator-bullseye.devtools.eqiad1.wikimedia.cloud' 'devtools test phab deploy'

scap version there reports as 4.79.0-1.

In T388022#11784455, @A_smart_kitten wrote:

And..... it sent me an email for it!

Wow! Cool. Somehow it did not work for me when I tried to test by creating a new user.

This is great to hear! Sounds resolved, then. Thanks for providing the patch.

In T388022#11784803, @Dzahn wrote:

This is great to hear! Sounds resolved, then.

Hopefully, yeah! If possible though I'd like to check that at least one other person can also receive emails, so that we know it's not something that's only working for me :)

Somehow it did not work for me when I tried to test by creating a new user.

Hmm, weird. I used the button within Phabricator to send a 'welcome email' to the accounts https://phabricator.wmcloud.org/p/testuser/ & https://phabricator.wmcloud.org/p/Dzahn/ just now. Let me know if you got either of them :)

In T388022#11783814, @Dzahn wrote:

I tried to do the scap deploy:

debug1: Server host key: ...
Host key verification failed.

17:42:34 connection to phabricator-bullseye.devtools.eqiad1.wikimedia.cloud failed and future stages will not be attempted for this target
17:42:34 phabricator/deployment: fetch stage(s): 100% (in-flight: 0; ok: 0; fail: 1; left: 0) 
17:42:34 1 targets had deploy errors
17:42:34 1 targets failed
17:42:34 1 of 1 default targets failed, exceeding limit
**Rollback all deployed groups? [Y/n]: Y
Invalid choice: 'Y'**

This invalid choice bug was fixed a long time ago in scap but deploy1006 was running an old version. I upgraded it today (T422257).

Ah, thank you very much, @dancy :)

Thanks @Pppery -- I am happy declaring this to be fixed for now :)
Since the fix was deployed (thanks again @Dzahn & @brennen!), https://phabricator.wmcloud.org/people/query/.2Jw6OEOnYXn/ shows that two accounts have been created on the test-instance, and both of them are marked as 'Verified' -- indicating (IIUC) that a confirmation email has got through to them, and that the link within it has been clicked.

I am unsure why @Dzahn may not have received any emails from the test-instance (T388022#11783879), though. I'm not yet sure whether this is the case for e.g. all email addresses on the @wikimedia.org domain or not.
I'm aware that the Beta Cluster has had problems sending to @wikimedia.org emails in the past (T212327: Beta Cluster mailer not sending emails to @wikimedia.org addresses) -- h/t p858snake in #wikimedia-tech -- so maybe it might be related to a (possible) more general Cloud VPS → @wikimedia.org addresses email deliverability issue? Either way, I guess it might be hard to know for sure without more testing (and it seems like it might be a technically distinct issue from the one that was causing problems in this task).

If anyone has any issues receiving emails from the Phabricator test instance, feel free to comment on this task or create a new one :)

As some cleanup-steps, I have:

• Removed the note on https://www.mediawiki.org/wiki/Phabricator that said the test-instance can't send emails & directed prospective users to fill in this form.
• Removed a similar note from https://www.mediawiki.org/wiki/Phabricator/Help.
• Edited the note on the test-instance's homepage so that it no longer says that the instance can't send emails. (It now says that it should be able to send emails, and lets folks know that they can leave a comment here/file a new task if they are still experiencing issues.)

@Dzahn/SRE collab, I will probably leave it for you to decide whether or not to keep/modify the text under https://wikitech.wikimedia.org/wiki/Phabricator#Verifying_a_user_on_the_test_instance, as that is probably more directed towards SREs than end-users.

Thank you very much @A_smart_kitten

I edited that slightly to phrase it in the past tense. Otherwise leaving the docs there just in case.

https://wikitech.wikimedia.org/w/index.php?title=Phabricator&diff=2399390&oldid=2387177

In T388022#11795547, @A_smart_kitten wrote:

I am unsure why @Dzahn may not have received any emails from the test-instance (T388022#11783879), though. I'm not yet sure whether this is the case for e.g. all email addresses on the @wikimedia.org domain or not.
I'm aware that the Beta Cluster has had problems sending to @wikimedia.org emails in the past (T212327: Beta Cluster mailer not sending emails to @wikimedia.org addresses) -- h/t p858snake in #wikimedia-tech -- so maybe it might be related to a (possible) more general Cloud VPS → @wikimedia.org addresses email deliverability issue? Either way, I guess it might be hard to know for sure without more testing (and it seems like it might be a technically distinct issue from the one that was causing problems in this task).

FWIW, I have now split this issue into its own task: T422559: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance