| • DAlangi_WMF | |
| Jan 6 2025, 1:05 PM |
| F96125642: image.png | |
| Mon, Jul 27, 3:19 PM |
| F96125632: image.png | |
| Mon, Jul 27, 3:19 PM |
| F76752056: image.png | |
| Apr 20 2026, 1:24 PM |
| F73157829: Screenshot 2026-03-19 at 13.13.47.png | |
| Mar 19 2026, 8:22 PM |
| F73157815: Screenshot 2026-03-19 at 13.07.19.png | |
| Mar 19 2026, 8:22 PM |
| F71895142: 76d527ee-0fbe-47bd-9623-6eea8649a2fa.jpeg | |
| Feb 12 2026, 12:10 PM |
| F71108731: Screenshot 2025-12-17 at 1.05.40 PM.png | |
| Dec 17 2025, 8:18 PM |
| F71108699: Screenshot 2025-12-17 at 12.50.25 PM.png | |
| Dec 17 2025, 8:18 PM |
Could not send confirmation email: Unknown error in PHP's mail() function.
exception.trace
Causing warnings during authentication (account creation).
Seems like this is related to: T325131: Setup sendmail on k8s container
Last 24 hours - 127 instances of message: Could not send confirmation email: Sendmail exited with non-zero exit code 74 - https://logstash.wikimedia.org/goto/1425bc3e85340cb5965e3757623deb8c
Saw the issue in the enwiki UI (non-reproducible):
It's level: WARNING but causes confirmation code to expire; a user need to start everything over.
Either do one of the debugging options @jhathaway mentioned in T383047#11075860, or add retry functionality.
The volume of this error has doubled after Jan 26-28 (Logstash permalink)
Hi all, I just experienced this error as well as a few community members have reached out that they did not receive an email from me during the Community Insights distribution yesterday (however, they did receive an echo ping to check their email). When I tried to send a message directly via EmailUser on the project to one of those community members just now to troubleshoot, I got the Sendmail exited with non-zero exit code 74 message. I'm unsure of the scale of the unsent emails as I didn't get a failure for these with the EmailUser API script I was using, but it should be visible in yesterday's log (if such a log exists, I presume it does per @Krinkle's last message but I seem to not have access).
This is still happening, ~3.3k per in last 4 weeks:
Currently, ~5k in the last 4 weeks https://logstash.wikimedia.org/goto/ebe07e84765c3f2896ffc26e64e24166
What would be the next steps? I assume this error is something that can happen - sendmail can fail due to various reasons. @Tgr - do we do any kind of retries here?
I'll post a patch for an automatic retry.
Change #1275406 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/core@master] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1275407 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/core@master] Mail: Retry transient mail() failures in UserMailer
I was sending a single message and got that error too. Just noting that re-sending seemed to work - at least technically, can't say if the user actually got the message.
On April 29th, I also encountered the same "Sendmail exited with non-zero exit code 74" on the Estonian Wikipedia. The user to whom I sent the mail mentioned today that he got the letter, but not the notification on Wikipedia.
Change #1305272 had a related patch set uploaded (by Reedy; author: CDanis):
[mediawiki/core@REL1_43] UserMailer: hack: preserve multiple error callbacks
Change #1305272 merged by jenkins-bot:
[mediawiki/core@REL1_43] UserMailer: hack: preserve multiple error callbacks
Change #1305615 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/core@master] Mail: Log PHP mail() send failures with recipient count
Change #1275406 merged by jenkins-bot:
[mediawiki/core@master] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305699 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_46] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305713 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_45] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305715 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_44] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305699 merged by jenkins-bot:
[mediawiki/core@REL1_46] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305716 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_43] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305713 merged by jenkins-bot:
[mediawiki/core@REL1_45] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305715 merged by jenkins-bot:
[mediawiki/core@REL1_44] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305716 merged by jenkins-bot:
[mediawiki/core@REL1_43] Mail: Extract sendWithMailFunction() from UserMailer::send()
Change #1305615 merged by jenkins-bot:
[mediawiki/core@master] Mail: Log PHP mail() send failures with recipient count
Change #1305891 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_46] Mail: Log PHP mail() send failures with recipient count
Change #1305892 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_45] Mail: Log PHP mail() send failures with recipient count
Change #1305894 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_44] Mail: Log PHP mail() send failures with recipient count
Change #1305895 had a related patch set uploaded (by Reedy; author: Kosta Harlan):
[mediawiki/core@REL1_43] Mail: Log PHP mail() send failures with recipient count
Change #1305894 merged by jenkins-bot:
[mediawiki/core@REL1_44] Mail: Log PHP mail() send failures with recipient count
Change #1305892 merged by jenkins-bot:
[mediawiki/core@REL1_45] Mail: Log PHP mail() send failures with recipient count
Change #1305891 merged by jenkins-bot:
[mediawiki/core@REL1_46] Mail: Log PHP mail() send failures with recipient count
Change #1305895 merged by jenkins-bot:
[mediawiki/core@REL1_43] Mail: Log PHP mail() send failures with recipient count
For what it's worth, I got this error today repeatedly: "Sendmail exited with non-zero exit code 74". I tried to email 2 different users.
I see several changes were merged recently. Does this imply that the errors should soon stop happening?
I'm able to reproduce. From my limit testing (about 5 times today), it seems that the sending of the main email succeeds (despite re-presenting the form (which normally implies failure) and despite displaying an error message), and the sending of the "send me a copy of this email" fails.
@Novem_Linguae if you're okay with being a test subject, would you like me to try sending you 2 emails, one without the "send me a copy" option checked and the other with that option checked?
Both emails sent, with different content:
Which did you get, if any?
I didn't receive copies of either, and in both cases I got "Sendmail exited with non-zero exit code 74"
On the bright side, this appears consistently reproducible in production today if an engineer wants to dig into it.
It seems to be happening much, much more often since 11:30 UTC today yesterday (oops).
logstash
Currently happening ~7 times per minute, which matches the rate of signups, so chances are emails are not working at all.
All exit code 74; maybe a disk filled up?
I created a test account. I did receive the email, but an error was logged nevertheless. So maybe msmtp is returning bogus codes? Was it upgraded recently?
Started around 11h on Sunday so I guess it cannot be a code change or package update.
SDZeroBot ran into this error 3 times today. The emails were in fact received according to the end-user. However, self-cc mails were not received and the bot got php-mail-error code from the API.
Change #1318259 had a related patch set uploaded (by Scott French; author: Scott French):
[operations/deployment-charts@master] mediawiki: Allow configurable msmtp timeout and bump to 5s
I'm able to consistently received the emails I send via Special:EmailUser while still receiving the Sendmail exited with non-zero exit code 74 error.
We were able to reproduce the issue in an environment were we can strace the php-fpm worker processes and their children, which surfaced that msmtp was consistently hitting the 1 second timeout waiting on a response from upstream. We're going to try bumping said timeout.
Change #1318259 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Allow configurable msmtp timeout and bump to 5s
Deployment completed around ~ 19:57 UTC (stashbot appears to be missing from -operations). Confirmed at the testservers stage that I can no longer reproduce the issue from mw-debug. Errors are dropping off: https://logstash.wikimedia.org/goto/aaae29458dde46a9bc25856765ccb64f
Now that the increased timeout is in place, we can perform the same strace exercise again to see how long it's taking for the upstream host (here mx-out1001) to respond.
Excerpt for the msmtp PID (note this was a multi-process strace, so the recvfrom is "split"):
2983755 20:05:59.392193 sendto(5, ".\r\n", 3, 0, NULL, 0) = 3 # end-of-data on my test email 2983755 20:05:59.392321 recvfrom(5, <unfinished ...> 2983755 20:06:03.547230 <... recvfrom resumed>"250 2.0.0 Ok: queued as 4h88jg2l45zMpLZ\r\n", 4096, 0, NULL, NULL) = 41 # mx-out1001 responds 2983755 20:06:03.547567 sendto(5, "QUIT\r\n", 6, 0, NULL, 0) = 6 2983755 20:06:03.548330 recvfrom(5, "221 2.0.0 Bye\r\n", 4096, 0, NULL, NULL) = 15 2983755 20:06:03.549246 close(5) = 0 2983755 20:06:03.549566 close(3) = 0 2983755 20:06:03.556980 exit_group(0) = ?
So, that's over 4s between end-of-data and acknowledgement. We should try to figure out why this is taking so long.
Dropping this from UBN to High now that that there's a mitigation in place.
Thank you! I confirm I have managed to send an email via the wikis. The page took a while to process the send (which I guess correspond to the 4 seconds delay)
There is some very strange behavior with regard to the message size, if I send a message of 343 bytes or less, there is no delay in queuing the message, however if the message size is 344 bytes or larger, there is about a four second delay:
just-right:
www-data@mw-web:/var/tmp$ cat > just-right <<EOF To: "Jesse" <jhathaway@wikimedia.org> Subject: MediaWiki email from user 0 From: "MediaWiki" <wiki@wikimedia.org> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXX EOF www-data@mw-web:/var/tmp$ time msmtp jhathaway@wikimedia.org <just-right real 0m0.078s user 0m0.000s sys 0m0.006s
too-big:
www-data@mw-web:/var/tmp$ cat >too-big <<EOF To: "Jesse" <jhathaway@wikimedia.org> Subject: MediaWiki email from user 0 From: "MediaWiki" <wiki@wikimedia.org> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX EOF www-data@mw-web:/var/tmp$ time msmtp jhathaway@wikimedia.org <too-big real 0m4.081s user 0m0.006s sys 0m0.000s
I ran tcpdump on mx-out1001.wikimedia.org and the network traffic seems to confirm what @Scott_French saw in the strace log, namely that postfix has a delayed 250 response when receiving the larger message.
Change #1318778 had a related patch set uploaded (by JHathaway; author: JHathaway):
[operations/puppet@production] mx: disable fuzzy_check for trusted networks
Change #1318778 merged by JHathaway:
[operations/puppet@production] mx: disable fuzzy_check for trusted networks
The 4s delay was do to Rspamd’s fuzzy_check module timing out when contacting its configuration servers. We did not intend for fuzzy_check to check email from trusted networks. Unlike other Rspamd modules it does not skip checking based on local_addrs or the email being marked as accept in a prefilter. This module has a separate IP whitelist field, which is now being used to disable checking.
Now that the underlying problem is resolved @Scott_French what are your thoughts on whether we should retain the timeout at 5s, or revert to the previous 1s timeout?
Very nice find, @jhathaway. Reverting the to the original 1s timeout (leaving ability to override it intact) sounds reasonable to me, though not a strong opinion. Depending on how things go tomorrow, I may have time to do that.
Apologies for that notable omission!
We have a mostly stock Rspamd config from Debian which has these config lines:
min_bytes = 1k; # Since small parts and small attachments causes too many FP timeout = 2s; retransmits = 1;
Postfix adds headers when it receives the message, prior to sending it to the Rspamd milter. This additional header data was evidently, 1000 - 344 = 656 bytes, with a message of 344 bytes or greater the minimum bytes threshold was reached and the fuzzy check was activated. When activated the fuzzy check reaches out to its config servers, but these requests starting timing out. The config timeout was 2 seconds, with one retry, giving us the 4 second delay we observed.
I don't have a strong opinion either, but I would probably lean towards reverting to a 1s timeout, to avoid an MX outage causing PHP worker starvation, though perhaps email sending is low volume enough that the risk is negligible?
Change #1319157 had a related patch set uploaded (by Scott French; author: Scott French):
[operations/deployment-charts@master] mw-*: Revert temporary mw.mail_timeout increase
Thanks for the thorough investigation!
The code 74 errors are now gone entirely. The (much less common) other types of errors are, unsurprisingly, still happening. In the last 7 days:
Change #1319157 merged by jenkins-bot:
[operations/deployment-charts@master] mw-*: Revert temporary mw.mail_timeout increase
Mentioned in SAL (#wikimedia-operations) [2026-08-04T17:45:55Z] <swfrench@deploy1003> Started scap sync-world: Deploy helmfile-only msmtp timeout override cleanup - T383047
Mentioned in SAL (#wikimedia-operations) [2026-08-04T17:46:56Z] <swfrench@deploy1003> swfrench: Deploy helmfile-only msmtp timeout override cleanup - T383047 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-08-04T17:50:00Z] <swfrench@deploy1003> Finished scap sync-world: Deploy helmfile-only msmtp timeout override cleanup - T383047 (duration: 04m 05s)
