This gives an error about /usr/bin/env: 'python': No such file or directory so it's probably been broken since the upgrade two weeks ago.
Description
Description
Details
Details
Customize query in gerrit
Related Objects
Related Objects
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
This feature might have been non-operational even before the upgrade. It's not advertised and we don't know of any complaints. Worth looking into but not high priority.
Comment Actions
Change 993759 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):
[operations/puppet@production] [phabricator] Fix commenting on tasks by email
Comment Actions
Change 993766 had a related patch set uploaded (by Paladox; author: Paladox):
[operations/puppet@production] Phabricator: switch python to python3 in phab_epipe
Comment Actions
Change 993766 abandoned by Paladox:
[operations/puppet@production] Phabricator: switch python to python3 in phab_epipe
Reason:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/993759
Comment Actions
Change 993759 merged by EoghanGaffney:
[operations/puppet@production] [phabricator] Fix commenting on tasks by email
Comment Actions
The patch did get the script to run, but there seems to be an error from Phabricator in addition to this:
[2024-01-29 19:28:10] EXCEPTION: (RuntimeException) Undefined index: subject at [<arcanist>/src/error/PhutilErrorHandler.php:251] arcanist(), ava(), phorge(), translations(), wmf-ext-misc() #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phorge>/scripts/mail/mail_handler.php:58]
Comment Actions
What's documented is creating tasks by email, which also fails but in a different way:
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
general@phabricator.wikimedia.org
(generated from task@phabricator.wikimedia.org)
local delivery failed
The following text was generated during the delivery attempt:
------ general@phabricator.wikimedia.org
(generated from task@phabricator.wikimedia.org) ------
/usr/local/bin/phab_epipe.py:57: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
parser = configparser.SafeConfigParser()
Error parsing message
Please contact a WMF Phabricator admin.
[2024-02-06 15:22:26] EXCEPTION: (RuntimeException) Undefined index: subject at [<arcanist>/src/error/PhutilErrorHandler.php:251]
arcanist(), ava(), phorge(), translations(), wmf-ext-misc()
#0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phorge>/scripts/mail/mail_handler.php:58]Comment Actions
/usr/local/bin/phab_epipe.py:57: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
Fixed by rOPUP1f87f1edb83b535ac424a793d8fe0ef848f670ee merged today.
Please contact a WMF Phabricator admin.
I'm curious if upstream https://we.phorge.it/rPa69db10c5e15be4b7d851099b48c30f7cfe11046 and https://we.phorge.it/rP186768ccfd8895af80137bc3c6f41281ed744cfc and https://we.phorge.it/rP644f179dd27a78b91c88366b9432f5faaeb2bbb4 might change anything once deployed here, given the script is using stdin
Comment Actions
Still an issue after deploying rOPUP1f87f1edb83b535ac424a793d8fe0ef848f670ee...should check again after pulling a newer upstream (see previous comment) to have at least more correct error output than now:
Error parsing message Please contact a WMF Phabricator admin. [2024-04-29 12:42:57] EXCEPTION: (RuntimeException) Undefined index: subject at [<arcanist>/src/error/PhutilErrorHandler.php:263] arcanist(), ava(), phorge(), translations(), wmf-ext-misc() #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phorge>/scripts/mail/mail_handler.php:58]
Comment Actions
After we deployed https://we.phorge.it/rP644f179dd27a78b91c88366b9432f5faaeb2bbb4 as part of T368453 on 2024-07-16, now the (less misleading) error message is:
[2024-09-04 16:45:35] EXCEPTION: (Exception) MimeMailParser::getHeaders() could not parse any email headers. at [<phorge>/externals/mimemailparser/MimeMailParser.class.php:314] arcanist(), ava(), phorge(), translations(), wmf-ext-misc() #0 MimeMailParser::getPartHeaders(array) called at [<phorge>/externals/mimemailparser/MimeMailParser.class.php:218] #1 MimeMailParser::getMessageBody(string) called at [<phorge>/scripts/mail/mail_handler.php:40]
I guess I should look into this a bit more, thus assigning to me.
Comment Actions
After looking into the bigger picture here, I opened a rabbit hole in https://we.phorge.it/T15940 (ancient external library code)
Comment Actions
Could not reproduce locally: Took my returned (Mail delivery failed) test message with all its headers, saved it as tmp.mbox, locally changed https://we.phorge.it/source/phorge/browse/master/scripts/mail/mail_handler.php$36 to $parser->setText(file_get_contents('./tmp.mbox'));, and ran php ./mail_handler.php.
Also tried with a bunch more complex testcase emails, no luck.
Comment Actions
aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/67
Add temporary debug output for T356077
Comment Actions
aklapper merged https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/67
Add temporary debug output for T356077
This comment was removed by Aklapper.
Comment Actions
Trying Phab's inbound mail simulation server side, all looks fine:
aklapper@phab1004:/srv/phab/phabricator$ sudo -s
root@phab1004:/srv/deployment/phabricator/deployment-cache/revs/84ada671d34e5fce4756e8c11d96d613ec66f623/phabricator# ./bin/mail receive-test --as aklapper --to T356077 < /home/aklapper/T356077.mbox
Reading message body from stdin...
Mail received! You can view details by running this command:
phabricator/ $ ./bin/mail show-inbound --id 12904
root@phab1004:/srv/deployment/phabricator/deployment-cache/revs/84ada671d34e5fce4756e8c11d96d613ec66f623/phabricator# ./bin/mail show-inbound --id 12904
[...]I'm surprised though that there is only my one single test entry in Phab's inbound list and nothing else (on my local instance it lists more history) though I did send an incoming email within the last two hours:
root@phab1004:/srv/deployment/phabricator/deployment-cache/revs/84ada671d34e5fce4756e8c11d96d613ec66f623/phabricator# ./bin/mail list-inbound 12904 Aklapper T356077 (No subject.)
Comment Actions
Very likely also a red herring: Is that Final-Recipient line in the attachment.dat file attached to the mail bounce message I got just informative and concatenating some output, or is that mailto: address actually trying to get used somewhere with that invalid rfc822; prefix?:
Reporting-MTA: dns; phab1004.eqiad.wmnet Action: failed Final-Recipient: rfc822;task@phabricator.wikimedia.org Status: 5.0.0
For the records, https://phabricator.wikimedia.org/applications/panel/PhabricatorManiphestApplication/email/ defines task@phabricator.wikimedia.org
Comment Actions
Only realizing now, after writing the last two comments, that receive-test unexpectedly did add the comment before them (which did not get parsed at all)...
Comment Actions
aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/69
Add even more temporary debug output for T356077
Comment Actions
brennen merged https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/69
Add even more temporary debug output for T356077
Comment Actions
I come to the conclusion that this is not a bug in Phabricator code, thus unassigning myself.
For some reason, the content of php://stdin is empty in our setup, so I assume the email does not "reach" php://stdin.
Anyone having ideas? IIRC we use Exim?
Longer version:
I deployed more debug spew output to compare remote (WMF) and local (my machine where things work).
The first call (setText($data)) already yields different results: if ($data === '' || !$data) is false locally but true remotely.
That's a simple $parser = new MimeMailParser(); $parser->setText(file_get_contents('php://stdin')); in mail_handler.php:
https://phabricator.wikimedia.org/source/phabricator/browse/wmf%252Fstable/scripts/mail/mail_handler.php$35-36
I initially wondered if it's a red herring but it isn't: strlen(parts[$part_id]) later outputs 220 which is exactly the length I get when locally feeding a zero-byte empty "email" file locally into $parser->setText(file_get_contents('php://stdin')) via more ./empty.mbox | php ./mail_handler.php
What initially confused me is later in getPartHeaders(), if (isset($part['headers'])) is true locally and remotely, but if ($part['headers']) is true locally but false remotely, which also implied that we get an empty string for headers but that the headers array key does exist.
Turns out that PHP's undocumented mailparse_msg_get_part() blindly creates the array structure, no matter if it got any meaningful stdin input or not.
Using more ./proper_email.mbox | php ./mail_handler.php works as expected for me locally (Fedora 40, SELinux enabled, PHP 8.3).
Comment Actions
IIRC we use Exim?
Yes, there is an exim config on the phab server and it has a
"domainlist phab_domains = phabricator.wikimedia.org"
Also on one of the central MX servers there is config that says "if mail goes to @phabricator.wikimedia.org then route it to phabricator".
I would suspect the issue might be around this part:
modules/phabricator/manifests/mailrelay.pp: file { '/usr/local/bin/phab_epipe.py':that phab_epipe.py file plays an important role.
"""2014 Chase Pettet <cpettet@wikimedia.org> This script acts as an intermediary between an MTA and phabricator providing some extra functionality and raw email piping.
Comment Actions
Thanks for the reminder about the existence of epipe.py. Running more T356077.mbox | python ./phab_epipe.py locally with Python 3.12.6, I get Error parsing message: a bytes-like object is required, not 'str'.
Not knowing Python, I found https://discuss.python.org/t/popen-in-very-old-python2-script-having-issues/32182 and https://stackoverflow.com/questions/44989808/subprocess-typeerror-a-bytes-like-object-is-required-not-str , and setting an explicit encoding on Popen makes that error and the output line Please contact a WMF Phabricator admin. go away (though I obviously cannot emulate the entire mail setup so I cannot confirm this is a complete fix).
Comment Actions
Change #1077709 had a related patch set uploaded (by Aklapper; author: Aklapper):
[operations/puppet@production] Phabricator: Make Popen constructor in phab_epipe.py return strings
Comment Actions
Change #1077709 merged by Dzahn:
[operations/puppet@production] Phabricator: Make Popen constructor in phab_epipe.py return strings
Comment Actions
@Aklapper surprise, it works :) comment above was a response to my own comment.
I had to
- enable email in general in my user profile
- enable that I also get email for my own actions
- set notification settings for "someone comments on ticket"
Then I got an email, and even though that comes from noreply@ it's not actually noreply.
You hit reply and it goes to a special address including the ticket number and it does show up here as comment.
Thanks a lot for debugging this!
Comment Actions
Uh nice, task creation in T376411 also worked out. "Even a blind chicken finds a grain at times", as they say here. :)
Comment Actions
It's kind of confusing that the Phabricator emails come from: <no-reply@ when you actually CAN reply and there is a working reply-to address as well.
Should we change the sender address?
Comment Actions
aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/72
Revert temporary debug output for T356077
Comment Actions
I'd say No, as you cannot reply to that very From address, and as it would very likely break things (people's mail filter rules etc) for little gain.
Comment Actions
aklapper merged https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/72
Revert temporary debug output for T356077
Comment Actions
I'd say No, as you cannot reply to that very From address, and as it would very likely break things (people's mail filter rules etc) for little gain.
Yeah, breaking filters is the main thing I'd worry about, I guess. Most clients should handle the reply-to ok, I think?