Hopefully no data loss as people's sent email shouldn't just disappear from their client/server. BUT, this isn't good.

@Dzahn anything from the move/reimage that could have caused this?
@mmodell help? :)

@greg this is not related to the reimage :) (i know why it's happening, it's due to upstream removal of some old configs) We never moved the "reply to email" config to the new syntax. I only moved it to allow the continuing of emails being sent. (https://github.com/wikimedia/puppet/commit/d45a46d7ee7fefc2552c03e2bb08c21667191341#diff-f0958b3f2f1547cbbd8476eaf38f82fc) (since i didn't know how to setup phab to receive email using the new syntax)

See https://secure.phabricator.com/book/phabricator/article/configuring_outbound_email/#configuring-mailers

In T224752#5227524, @Paladox wrote:

@greg this is not related to the reimage :) (i know why it's happening, it's due to upstream removal of some old configs) We never moved the "reply to email" config to the new syntax. I only moved it to allow the continuing of emails being sent. (https://github.com/wikimedia/puppet/commit/d45a46d7ee7fefc2552c03e2bb08c21667191341#diff-f0958b3f2f1547cbbd8476eaf38f82fc) (since i didn't know how to setup phab to receive email using the new syntax)

See https://secure.phabricator.com/book/phabricator/article/configuring_outbound_email/#configuring-mailers

I don't think this is true. As far as I can tell it's totally related to the server migration because the last email I see in the queue was pre-migration and because there hasn't been a code deployment since before the server migration.

@Dzahn, any ideas? I've been poking at it and I can't see anything that's wrong so far.

I can't seem to find where mx1001 / mx2001 are configured to forward mail?

from /var/log/exim4/mainlog:

2019-05-31 22:30:20 Warning: No server certificate defined; will use a selfsigned one.

I'm not sure if the above problem is the entire issue? It seems like it should be working.

Test

As far as I can tell everything seems to be configured correctly but the messages just disappear. I suspect that something changed with the exim version on stretch?

Note that I've gone through all the upstream troubleshooting steps, however, we have a custom exim setup which is not at all supported by upstream and I wasn't involved in setting any of it up so it's pretty much a mystery to me.

adding @chasemp as he was the one who set up phab_epipe.py as far as I remember.

@mmodell could it be because we don't have php-mailparse installed when on stretch?

We installed php-mailparse when using php5 though.

Change 513713 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] phabricator: Install php-mailparse

https://gerrit.wikimedia.org/r/513713

hmm

php-mailparse is already the newest version (3.0.2+2.1.6-12-gae1ef14-3+0~20180910132529.4+stretch~1.gbpcad4a8+wmf1)

@mmodell Yup, it may be installed but the php class manages the mods (removes ones that are not maintained by the class).

For example if we removed:

'mysqlnd':
    package_name => 'php7.2-mysqlnd',
    sapis        => ['cli', 'fpm'],

puppet will remove /etc/php/7.2/mods-available/mysqlnd.ini.

Now i've just installed php-mailparse, which i found here /etc/php/7.2/mods-available/mailparse.ini

after running puppet i got:

Notice: /Stage[main]/Php/File[/etc/php/7.2/cli/conf.d/25-mailparse.ini]/ensure: removed
Notice: /Stage[main]/Php/File[/etc/php/7.2/fpm/conf.d/25-mailparse.ini]/ensure: removed

test test

test again!

one more time

ok it looks like it works.. https://gerrit.wikimedia.org/r/513713 should be the fix (Applied manually on phab1003)

@Paladox: nice catch noticing that the mailparse module wasn't enabled on fpm in stretch!

rOPUPc23fe1fc58ac: phabricator: Install php-mailparse

Change 513713 merged by Fsero:
[operations/puppet@production] phabricator: Install php-mailparse

https://gerrit.wikimedia.org/r/513713

merged and applied

Change 514742 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Increase priority of php-mailparse

https://gerrit.wikimedia.org/r/514742

Change 514742 merged by Fsero:
[operations/puppet@production] Increase priority of php-mailparse

https://gerrit.wikimedia.org/r/514742

testing reply via email.

nice fix! sorry, i was off for quite some time and reading this now. indeed a good catch that we did not catch during the migration. thanks all!