I was rolling a change to the slowroll domains on the frmx's and it took place in the middle of a delivery run from civi1001. This change caused a restart of postfix and the message that was in flight had postfix terminate in the middle of the process: Let's see if it is possible to send a graceful restart / reload when we update main.cf to keep this from happening.
Civi side error
2020-11-11 00:14:01,579>INFO>---=========== 2020-11-11 00:14:01,579>INFO>---sleep 30 (26431) 2020-11-11 00:14:01,579>INFO>-------------- 2020-11-11 00:14:31,570>INFO>-------------- end command output 2020-11-11 00:14:31,585>INFO>---=========== 2020-11-11 00:14:31,585>INFO>---/usr/local/bin/drush --user=1 -v -r /srv/org.wikimedia.civicrm/drupal thank-you-send (27452) 2020-11-11 00:14:31,585>INFO>-------------- 2020-11-11 00:14:31,806>ERROR>--Executing: mysql --defaults-extra-file=/tmp/drush_Y1HEWg --database=drupal --host=fundraisingdb-write.wmnet --silent < /tmp/drush_2g82Hb 2020-11-11 00:14:31,821>ERROR>--Executing: mysql --defaults-extra-file=/tmp/drush_aoibvr --database=drupal --host=fundraisingdb-write.wmnet --silent < /tmp/drush_macUdm 2020-11-11 00:14:32,065>ERROR>--WD thank_you: Attempting to send 1350 thank you mails for [info] <snip> 2020-11-11 00:16:56,139>ERROR>--Drush command terminated abnormally due to an unrecoverable error. [error] 2020-11-11 00:16:56,139>INFO>---***ERRORS*** 2020-11-11 00:16:56,139>INFO>---EMAIL_SYSTEM_FAILURE:- 2020-11-11 00:16:56,139>INFO>--- EMAIL_SYSTEM_FAILURE UNHANDLED PHPMAILER EXCEPTION SENDING THANK YOU MESSAGE 2020-11-11 00:16:56,140>INFO>---thank_you_send_mail 2020-11-11 00:16:56,140>INFO>--- 2020-11-11 00:16:56,140>INFO>---<strong>SMTP Error: data not accepted.</strong><br /> 2020-11-11 00:16:56,140>INFO>--- 2020-11-11 00:16:56,140>INFO>--- 2020-11-11 00:16:56,140>INFO>---#0 /srv/org.wikimedia.civicrm/vendor/phpmailer/phpmailer/class.phpmailer.php(1340): PHPMailer->smtpSend('Date: Tue, 10 N...', 'This is a multi...') 2020-11-11 00:16:56,140>INFO>---#1 /srv/org.wikimedia.civicrm/vendor/phpmailer/phpmailer/class.phpmailer.php(1215): PHPMailer->postSend() 2020-11-11 00:16:56,140>INFO>---#2 /srv/org.wikimedia.civicrm/sites/default/civicrm/extensions/org.wikimedia.omnimail/Civi/Omnimail/SMTPMailer.php(82): PHPMailer->send() 2020-11-11 00:16:56,140>INFO>---#3 /srv/org.wikimedia.civicrm/sites/default/civicrm/extensions/org.wikimedia.omnimail/Civi/Omnimail/MailFactory.php(93): Civi\Omnimail\SMTPMailer->send(Array, Array) 2020-11-11 00:16:56,140>INFO>---#4 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(548): Civi\Omnimail\MailFactory->send(Array, Array) 2020-11-11 00:16:56,140>INFO>---#5 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(432): thank_you_send_mail(Array) 2020-11-11 00:16:56,140>INFO>---#6 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(851): thank_you_for_contribution('52475447') 2020-11-11 00:16:56,140>INFO>---#7 /srv/org.wikimedia.civicrm/drupal/includes/module.inc(934): thank_you_batch_process() 2020-11-11 00:16:56,140>INFO>---#8 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.drush.inc(41): module_invoke('thank_you', 'batch_process') 2020-11-11 00:16:56,140>INFO>---#9 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(422): drush_thank_you_send() 2020-11-11 00:16:56,140>INFO>---#10 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(231): _drush_invoke_hooks(Array, Array) 2020-11-11 00:16:56,140>INFO>---#11 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(199): drush_command() 2020-11-11 00:16:56,140>INFO>---#12 /srv/org.wikimedia.civicrm/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch(Array) 2020-11-11 00:16:56,140>INFO>---#13 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch() 2020-11-11 00:16:56,140>INFO>---#14 /srv/org.wikimedia.civicrm/vendor/drush/drush/drush.php(12): drush_main() 2020-11-11 00:16:56,140>INFO>---#15 {main}
Corresponding restart of postfix:
Nov 11 00:15:54 frmx1001 postfix/postfix-script[17195]: stopping the Postfix mail system Nov 11 00:15:54 frmx1001 postfix/master[5107]: terminating on signal 15 Nov 11 00:15:55 frmx1001 postfix/postfix-script[17317]: warning: symlink leaves directory: /etc/postfix/./makedefs.out Nov 11 00:15:55 frmx1001 postfix/postfix-script[17356]: starting the Postfix mail system Nov 11 00:15:55 frmx1001 postfix/master[17358]: daemon started -- version 3.4.14, configuration /etc/postfix