Page MenuHomePhabricator

Adjust postfix to do graceful restarts when possible
Closed, ResolvedPublic

Description

I was rolling a change to the slowroll domains on the frmx's and it took place in the middle of a delivery run from civi1001. This change caused a restart of postfix and the message that was in flight had postfix terminate in the middle of the process: Let's see if it is possible to send a graceful restart / reload when we update main.cf to keep this from happening.

Civi side error

2020-11-11 00:14:01,579>INFO>---===========
2020-11-11 00:14:01,579>INFO>---sleep 30 (26431)
2020-11-11 00:14:01,579>INFO>--------------
2020-11-11 00:14:31,570>INFO>-------------- end command output
2020-11-11 00:14:31,585>INFO>---===========
2020-11-11 00:14:31,585>INFO>---/usr/local/bin/drush --user=1 -v -r /srv/org.wikimedia.civicrm/drupal thank-you-send (27452)
2020-11-11 00:14:31,585>INFO>--------------
2020-11-11 00:14:31,806>ERROR>--Executing: mysql --defaults-extra-file=/tmp/drush_Y1HEWg --database=drupal --host=fundraisingdb-write.wmnet --silent  < /tmp/drush_2g82Hb
2020-11-11 00:14:31,821>ERROR>--Executing: mysql --defaults-extra-file=/tmp/drush_aoibvr --database=drupal --host=fundraisingdb-write.wmnet --silent  < /tmp/drush_macUdm
2020-11-11 00:14:32,065>ERROR>--WD thank_you: Attempting to send 1350 thank you mails for                 [info]

<snip>

2020-11-11 00:16:56,139>ERROR>--Drush command terminated abnormally due to an unrecoverable error.       [error]
2020-11-11 00:16:56,139>INFO>---***ERRORS***
2020-11-11 00:16:56,139>INFO>---EMAIL_SYSTEM_FAILURE:-
2020-11-11 00:16:56,139>INFO>---    EMAIL_SYSTEM_FAILURE UNHANDLED PHPMAILER EXCEPTION SENDING THANK YOU MESSAGE
2020-11-11 00:16:56,140>INFO>---thank_you_send_mail
2020-11-11 00:16:56,140>INFO>---
2020-11-11 00:16:56,140>INFO>---<strong>SMTP Error: data not accepted.</strong><br />
2020-11-11 00:16:56,140>INFO>---
2020-11-11 00:16:56,140>INFO>---
2020-11-11 00:16:56,140>INFO>---#0 /srv/org.wikimedia.civicrm/vendor/phpmailer/phpmailer/class.phpmailer.php(1340): PHPMailer->smtpSend('Date: Tue, 10 N...', 'This is a multi...')
2020-11-11 00:16:56,140>INFO>---#1 /srv/org.wikimedia.civicrm/vendor/phpmailer/phpmailer/class.phpmailer.php(1215): PHPMailer->postSend()
2020-11-11 00:16:56,140>INFO>---#2 /srv/org.wikimedia.civicrm/sites/default/civicrm/extensions/org.wikimedia.omnimail/Civi/Omnimail/SMTPMailer.php(82): PHPMailer->send()
2020-11-11 00:16:56,140>INFO>---#3 /srv/org.wikimedia.civicrm/sites/default/civicrm/extensions/org.wikimedia.omnimail/Civi/Omnimail/MailFactory.php(93): Civi\Omnimail\SMTPMailer->send(Array, Array)
2020-11-11 00:16:56,140>INFO>---#4 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(548): Civi\Omnimail\MailFactory->send(Array, Array)
2020-11-11 00:16:56,140>INFO>---#5 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(432): thank_you_send_mail(Array)
2020-11-11 00:16:56,140>INFO>---#6 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.module(851): thank_you_for_contribution('52475447')
2020-11-11 00:16:56,140>INFO>---#7 /srv/org.wikimedia.civicrm/drupal/includes/module.inc(934): thank_you_batch_process()
2020-11-11 00:16:56,140>INFO>---#8 /srv/org.wikimedia.civicrm/sites/all/modules/thank_you/thank_you.drush.inc(41): module_invoke('thank_you', 'batch_process')
2020-11-11 00:16:56,140>INFO>---#9 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(422): drush_thank_you_send()
2020-11-11 00:16:56,140>INFO>---#10 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(231): _drush_invoke_hooks(Array, Array)
2020-11-11 00:16:56,140>INFO>---#11 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/command.inc(199): drush_command()
2020-11-11 00:16:56,140>INFO>---#12 /srv/org.wikimedia.civicrm/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch(Array)
2020-11-11 00:16:56,140>INFO>---#13 /srv/org.wikimedia.civicrm/vendor/drush/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
2020-11-11 00:16:56,140>INFO>---#14 /srv/org.wikimedia.civicrm/vendor/drush/drush/drush.php(12): drush_main()
2020-11-11 00:16:56,140>INFO>---#15 {main}

Corresponding restart of postfix:

Nov 11 00:15:54 frmx1001 postfix/postfix-script[17195]: stopping the Postfix mail system
Nov 11 00:15:54 frmx1001 postfix/master[5107]: terminating on signal 15
Nov 11 00:15:55 frmx1001 postfix/postfix-script[17317]: warning: symlink leaves directory: /etc/postfix/./makedefs.out
Nov 11 00:15:55 frmx1001 postfix/postfix-script[17356]: starting the Postfix mail system
Nov 11 00:15:55 frmx1001 postfix/master[17358]: daemon started -- version 3.4.14, configuration /etc/postfix

Event Timeline

Dwisehaupt renamed this task from Adjust postfix to go graceful restarts when possible to Adjust postfix to do graceful restarts when possible.Nov 11 2020, 12:39 AM
Dwisehaupt claimed this task.
Dwisehaupt moved this task from Backlog to Done on the fundraising-tech-ops board.

Pushed change out to support reload:

[frack::puppet] 8c3f4b64 Add reload function to postfix service