Page MenuHomePhabricator

php-fpm isn't restarted when deploys are rolled back
Closed, ResolvedPublic

Description

Today, I tried doing a deployment and then noticed error reports on the canaries and rolled back the deployment.
However, looks like php-fpm isn't restarted on a rollback, so I had to get ops to manually restart php-fpm7.2 on the canaries.

How do we fix this?

Details

Related Gerrit Patches:

Event Timeline

ssastry triaged this task as High priority.Tue, Nov 19, 7:13 PM
ssastry created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Nov 19, 7:13 PM
jijiki added a subscriber: jijiki.EditedTue, Nov 19, 10:34 PM

@ssastr php-fpm will be restarted during scap deployments only when a server's opcache free is below 100MB, I can check the code to see if there is an exception to that (eg canaries).

@ssastr php-fpm will be restarted during scap deployments only when a server's opcache free is below 100MB, I can check the code to see if there is an exception to that (eg canaries).

This is a special case for Parsoid/PHP because of Parsoid's use of scap3 that uses symlinks. See T236275: Parsoid-php doesn't get updated after a code deploy for context. This ticket is a followup for the rollback scenario.

jijiki moved this task from Backlog to Doing on the serviceops board.Wed, Nov 20, 6:54 PM

Change 552120 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/parsoid/deploy@master] Ensure FPM is depooled throughout the deployment process on a node

https://gerrit.wikimedia.org/r/552120

Change 552120 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Ensure FPM is depooled throughout the deployment process on a node

https://gerrit.wikimedia.org/r/552120

Mentioned in SAL (#wikimedia-operations) [2019-11-26T16:09:12Z] <ssastry@deploy1001> Started deploy [parsoid/deploy@ee63341]: Testing rollback fixes (T238685)

Mentioned in SAL (#wikimedia-operations) [2019-11-26T16:10:19Z] <ssastry@deploy1001> Finished deploy [parsoid/deploy@ee63341]: Testing rollback fixes (T238685) (duration: 01m 07s)

ssastry closed this task as Resolved.Tue, Nov 26, 4:34 PM

@mobrovac verified this worked. But, he found that servers aren't depooled during rollback. Will file a separate bug for that.