Today, I tried doing a deployment and then noticed error reports on the canaries and rolled back the deployment.
However, looks like php-fpm isn't restarted on a rollback, so I had to get ops to manually restart php-fpm7.2 on the canaries.
How do we fix this?
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Ensure FPM is depooled throughout the deployment process on a node | mediawiki/services/parsoid/deploy | master | +15 -8 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ssastry | T229015 Tracking: Direct live production traffic at Parsoid/PHP | |||
| Resolved | • mobrovac | T238685 php-fpm isn't restarted when deploys are rolled back |
@ssastr php-fpm will be restarted during scap deployments only when a server's opcache free is below 100MB, I can check the code to see if there is an exception to that (eg canaries).
This is a special case for Parsoid/PHP because of Parsoid's use of scap3 that uses symlinks. See T236275: Parsoid-php doesn't get updated after a code deploy for context. This ticket is a followup for the rollback scenario.
Change 552120 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/parsoid/deploy@master] Ensure FPM is depooled throughout the deployment process on a node
Change 552120 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Ensure FPM is depooled throughout the deployment process on a node
Mentioned in SAL (#wikimedia-operations) [2019-11-26T16:09:12Z] <ssastry@deploy1001> Started deploy [parsoid/deploy@ee63341]: Testing rollback fixes (T238685)
Mentioned in SAL (#wikimedia-operations) [2019-11-26T16:10:19Z] <ssastry@deploy1001> Finished deploy [parsoid/deploy@ee63341]: Testing rollback fixes (T238685) (duration: 01m 07s)
@mobrovac verified this worked. But, he found that servers aren't depooled during rollback. Will file a separate bug for that.