Mentioned in SAL (#wikimedia-operations) [2022-07-25T21:20:50Z] <brennen> running a no-op sync-world for T313770 to hopefully get 1.39.0-wmf.21 (T308074) to all servers.

Mentioned in SAL (#wikimedia-operations) [2022-07-25T21:24:21Z] <brennen@deploy1002> Started scap: no-op deploy to get wmf.21 on all boxen (T313770)

Mentioned in SAL (#wikimedia-operations) [2022-07-25T21:27:55Z] <brennen@deploy1002> Finished scap: no-op deploy to get wmf.21 on all boxen (T313770) (duration: 03m 33s)

confirmed /srv/mediawiki/php-1.39.0-wmf.21 exists on every single appserver

it has been pointed out on IRC this affects only the canary servers

canary servers have been used for testing a new PHP version

Noting also that wikiversions.json is up to date on the canaries.

Something related to T311386: Install php 7.4 in production?

Further from notes from IRC (thanks Zabe and Dzahn for investigation):

• Errors are all PHP 7.2
• Canaries just not getting restarted?
  • Something in a recent Scap change?

Re: scap changes, maybe something related to 78cc0f85? @jnuche - any thoughts there?

Mentioned in SAL (#wikimedia-operations) [2022-07-26T00:11:06Z] <TimStarling> restarted php7.2-fpm on the 9 canary hosts in eqiad T313770

I confirmed that mw1450 was affected and was reporting wmf.19 in Special:Version.

I confirmed that restart-php-fpm-all is still functional by running it on mw1450.

Zabe suggests https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/811373 may be the problem, which seems likely.

I restarted the canaries with cumin. I concatenated /etc/dsh/group/{appserver,api_appserver}, removed comments, and ran

for h in $(<~/all_appservers); do 
    echo -n "$h "
    curl -sx $h:80 -H'X-Forwarded-Proto: https' 'http://en.wikipedia.org/w/api.php?action=query&format=json&meta=siteinfo' | jq .query.general.generator
done

This confirmed that all servers are now running wmf.21.

From logstash, looks like last canary restart was on the 20th.

rMSCAb616a7ffd1b7: Clean up php fpm restart touches canary restarts directly. Maybe that combined with the later rMSCA78cc0f8553dd: _restart_php: Exclude empty host lists causes unexpected behavior, since canary restarts were still happening after the first patch was deployed - assuming scap versions correspond to scap deploys.

I think arguably this should block backport windows as well as train deploys. At a minimum, deployers should exercise greater caution. I'll send a note to deployers for the next couple of windows until we sort out a fix. (I'm guessing a fix is relatively straightforward.)

Is this related to the issues we were seeing in backport windows where sometimes the patches would not take effect on some servers? (most recently that I know of on July 20, see #wikimedia-operations logs starting from 20:11:45).

Change 817200 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/tools/scap@master] Restart the canaries and testservers as well

https://gerrit.wikimedia.org/r/817200

I think I have a patch that should be fixing the regression at https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/817200

Change 817206 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] scap: restart php on 'scap pull'

https://gerrit.wikimedia.org/r/817206

Change 817206 merged by Giuseppe Lavagetto:

[operations/puppet@production] scap: restart php on 'scap pull'

https://gerrit.wikimedia.org/r/817206

Mentioned in SAL (#wikimedia-operations) [2022-07-26T09:31:33Z] <_joe_> running puppet on the mw-canary hosts T313770

Change 817212 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/tools/scap@master] Do not set the PHP variable, now unused

https://gerrit.wikimedia.org/r/817212

Change 817212 merged by jenkins-bot:

[mediawiki/tools/scap@master] Stop using utils.sudo_check_call in php restarts

https://gerrit.wikimedia.org/r/817212

Mentioned in SAL (#wikimedia-operations) [2022-07-26T12:02:27Z] <oblivian@deploy1002> Synchronized README: testing fix for php restarts T313770 (duration: 03m 15s)

Mentioned in SAL (#wikimedia-operations) [2022-07-26T12:32:21Z] <jnuche@deploy1002> Synchronized README: Verifying fix for T313770 (duration: 03m 14s)

Change 817200 abandoned by Giuseppe Lavagetto:

[mediawiki/tools/scap@master] Restart the canaries and testservers as well

Reason:

not needed.

https://gerrit.wikimedia.org/r/817200