Page MenuHomePhabricator

Migrate scap's maintenance script invocations to PHP 8.1
Closed, ResolvedPublic

Description

During the course of a deployment, there are a number of maintenance scripts that scap may run, such as purgeMessageBlobStore.php and rebuildLocalisationCache.php.

There is also the scap mwscript command for running arbitrary maintenance scripts in the context of the MediaWiki staging directory (i.e., /srv/mediawiki-staging).

My understanding is that following recent work by @dduvall, all of this happens in a Docker container with a bind-mount for the staging directory, using the mediawiki_runtime_image which currently defaults to docker-registry.wikimedia.org/php7.4-fpm-multiversion-base.

Given where we are with the production migration of PHP 8.1 (T383845), it probably makes sense to switch this to the 8.1 equivalent.

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
ResolvedReedy
StalledNone
OpenNone
OpenNone
OpenNone
ResolvedReedy
ResolvedKrinkle
ResolvedKrinkle
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedLucas_Werkmeister_WMDE
ResolvedNone
ResolvedJdforrester-WMF
ResolvedDaimona
ResolvedJdforrester-WMF
DeclinedNone
ResolvedScott_French
ResolvedScott_French
ResolvedScott_French
Resolvedcscott
ResolvedScott_French
DuplicatePRODUCTION ERRORNone
ResolvedPRODUCTION ERRORMichael
ResolvedPRODUCTION ERRORMichael
ResolvedMichael
DuplicatePRODUCTION ERRORNone
ResolvedTgr
ResolvedNone
ResolvedDAlangi_WMF
ResolvedTgr
ResolvedDAlangi_WMF
ResolvedTgr
ResolvedTgr
ResolvedAtieno
OpenNone
Resolvedbrouberol
ResolvedScott_French
ResolvedScott_French
ResolvedScott_French
ResolvedScott_French
ResolvedScott_French
ResolvedScott_French
ResolvedKrinkle
ResolvedKrinkle
ResolvedScott_French
ResolvedKrinkle
ResolvedTgr
ResolvedScott_French
Resolvedjnuche
ResolvedJdforrester-WMF
ResolvedBUG REPORTbd808
ResolvedReedy
ResolvedReedy
Resolvedseanleong-WMDE
StalledNone
OpenNone
ResolvedLucas_Werkmeister_WMDE
ResolvedDaimona
ResolvedDaimona
ResolvedDaimona
OpenNone
ResolvedUmherirrender
OpenNone
ResolvedUmherirrender
ResolvedUmherirrender
Resolved mszabo
Resolvedtstarling
ResolvedUmherirrender
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedPhysikerwelt
ResolvedTgr
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedNone
ResolvedUmherirrender
ResolvedNone
ResolvedNone
ResolvedkarapayneWMDE
ResolvedAudreyPenven_WMDE
ResolvedAudreyPenven_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedUmherirrender
Resolvedthiemowmde
ResolvedLucas_Werkmeister_WMDE
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedUmherirrender
Resolved mszabo
ResolvedxSavitar
ResolvedUmherirrender
ResolvedUmherirrender
ResolvedUmherirrender
OpenNone
OpenNone
OpenNone
OpenDannyS712
ResolvedUmherirrender
Resolved larissagaulia
ResolvedUmherirrender
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedKrinkle
ResolvedScott_French

Event Timeline

@dduvall - When you get a chance, could you confirm that my understanding here is correct?

If so, then making the change is in and of itself fairly simple (i.e., just update the config, and then update the default to match).

The natural first step, though, would be to enumerate all the of relevant scripts and assess whether there are possible risks / surprises to consider moving to 8.1.

@dduvall - When you get a chance, could you confirm that my understanding here is correct?

Confirming on Dan's behalf.

If so, then making the change is in and of itself fairly simple (i.e., just update the config, and then update the default to match).

The natural first step, though, would be to enumerate all the of relevant scripts and assess whether there are possible risks / surprises to consider moving to 8.1.

Scap runs these scripts via mwscript (in a container):

eval.php
rebuildLocalisationCache.php
mergeMessageFileList.php
purgeMessageBlobStore.php

Thank you very much, @dancy. I'll take a closer look next week at the scripts you listed, how they're used, and what kinds of risks we might need to consider.

I reviewed the scripts today, and aside from rebuildLocalisationCache.php, these all appear to be fairly straightforward in their implementation and / or use case during deployments. Even in rebuildLocalisationCache.php, which touches a lot more core MediaWiki code than the others, my understanding is that it's touching code paths (e.g., in LocalisationCache) that we should be exercising in some form on 8.1 in production already.

One additional thought: I can do some basic smoke testing by switching mediawiki_runtime_image in train-dev, in an effort to shake out any "obvious" issues. I'll give that a try.

I was able to do some very testing in train-dev by switching mediawiki_runtime_image to php8.1-fpm-multiversion-base and deploying with scap (after clearing out the l10n cache files, ExtensionMessages-*.php files, etc.). No issues encountered upon doing so, and the resulting files have the same content as on 7.4. Of course, this is not entirely representative - e.g., only a single language (en) is used in train-dev - but I think bodes well for attempting to switch scap over.

Mentioned in SAL (#wikimedia-operations) [2025-04-08T17:14:24Z] <swfrench@deploy1003> Started scap sync-world: Pilot stop-before-sync scap run using PHP 8.1 container image for maintenance scripts - T390225

Mentioned in SAL (#wikimedia-operations) [2025-04-08T17:35:02Z] <swfrench@deploy1003> Started scap sync-world: Pilot scap run using PHP 8.1 container image for maintenance scripts - T390225

Mentioned in SAL (#wikimedia-operations) [2025-04-08T17:38:22Z] <swfrench@deploy1003> Finished scap sync-world: Pilot scap run using PHP 8.1 container image for maintenance scripts - T390225 (duration: 03m 19s)

I was able to use -Dmediawiki_runtime_image:docker-registry.wikimedia.org/php8.1-fpm-multiversion-base (thanks @dancy for reminding me this was possible) to switch to 8.1 for two pilot scap runs this morning. I did that in two steps:

  1. A first run with --stop-before-sync so that I could check for surprising content changes in the l10n cache files.
  2. A normal scap run that would get all the way through the deployment (i.e., all the way through purgeMessageBlobStore.php, which happens at the end).

Both of those worked without an issue - no errors / failures, no surprising l10n cache content changes.

Now, before merging my patch, I'd like to test one more thing: a full l10n cache rebuild, since the above only exercised the "no changes, so no rebuild" case in rebuildLocalisationCache.php.

@dancy has kindly pointed out that I can do this in train-dev by unsetting SCAP_MW_LANG while running the train (after clearing out the multi-version paths in /srv/mediawiki-staging).

As long as that works as expected, which I intend to test shortly, I have no concerns about merging my patch.

Alright, after a bit of head-scratching, I believe we're good to go with this.

Specifically, I initially did not realize that the cdb files embed timestamps for all of the dependency files used in their generation. Thus, naively clearing out /srv/mediawiki-staging/php-{version} between runs would result in cdb files with different digests.

If I specifically examine the JSON dumps thereof, I can easily see that's the only content difference. Similarly, if I delete only /srv/mediawiki-staging/php-{version}/cache/l10n between scap runs, rather than having to re-initialize php-{version} in its entirety, I get identical content.

Long story short, indeed it seems that 7.4 and 8.1 images produce the same content.

I'll merge my patch and wrap this up during tomorrow's UTC-late infra window.

Change #1134758 merged by Scott French:

[operations/puppet@production] scap: Use PHP 8.1 when executing maintenance scripts

https://gerrit.wikimedia.org/r/1134758

Mentioned in SAL (#wikimedia-operations) [2025-04-09T17:45:41Z] <swfrench@deploy1003> Started scap sync-world: Test stop-before-sync scap run after switching to PHP 8.1 container image for maintenance scripts - T390225

Mentioned in SAL (#wikimedia-operations) [2025-04-09T17:47:11Z] <swfrench@deploy1003> Started scap sync-world: Test scap run after switching to PHP 8.1 container image for maintenance scripts - T390225

Mentioned in SAL (#wikimedia-operations) [2025-04-09T17:50:04Z] <swfrench@deploy1003> Finished scap sync-world: Test scap run after switching to PHP 8.1 container image for maintenance scripts - T390225 (duration: 03m 10s)

Change #1135509 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] Revert "scap: Use PHP 8.1 when executing maintenance scripts"

https://gerrit.wikimedia.org/r/1135509

The mediawiki_runtime_image default change is now live with scap 4.153.0, so all that remains here is to clean up the scap.cfg override (https://gerrit.wikimedia.org/r/1135509).

Change #1135509 merged by Scott French:

[operations/puppet@production] Revert "scap: Use PHP 8.1 when executing maintenance scripts"

https://gerrit.wikimedia.org/r/1135509

With the override cleaned up, there should be nothing else to do here.