Page MenuHomePhabricator

Beta cluster jobrunner opcache is not updated during automated deployments
Closed, ResolvedPublic

Description

Background context: When changing the language links for Main Page, we don't see any wikidata recentchange injection on the subscribed clients.

Looking in udp2log logs arriving in /srv/mw-log on mwlog.svc.deployment-prep.eqiad1.wikimedia.cloud, we can see that the change propagation jobs begin as expected.

Problem: A wikidata client job crashes due to what looks like an inconsistent mix of code versions, calling a 10-parameter constructor with 8 parameters. Both sides of this code were changed in fbec9bc3edd363d0abbd407dc7e9e2fe43055676 and the deployed code on the job runner matches the new version. For some reason, the job emits an error as if the calling side of the code is at an older revision:

/srv/mw-log/exception.log:2026-09-21 06:40:39.230302 [004ed7c5-8b07-445f-bf16-5de215b3a7ed] deployment-jobrunner06 dewiki 1.47.0-alpha exception ERROR: [004ed7c5-8b07-445f-bf16-5de215b3a7ed] /rpc/RunSingleJob.php   ArgumentCountError: Too few arguments to function Wikibase\Client\Changes\ChangeHandler::__construct(), 8 passed in /srv/mediawiki/php-master/extensions/Wikibase/client/WikibaseClient.ServiceWiring.php on line 191 and exactly 10 expected {"exception_url":"/rpc/RunSingleJob.php","reqId":"004ed7c5-8b07-445f-bf16-5de215b3a7ed","caught_by":"mwe_handler"}

File modification timestamp precedes the logline...

-rw-r--r-- 1 mwdeploy mwdeploy 44472 Sep 19 09:01 /srv/mediawiki/php-master/extensions/Wikibase/client/WikibaseClient.ServiceWiring.php

I'm able to consistently trigger the error, even after verifying the code at this path on the jobrunner is up to date.

Event Timeline

awight renamed this task from Beta cluster seems to not propagate wikidata changes to subscribing clients to Beta cluster jobrunner may be running stale code.Mon, Sep 21, 1:36 PM
awight updated the task description. (Show Details)

I found that I had to clear the opcache on the jobrunner again, after deploying a configuration change. This should be automated (ideally) or at least documented as a required step.

Clearing is done with:

php7adm /opcache-free
awight renamed this task from Beta cluster jobrunner may be running stale code to Beta cluster jobrunner opcache is not updated during automated deployments.Tue, Sep 22, 7:25 AM

Hi @JMonton-WMF, @awight, deployment-jobrunner06 is a new PHP 8.5 jobrunner slated to replace the old PHP 8.3 host deployment-jobrunner05. Please see T435393#12320355 for the background.

You are correct the new PHP 8.5 was not added to the jobrunner dsh group; this was an oversight on my part. Due to that, deployment-jobrunner06 was not an indirect member of mw_web_clusters. As a result, the restart script (php_fpm_restart_script; /usr/local/sbin/restart-php-fpm-all) was not invoked by Scap.

I have added the host to the dsh group. As expected, each Scap deploy will now invoke the restart-php-fpm-all:

Sep 22 20:00:11 deployment-jobrunner06 sudo[3383522]: mwdeploy : PWD=/var/lib/mwdeploy ; USER=root ; COMMAND=/usr/local/sbin/restart-php-fpm-all

Marking this as resolved!

Southparkfan claimed this task.

Mentioned in SAL (#wikimedia-cloud) [2026-09-22T20:08:55Z] <Southparkfan> add deployment-jobrunner06 to 'jobrunner' dsh group - T438656