startupregistrystats-testwiki periodic job fails
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Clement_Goubert
	Sep 19 2023, 4:34 PM

Description

This job started failing at 04:10 UTC on Sep 19th

cgoubert@mwmaint1002:~$ rgrep mediawiki_job_startupregistrystats /var/log/syslog | grep Failed
Sep 19 04:10:30 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Failed with result 'exit-code'.

Log:

P52535 (An Untitled Masterwork)

1	Sep 19 16:10:00 mwmaint1002 systemd[1]: Starting MediaWiki periodic job startupregistrystats-testwiki...
2	Sep 19 16:10:00 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Checking 1018 modules...
3	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: [a897b30bcf761f7331a22637] [no req] TypeError: Return value of BlameStartupRegistry::getInternalStartupJs() must be of the type string, array returned
4	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Backtrace:
5	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: from /srv/mediawiki/php-1.41.0-wmf.27/extensions/WikimediaMaintenance/blameStartupRegistry.php(321)
6	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #0 /srv/mediawiki/php-1.41.0-wmf.27/extensions/WikimediaMaintenance/blameStartupRegistry.php(201): BlameStartupRegistry->getInternalStartupJs(MediaWiki\ResourceLoader\ResourceLoader, MediaWiki\ResourceLoader\Context)
7	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #1 /srv/mediawiki/php-1.41.0-wmf.27/maintenance/includes/MaintenanceRunner.php(703): BlameStartupRegistry->execute()
8	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #2 /srv/mediawiki/php-1.41.0-wmf.27/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
9	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #3 /srv/mediawiki/multiversion/MWScript.php(159): require_once(string)
10	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #4 {main}
11	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Traceback (most recent call last):
12	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: File "/usr/local/bin/mw-cli-wrapper", line 57, in <module>
13	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: subprocess.run(cmd, check=True, shell=True)
14	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: File "/usr/lib/python3.7/subprocess.py", line 487, in run
15	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: output=stdout, stderr=stderr)
16	Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: subprocess.CalledProcessError: Command '/usr/local/bin/mwscript extensions/WikimediaMaintenance/blameStartupRegistry.php --wiki testwiki --record-stats' returned non-zero exit status 255.
17	Sep 19 16:10:28 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Main process exited, code=exited, status=1/FAILURE
18	Sep 19 16:10:28 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Failed with result 'exit-code'.
19	Sep 19 16:10:28 mwmaint1002 systemd[1]: Failed to start MediaWiki periodic job startupregistrystats-testwiki.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
blameStartupRegistry: migrate off string-assumed Module::getScript()	mediawiki/extensions/WikimediaMaintenance	master	+3 -1
Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"	mediawiki/core	master	+19 -36
Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"	mediawiki/core	wmf/1.41.0-wmf.27	+19 -36

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	brennen	T345888 1.41.0-wmf.27 deployment blockers
		Resolved		None	T346800 startupregistrystats-testwiki periodic job fails

Event Timeline

Clement_Goubert created this task.Sep 19 2023, 4:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2023, 4:34 PM

Clement_Goubert added a parent task: T345888: 1.41.0-wmf.27 deployment blockers.Sep 19 2023, 4:34 PM

Ladsgroup added a subscriber: Krinkle.Sep 19 2023, 4:35 PM

Ladsgroup subscribed.Sep 19 2023, 4:38 PM

Urbanecm_WMF subscribed.Sep 19 2023, 4:39 PM

Timeline of the issue suggests it started because of wmf.27 deployment to testwiki. The changelog suggests https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 might be a possible cause of this issue, as blameStartupRegistry.php calls MediaWiki\ResourceLoader\StartUpModule::getScript() and this method's signature was changed in that patch, resulting in the error.

Marking UBN as train blocker.

Aklapper added projects: MediaWiki-ResourceLoader, MediaWiki-extensions-WikimediaMaintenance.Sep 19 2023, 5:04 PM

Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptSep 19 2023, 5:04 PM

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

Change 959008 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959008

Change 959009 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@wmf/1.41.0-wmf.27] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959009

matmarex subscribed.Sep 19 2023, 6:46 PM

Change 959008 merged by jenkins-bot:

[mediawiki/core@master] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959008

In T346800#9180310, @Jdforrester-WMF wrote:

In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

Pardon me if it is obvious for everyone, but I don't think there's any related script in puppet? I'm only aware of extensions/WikimediaMaintenance/blameStartupRegistry.php, I presume that's what you mean?

In T346800#9180453, @Urbanecm_WMF wrote:

In T346800#9180310, @Jdforrester-WMF wrote:

In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

Pardon me if it is obvious for everyone, but I don't think there's any related script in puppet? I'm only aware of extensions/WikimediaMaintenance/blameStartupRegistry.php, I presume that's what you mean?

Yes, that, as called by https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/profile/manifests/mediawiki/maintenance/startupregistrystats.pp but you're right that the fix would be in the WikimediaMaintenance repo (and thus would work nicely with the train).

Change 959009 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.27] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959009

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:07:19Z] <brennen@deploy2002> Started scap: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]]

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2023, 7:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:29:40Z] <brennen@deploy2002> jforrester and brennen: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:48:06Z] <brennen@deploy2002> Finished scap: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]] (duration: 40m 46s)

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.27; 2023-09-19).Sep 19 2023, 8:01 PM

I suppose this is fixed. We can handle re-applying the change in another task (or just in Gerrit).

Change 959070 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaMaintenance@master] blameStartupRegistry: migrate off string-assumed Module::getScript()

https://gerrit.wikimedia.org/r/959070

gerritbot added a project: Patch-For-Review.Sep 20 2023, 3:35 AM

Change 959070 merged by jenkins-bot:

[mediawiki/extensions/WikimediaMaintenance@master] blameStartupRegistry: migrate off string-assumed Module::getScript()

https://gerrit.wikimedia.org/r/959070

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2023, 2:10 PM

ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.28; 2023-09-26); removed MW-1.41-notes (1.41.0-wmf.27; 2023-09-19).Sep 20 2023, 3:00 PM

DAlangi_WMF mentioned this in T410764: MediaWiki periodic job startupregistrystats-mediawikiwiki failed.Dec 3 2025, 6:31 PM