Page MenuHomePhabricator
Create Task
Maniphest T346800

startupregistrystats-testwiki periodic job fails
Closed, ResolvedPublic
Actions

Description

This job started failing at 04:10 UTC on Sep 19th

cgoubert@mwmaint1002:~$ rgrep mediawiki_job_startupregistrystats /var/log/syslog | grep Failed
Sep 19 04:10:30 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Failed with result 'exit-code'.

Log:

P52535 (An Untitled Masterwork)
1Sep 19 16:10:00 mwmaint1002 systemd[1]: Starting MediaWiki periodic job startupregistrystats-testwiki...
2Sep 19 16:10:00 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Checking 1018 modules...
3Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: [a897b30bcf761f7331a22637] [no req] TypeError: Return value of BlameStartupRegistry::getInternalStartupJs() must be of the type string, array returned
4Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Backtrace:
5Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: from /srv/mediawiki/php-1.41.0-wmf.27/extensions/WikimediaMaintenance/blameStartupRegistry.php(321)
6Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #0 /srv/mediawiki/php-1.41.0-wmf.27/extensions/WikimediaMaintenance/blameStartupRegistry.php(201): BlameStartupRegistry->getInternalStartupJs(MediaWiki\ResourceLoader\ResourceLoader, MediaWiki\ResourceLoader\Context)
7Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #1 /srv/mediawiki/php-1.41.0-wmf.27/maintenance/includes/MaintenanceRunner.php(703): BlameStartupRegistry->execute()
8Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #2 /srv/mediawiki/php-1.41.0-wmf.27/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
9Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #3 /srv/mediawiki/multiversion/MWScript.php(159): require_once(string)
10Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: #4 {main}
11Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: Traceback (most recent call last):
12Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: File "/usr/local/bin/mw-cli-wrapper", line 57, in <module>
13Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: subprocess.run(cmd, check=True, shell=True)
14Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: File "/usr/lib/python3.7/subprocess.py", line 487, in run
15Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: output=stdout, stderr=stderr)
16Sep 19 16:10:28 mwmaint1002 mediawiki_job_startupregistrystats-testwiki[32079]: subprocess.CalledProcessError: Command '/usr/local/bin/mwscript extensions/WikimediaMaintenance/blameStartupRegistry.php --wiki testwiki --record-stats' returned non-zero exit status 255.
17Sep 19 16:10:28 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Main process exited, code=exited, status=1/FAILURE
18Sep 19 16:10:28 mwmaint1002 systemd[1]: mediawiki_job_startupregistrystats-testwiki.service: Failed with result 'exit-code'.
19Sep 19 16:10:28 mwmaint1002 systemd[1]: Failed to start MediaWiki periodic job startupregistrystats-testwiki.

Details

SubjectRepoBranchLines +/-
blameStartupRegistry: migrate off string-assumed Module::getScript()mediawiki/extensions/WikimediaMaintenancemaster+3 -1
Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"mediawiki/coremaster+19 -36
Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"mediawiki/corewmf/1.41.0-wmf.27+19 -36
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasebrennenT345888 1.41.0-wmf.27 deployment blockers
 ResolvedNoneT346800 startupregistrystats-testwiki periodic job fails

Event Timeline

Clement_Goubert created this task.Sep 19 2023, 4:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2023, 4:34 PM
Clement_Goubert added a parent task: T345888: 1.41.0-wmf.27 deployment blockers.Sep 19 2023, 4:34 PM
Ladsgroup added a subscriber: Krinkle.Sep 19 2023, 4:35 PM
Ladsgroup subscribed.Sep 19 2023, 4:38 PM
Urbanecm_WMF subscribed.Sep 19 2023, 4:39 PM
Urbanecm_WMF added a comment.Sep 19 2023, 4:55 PM

Timeline of the issue suggests it started because of wmf.27 deployment to testwiki. The changelog suggests https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 might be a possible cause of this issue, as blameStartupRegistry.php calls MediaWiki\ResourceLoader\StartUpModule::getScript() and this method's signature was changed in that patch, resulting in the error.

brennen triaged this task as Unbreak Now! priority.Sep 19 2023, 5:02 PM
brennen subscribed.

Marking UBN as train blocker.

Aklapper added projects: MediaWiki-ResourceLoader, MediaWiki-extensions-WikimediaMaintenance.Sep 19 2023, 5:04 PM
Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptSep 19 2023, 5:04 PM
brennen added a comment.Sep 19 2023, 6:12 PM

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Jdforrester-WMF subscribed.Sep 19 2023, 6:36 PM
In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

gerritbot added a comment.Sep 19 2023, 6:36 PM

Change 959008 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959008

gerritbot added a project: Patch-For-Review.Sep 19 2023, 6:36 PM

Change 959009 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@wmf/1.41.0-wmf.27] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959009

matmarex subscribed.Sep 19 2023, 6:46 PM
gerritbot added a comment.Sep 19 2023, 7:00 PM

Change 959008 merged by jenkins-bot:

[mediawiki/core@master] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959008

Urbanecm_WMF added a comment.Sep 19 2023, 7:04 PM
In T346800#9180310, @Jdforrester-WMF wrote:
In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

Pardon me if it is obvious for everyone, but I don't think there's any related script in puppet? I'm only aware of extensions/WikimediaMaintenance/blameStartupRegistry.php, I presume that's what you mean?

Jdforrester-WMF added a comment.Sep 19 2023, 7:06 PM
In T346800#9180453, @Urbanecm_WMF wrote:
In T346800#9180310, @Jdforrester-WMF wrote:
In T346800#9180165, @brennen wrote:

Is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/957830 safe to revert?

Yes for the train, though I think a proper fix to the puppet file would be better than a revert from main.

Pardon me if it is obvious for everyone, but I don't think there's any related script in puppet? I'm only aware of extensions/WikimediaMaintenance/blameStartupRegistry.php, I presume that's what you mean?

Yes, that, as called by https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/profile/manifests/mediawiki/maintenance/startupregistrystats.pp but you're right that the fix would be in the WikimediaMaintenance repo (and thus would work nicely with the train).

gerritbot added a comment.Sep 19 2023, 7:06 PM

Change 959009 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.27] Revert "ResourceLoader: Set 'virtualFilePath' for startup.js"

https://gerrit.wikimedia.org/r/959009

Stashbot added a comment.Sep 19 2023, 7:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:07:19Z] <brennen@deploy2002> Started scap: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]]

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2023, 7:10 PM
Stashbot added a comment.Sep 19 2023, 7:29 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:29:40Z] <brennen@deploy2002> jforrester and brennen: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Stashbot added a comment.Sep 19 2023, 7:48 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T19:48:06Z] <brennen@deploy2002> Finished scap: Backport for [[gerrit:959009|Revert "ResourceLoader: Set 'virtualFilePath' for startup.js" (T346800)]] (duration: 40m 46s)

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.27; 2023-09-19).Sep 19 2023, 8:01 PM
matmarex closed this task as Resolved.Sep 19 2023, 9:00 PM

I suppose this is fixed. We can handle re-applying the change in another task (or just in Gerrit).

gerritbot added a comment.Sep 20 2023, 3:35 AM

Change 959070 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaMaintenance@master] blameStartupRegistry: migrate off string-assumed Module::getScript()

https://gerrit.wikimedia.org/r/959070

gerritbot added a project: Patch-For-Review.Sep 20 2023, 3:35 AM
gerritbot added a comment.Sep 20 2023, 2:05 PM

Change 959070 merged by jenkins-bot:

[mediawiki/extensions/WikimediaMaintenance@master] blameStartupRegistry: migrate off string-assumed Module::getScript()

https://gerrit.wikimedia.org/r/959070

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2023, 2:10 PM
ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.28; 2023-09-26); removed MW-1.41-notes (1.41.0-wmf.27; 2023-09-19).Sep 20 2023, 3:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL