MediaWiki periodic job startupregistrystats-testwiki failed
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Nov 4 2025, 8:16 PM

Description

Common information

dashboard: https://w.wiki/DocP
description: Use kube-env mw-cron codfw; kubectl get jobs -l team=mediawiki-platform,cronjob=startupregistrystats-testwiki --field-selector status.successful=0 to see failures
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job startupregistrystats-testwiki failed

alertname: MediaWikiCronJobFailed
label_cronjob: startupregistrystats-testwiki
label_team: mediawiki-platform
prometheus: k8s
severity: task
site: codfw
source: prometheus
team: mediawiki-platform

Firing alerts

dashboard: https://w.wiki/DocP
description: Use kube-env mw-cron codfw; kubectl get jobs -l team=mediawiki-platform,cronjob=startupregistrystats-testwiki --field-selector status.successful=0 to see failures
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job startupregistrystats-testwiki failed
alertname: MediaWikiCronJobFailed
label_cronjob: startupregistrystats-testwiki
label_team: mediawiki-platform
prometheus: k8s
severity: task
site: codfw
source: prometheus
team: mediawiki-platform
Source

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	mediawiki: Update location of startupregistrystats script	operations/puppet	production	+3 -3

Customize query in gerrit

Related Objects

Mentioned In: T410764: MediaWiki periodic job startupregistrystats-mediawikiwiki failed
T394956: Restore support for gauge metrics from MediaWiki PHP (post-Prometheus migration)
T409642: mw-cron "startupregistrystats" failing
Mentioned Here: T394956: Restore support for gauge metrics from MediaWiki PHP (post-Prometheus migration)

Event Timeline

phaultfinder created this task.Nov 4 2025, 8:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2025, 8:16 PM

From the Logstash mw-cron dashboard, we can see the blameStartupRegistry.php maintenance script invocation has been failing since Tue Nov 4, as follows:

Script '/srv/mediawiki/php-1.46.0-wmf.1/extensions/WikimediaMaintenance/blameStartupRegistry.php' not found (tried path '/srv/mediawiki/php-1.46.0-wmf.1/extensions/WikimediaMaintenance/blameStartupRegistry.php' and class '/srv/mediawiki/php-1\46\0-wmf\1/extensions/WikimediaMaintenance/blameStartupRegistry\php').

Running it manually via mwscript on the deployment serve shows that it does return an error code:

[22:02 UTC] krinkle at deploy1003.eqiad.wmnet
$ mwscript extensions/WikimediaMaintenance/blameStartupRegistry.php testwiki
[22:02 UTC] krinkle at deploy1003.eqiad.wmnet
(exit=1) $ echo $?
1

This was cauased by these patches on Oct 29:

It went out on Tue 4 Nov with the train: https://sal.toolforge.org/production?p=0&q=%221.46.0-wmf.1%22&d=

Fixed on Sat 8 Nov by Taavi shortly after I reported this breakage on IRC (mediawiki-core): https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMaintenance/+/1203232

Other information

The data in Grafana showed no gap or alert either, due to an unresolved Graphite>Prometheus regression: T394956: Restore support for gauge metrics from MediaWiki PHP (post-Prometheus migration)

Krinkle merged a task: T409232: MediaWiki periodic job startupregistrystats-mediawikiwiki failed.Nov 9 2025, 10:42 PM

Change #1202872 had a related patch set uploaded (by Krinkle; author: Zabe):

[operations/puppet@production] mediawiki: Update location of startupregistrystats script

https://gerrit.wikimedia.org/r/1202872

Change #1203281 had a related patch set uploaded (by Krinkle; author: Majavah):

[mediawiki/extensions/WikimediaMaintenance@wmf/1.46.0-wmf.1] Fix symbolic links

https://gerrit.wikimedia.org/r/1203281

Krinkle mentioned this in T394956: Restore support for gauge metrics from MediaWiki PHP (post-Prometheus migration).Nov 9 2025, 10:44 PM

Zabe subscribed.Nov 9 2025, 10:57 PM

OWresch-WMF moved this task from Inbox, needs triage to In progress (DO NOT USE) on the MediaWiki-Platform-Team board.Nov 10 2025, 3:44 PM

Change #1202872 merged by Effie Mouzeli: