MediaWiki periodic job initsitestats failed
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	phaultfinder
	Sep 10 2025, 11:55 PM

Description

Common information

dashboard: https://w.wiki/DocP
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-special-pages,cronjob=initsitestats --field-selector status.successful=0 to see failures
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job initsitestats failed

alertname: MediaWikiCronJobFailed
label_cronjob: initsitestats
label_team: mediawiki-special-pages
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-special-pages

Firing alerts

dashboard: https://w.wiki/DocP
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-special-pages,cronjob=initsitestats --field-selector status.successful=0 to see failures
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job initsitestats failed
alertname: MediaWikiCronJobFailed
label_cronjob: initsitestats
label_team: mediawiki-special-pages
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-special-pages
Source

Related Objects

Mentioned In: T416576: Make mw-cron jobs alert thresholds easily configurable
T416440: MediaWiki periodic job update-special-pages-s2 failed
Mentioned Here: T404277: Switchover s8 master (db1209 -> db1193)
T404026: MediaWiki periodic job initsitestats failed

Event Timeline

phaultfinder created this task.Sep 10 2025, 11:55 PM

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptSep 10 2025, 11:55 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

A_smart_kitten subscribed.Sep 11 2025, 7:03 AM

Please could an SRE grab the stack trace for this when they have time? I'm assuming this is probably another transient failure similar to T404026, but imo we should check what caused this before closing this out.

Sure thing

wikidatawiki Updating site statistics...Wikimedia\Rdbms\DBReadOnlyError from line 693 of /srv/mediawiki/php-1.45.0-wmf.18/includes/libs/rdbms/database/Database.php: Database is read-only: Maintenance until 23:45 UTC - T404277

T404277: Switchover s8 master (db1209 -> db1193) related transient failure.

Thanks @Clement_Goubert! (by the way, if there's a more preferable way for me to request stack traces for this kind of thing than tagging ServiceOps new, please let me know :))

Resolving as a transient failure that will have since recovered.

Tagging ServiceOps new is fine, thank you :)

A_smart_kitten moved this task from To triage to Special:Statistics on the MediaWiki-Special-pages board.Sep 13 2025, 8:04 AM

A_smart_kitten mentioned this in T416440: MediaWiki periodic job update-special-pages-s2 failed.Feb 4 2026, 5:34 PM

Blake mentioned this in T416576: Make mw-cron jobs alert thresholds easily configurable.Feb 5 2026, 11:39 AM