dashboard: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:update-special-pages-s7)))))&_g=(time:(from:now-24h,to:now))
description: Use kube-env mw-cron codfw; kubectl get jobs -l team=mediawiki-special-pages,cronjob=update-special-pages-s7 --field-selector status.successful=0; kubectl logs -l cronjob=update-special-pages-s7 --timestamps=true --prefix=true -c mediawiki-main-app to see failures and logs
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job update-special-pages-s7 failed
alertname: MediaWikiCronJobFailed
label_cronjob: update-special-pages-s7
label_team: mediawiki-special-pages
prometheus: k8s
severity: task
site: codfw
source: prometheus
team: mediawiki-special-pages
Source

Related Objects

Mentioned In: T417020: Proposal: mw-cron failure tasks that get automatically filed for unstewarded components should also tag the ServiceOps Phabricator project
Mentioned Here: T341555: Implement periodic maintenance scripts for mw-on-k8s
T405052: MediaWiki periodic job cleanup-upload-stash failed
T416576: Make mw-cron jobs alert thresholds easily configurable
T412799: Improve MediaWiki periodic job alerts
T416440: MediaWiki periodic job update-special-pages-s2 failed

Event Timeline

phaultfinder created this task.Feb 10 2026, 6:57 AM

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptFeb 10 2026, 6:57 AM

Restricted Application added subscribers: A_smart_kitten, Aklapper. · View Herald Transcript

Hello serviceops! Please could you get the stack trace and error time for this when an SRE has some time spare?

@Blake this seems similar to T416440 , could you PTAL?

Also I know that @Clement_Goubert did some work in T412799 (and parent task) to allow those alerts to be more self service, could it be used here?

I cleared the job out this morning, it was identical to the failure mode in T416440, which means (imo) that we should address it in the same way we address the category of mw-cron failures we examine in T416576.

From logstash:

@timestamp
	Feb 10, 2026 @ 06:51:37.026

fawiki Mostlinkedtemplates            [QueryPage] Wikimedia\Rdbms\DBQueryError from line 1226 of /srv/mediawiki/php-1.46.0-wmf.14/includes/libs/Rdbms/Database/Database.php: Error 1290: The MariaDB server is running with the --read-only option so it cannot execute this statement

I'm not sure what logstash ACLs are like. @A_smart_kitten, are you able to use the logstash links in these tickets, or are those locked down? Thanks!

In T416970#11600936, @Blake wrote:

@A_smart_kitten, are you able to use the logstash links in these tickets, or are those locked down?

No, I'm not personally able to access those links.

(In case you're interested -- for some context about why I do anything with these cron-job failure tickets myself...

In T405052#11206396, @A_smart_kitten wrote:

[...] I just try to do a bit of volunteer initial triaging of the cron-job failure tasks that get filed in unstewarded components...to try to prevent them from just sitting here forever, lol. [...]

See also T341555#10760093 & the following comments there for some additional context.)

Ah, okay, thanks very much - hopefully the work in T416576 will reduce the number of unactionable cron failures we receive alerts for.

For now, the failed job has been cleaned up, and there's nothing more to do here.

A_smart_kitten mentioned this in T417020: Proposal: mw-cron failure tasks that get automatically filed for unstewarded components should also tag the ServiceOps Phabricator project.Feb 10 2026, 2:55 PM

A_smart_kitten moved this task from To triage to Maintenance reports on the MediaWiki-Special-pages board.Apr 22 2026, 11:14 AM

MediaWiki periodic job update-special-pages-s7 failedClosed, ResolvedPublicPRODUCTION ERRORActions

Description

Common information

Firing alerts

Related Objects

Event Timeline

MediaWiki periodic job update-special-pages-s7 failed
Closed, ResolvedPublicPRODUCTION ERROR
Actions