Page MenuHomePhabricator
Create Task
Maniphest T416440

MediaWiki periodic job update-special-pages-s2 failed
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Common information

  • alertname: MediaWikiCronJobFailed
  • label_cronjob: update-special-pages-s2
  • label_team: mediawiki-special-pages
  • prometheus: k8s
  • severity: task
  • site: codfw
  • source: prometheus
  • team: mediawiki-special-pages

Firing alerts

Related Objects

Event Timeline

phaultfinder created this task.Wed, Feb 4, 6:22 AM
Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptWed, Feb 4, 6:22 AM
Restricted Application added subscribers: A_smart_kitten, Aklapper. · View Herald Transcript
A_smart_kitten added projects: serviceops, ServiceOps new.Wed, Feb 4, 9:53 AM

Hello serviceops! Please could an an SRE grab the stack trace & error time when you have some time spare?

Blake claimed this task.Wed, Feb 4, 5:19 PM
Blake moved this task from Inbox to In Progress on the ServiceOps new board.
Blake subscribed.

Hello! It looks like the MariaDB server was briefly read-only for some maintenance during the time when the cron executed:

2026-02-04T06:16:25.490906024Z nowiki Mostimages                     [QueryPage] Wikimedia\Rdbms\DBQueryError from line [0/0]
of /srv/mediawiki/php-1.46.0-wmf.13/includes/libs/Rdbms/Database/Database.php: Error 1290: The MariaDB server is running with
 the --read-only option so it cannot execute this statement

That work was taking place in T416300, and should now be complete - I'd expect this cron to succeed on next run.

A_smart_kitten added a comment.Wed, Feb 4, 5:34 PM

Thanks for fetching that! From my perspective this task can probably now be closed (although I don't know whether the failed job would need to be deleted first or not). I'll leave it up to y'all about whether or not you think it's worth rerunning this job :)

(For cross-referencing purposes: it seems like this might be a similar reason for the job failing as in T404280: MediaWiki periodic job initsitestats failed)

MLechvien-WMF subscribed.Thu, Feb 5, 8:34 AM

If this happens from time to time, would it worth referencing those example bugs (or the MariaDB switchover) in the runbook @Blake ?

Blake added a comment.Thu, Feb 5, 9:54 AM

@MLechvien-WMF If possible, I'd rather find a way to make the script resilient to the maintenance more generally. Ideally, I think we'd just schedule a re-run on failure, and only alert if the cron hasn't successfully run in a particular time window. I'll do some reading, and see what might make sense here. I'll change the docs if there's no better way to automate this away.

Blake mentioned this in T416576: Make mw-cron jobs alert thresholds easily configurable.Thu, Feb 5, 11:39 AM

I'd like to have a chat with the team about how we alert for mw-crons. I've created T416576 to track the follow-up here, but in the short term, the failed job has been deleted, and we expect it to succeed on next run, so closing this out.

Blake closed this task as Resolved.Thu, Feb 5, 11:41 AM
MLechvien-WMF mentioned this in T416970: MediaWiki periodic job update-special-pages-s7 failed.Tue, Feb 10, 10:55 AM
A_smart_kitten mentioned this in T417020: Proposal: mw-cron failure tasks that get automatically filed for unstewarded components should also tag the ServiceOps Phabricator project.Tue, Feb 10, 2:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits