Page MenuHomePhabricator

MediaWiki periodic job cleanup-upload-stash failed
Closed, ResolvedPublicPRODUCTION ERROR

Description

Common information

  • alertname: MediaWikiCronJobFailed
  • label_cronjob: cleanup-upload-stash
  • label_team: mediawiki-file-management
  • prometheus: k8s
  • severity: task
  • site: codfw
  • source: prometheus
  • team: mediawiki-file-management

Firing alerts


  • dashboard: https://w.wiki/DocP
  • description: Use kube-env mw-cron codfw; kubectl get jobs -l team=mediawiki-file-management,cronjob=cleanup-upload-stash --field-selector status.successful=0 to see failures
  • runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
  • summary: MediaWiki periodic job cleanup-upload-stash failed
  • alertname: MediaWikiCronJobFailed
  • label_cronjob: cleanup-upload-stash
  • label_team: mediawiki-file-management
  • prometheus: k8s
  • severity: task
  • site: codfw
  • source: prometheus
  • team: mediawiki-file-management
  • Source

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptDec 11 2025, 1:53 AM
Restricted Application added subscribers: A_smart_kitten, Aklapper. · View Herald Transcript

Hello ServiceOps :) Please could an SRE grab the stack trace and error time for this when you get an opportunity?

Logs in logstash indicate a failure connecting to etcd.

kvwiki Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached in /srv/mediawiki/php-1.46.0-wmf.5/includes/Config/EtcdConfig.php on line 192

This stops execution of the loop. If that's not desirable, the configuration of the periodic job in modules/profile/manifests/mediawiki/maintenance/cleanup_upload_stash.pp could be changed to add foreachwiki_ignore_errors => true, that way the script would proceed with the next wiki and only error out if they're all unsuccessful.

FWIW that seems similar on its face to T346971: Uncaught ConfigException: Failed to load configuration from etcd, but with a slightly different message. (Though, looking at the code, it seems like the warning in your comment may have been followed by a "Failed to load configuration from etcd" ConfigException?)

If that's not desirable, the configuration of the periodic job in modules/profile/manifests/mediawiki/maintenance/cleanup_upload_stash.pp could be changed to add foreachwiki_ignore_errors => true, that way the script would proceed with the next wiki and only error out if they're all unsuccessful.

I don't know if I have the authority to make that decision; but I would personally suggest that we could probably keep things as-is for now, given that (AFAIK) this is the first time this cron-job has failed for this reason. Disclaimer that I do not know what the criticality of this particular job is, though.

With regards to this specific task -- it looks like the job may have been re-ran by now, so feel free to close this out if there's nothing left to do on your side (I don't know if this failed job will have to be manually deleted or not) :)

Clement_Goubert claimed this task.

Yup, that's the same error. It's usually transient, otherwise we have bigger problems.

Nothing more to do, the job has run successfully for 5 days straight now. Failed job object deleted, resolving task.