MediaWiki periodic job cleanup-upload-stash failed
Open, LowPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	phaultfinder
	Wed, Jun 3, 2:08 AM

Description

Common information

dashboard: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:cleanup-upload-stash)))))&_g=(time:(from:now-24h,to:now))
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-file-management,cronjob=cleanup-upload-stash --field-selector status.successful=0; kubectl logs -l cronjob=cleanup-upload-stash --timestamps=true --prefix=true -c mediawiki-main-app to see failures and logs
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job cleanup-upload-stash failed

alertname: MediaWikiCronJobFailed
label_cronjob: cleanup-upload-stash
label_team: mediawiki-file-management
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-file-management

Firing alerts

dashboard: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:cleanup-upload-stash)))))&_g=(time:(from:now-24h,to:now))
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-file-management,cronjob=cleanup-upload-stash --field-selector status.successful=0; kubectl logs -l cronjob=cleanup-upload-stash --timestamps=true --prefix=true -c mediawiki-main-app to see failures and logs
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job cleanup-upload-stash failed
alertname: MediaWikiCronJobFailed
label_cronjob: cleanup-upload-stash
label_team: mediawiki-file-management
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-file-management
Source

Event Timeline

phaultfinder created this task.Wed, Jun 3, 2:08 AM

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptWed, Jun 3, 2:08 AM

Restricted Application added subscribers: A_smart_kitten, Aklapper. · View Herald Transcript

Maintenance_bot added a project: Commons.Wed, Jun 3, 2:30 AM

Runs daily, error is:

[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268126612Z lawikibooks Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl e
rror: 28) Timeout was reached in /srv/mediawiki/php-1.47.0-wmf.4/includes/Config/EtcdConfig.php:207    
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268140914Z lawikibooks Stack trace:
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268146629Z lawikibooks #0 /srv/mediawiki/php-1.47.0-wmf.4/includes/Config/EtcdConfig.php(130): MediaWiki\Config\EtcdConfig->lo
ad()                                                                                                   
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268151075Z lawikibooks #1 /srv/mediawiki/wmf-config/CommonSettings.php(199): MediaWiki\Config\EtcdConfig->getModifiedIndex()
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268170041Z lawikibooks #2 /srv/mediawiki/php-1.47.0-wmf.4/LocalSettings.php(4): require('/srv/mediawiki/...')
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268200656Z lawikibooks #3 /srv/mediawiki/php-1.47.0-wmf.4/includes/Setup.php(213): require_once('/srv/mediawiki/...')
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268247197Z lawikibooks #4 /srv/mediawiki/php-1.47.0-wmf.4/maintenance/run.php(51): require_once('/srv/mediawiki/...')
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268303029Z lawikibooks #5 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268314670Z lawikibooks #6 {main}
[pod/cleanup-upload-stash-29674140-kl8md/mediawiki-main-app] 2026-06-03T02:01:54.268324025Z lawikibooks   thrown in /srv/mediawiki/php-1.47.0-wmf.4/includes/Config/EtcdConfig.php on line 207

If the next run is clean we can close.

MediaWiki periodic job cleanup-upload-stash failedOpen, LowPublicPRODUCTION ERRORActions

Description

Common information

Firing alerts

Event Timeline

MediaWiki periodic job cleanup-upload-stash failed
Open, LowPublicPRODUCTION ERROR
Actions