MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failed
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	phaultfinder
	Apr 3 2026, 12:49 AM

Project Tags

Referenced Files

	F75166324: image.png
	Apr 6 2026, 4:46 PM

Subscribers

Description

Common information

dashboard: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:refreshlinks-delete-from-nonexistent-s3)))))&_g=(time:(from:now-24h,to:now))
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-page-derived-data,cronjob=refreshlinks-delete-from-nonexistent-s3 --field-selector status.successful=0; kubectl logs -l cronjob=refreshlinks-delete-from-nonexistent-s3 --timestamps=true --prefix=true -c mediawiki-main-app to see failures and logs
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failed

alertname: MediaWikiCronJobFailed
label_cronjob: refreshlinks-delete-from-nonexistent-s3
label_team: mediawiki-page-derived-data
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-page-derived-data

Firing alerts

dashboard: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:refreshlinks-delete-from-nonexistent-s3)))))&_g=(time:(from:now-24h,to:now))
description: Use kube-env mw-cron eqiad; kubectl get jobs -l team=mediawiki-page-derived-data,cronjob=refreshlinks-delete-from-nonexistent-s3 --field-selector status.successful=0; kubectl logs -l cronjob=refreshlinks-delete-from-nonexistent-s3 --timestamps=true --prefix=true -c mediawiki-main-app to see failures and logs
runbook: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting
summary: MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failed
alertname: MediaWikiCronJobFailed
label_cronjob: refreshlinks-delete-from-nonexistent-s3
label_team: mediawiki-page-derived-data
prometheus: k8s
severity: task
site: eqiad
source: prometheus
team: mediawiki-page-derived-data
Source

Related Objects

Mentioned In: T422410: MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failed
Mentioned Here: T422455: Massive increase in "EtcdConfig failed to fetch data: Timeout was reached" warnings and errors since March 17th
T346971: Uncaught ConfigException: Failed to load configuration from etcd
T422308: MediaWiki periodic job update-flaggedrev-stats failed

Event Timeline

phaultfinder created this task.Apr 3 2026, 12:49 AM

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptApr 3 2026, 12:49 AM

Restricted Application added subscribers: A_smart_kitten, Aklapper. · View Herald Transcript

Seems like a temporary network error

Time from that Logstash screenshot (FWICS): 2026-04-03 00:43:10

attempting to copy down some of the errors in that screenshot (for future searchability)

ocwikibooks Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached [truncated in screenshot] /srv/mediawiki/php-1.46.0-wmf.22/includes/Config/EtcdConfig.php:207

ocwikibooks Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached in /srv/mediawiki/php-1.46.0-wmf.22/includes/Config/EtcdConfig.php on line 180

So yeah, seems like T346971: Uncaught ConfigException: Failed to load configuration from etcd, similarly to T422308#11790652.
Seems to be happening a relatively noticeable amount for mw-cron jobs recently, maybe: https://phabricator.wikimedia.org/maniphest/query/JPdsxxgWXcKR/#R

A_smart_kitten mentioned this in T422410: MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failed.Apr 6 2026, 5:59 PM

@A_smart_kitten Thank you!

I closed this task since it is probably not an issue specifically with the cronjobs, see T422455.

Ah yeah (and no problem with closing this task!) - thanks for linking T422455!

MediaWiki periodic job refreshlinks-delete-from-nonexistent-s3 failedClosed, ResolvedPublicPRODUCTION ERRORActions