Page MenuHomePhabricator
Create Task
Maniphest T432110

EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached
Open, In Progress, MediumPublic
Actions

Description

Summary

The same as T429156: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached but has appeared again for MediaModeration scanning jobs. Seen the failure cause a exception and then alert 4 times in around 3 days

Example stacktrace

madwiki Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 180
madwiki Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 180
madwiki Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php:207
madwiki Stack trace:
madwiki #0 /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php(130): MediaWiki\Config\EtcdConfig->load()
madwiki #1 /srv/mediawiki/wmf-config/CommonSettings.php(197): MediaWiki\Config\EtcdConfig->getModifiedIndex()
madwiki #2 /srv/mediawiki/php-1.47.0-wmf.10/LocalSettings.php(4): require('/srv/mediawiki/...')
madwiki #3 /srv/mediawiki/php-1.47.0-wmf.10/includes/Setup.php(212): require_once('/srv/mediawiki/...')
madwiki #4 /srv/mediawiki/php-1.47.0-wmf.10/maintenance/run.php(51): require_once('/srv/mediawiki/...')
madwiki #5 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
madwiki #6 {main}
madwiki   thrown in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 207

Acceptance criteria

  • This error is avoided in maintenance scripts

Related Objects

Event Timeline

Dreamy_Jazz created this task.Jul 14 2026, 12:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2026, 12:53 PM
Dreamy_Jazz added a comment.Jul 14 2026, 12:56 PM

It would also be nice if the alert for these failing jobs is fired once per job. For example, a failure on Sunday caused 4 alerts to be sent to our Slack channel until I was on my PC for Monday to check the failing job

Dreamy_Jazz added a comment.Jul 14 2026, 4:29 PM

And again for the purge temporary accounts job:

fiwikiquote Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 180
fiwikiquote Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 180
fiwikiquote Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php:207
fiwikiquote Stack trace:
fiwikiquote #0 /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php(130): MediaWiki\Config\EtcdConfig->load()
fiwikiquote #1 /srv/mediawiki/wmf-config/CommonSettings.php(197): MediaWiki\Config\EtcdConfig->getModifiedIndex()
fiwikiquote #2 /srv/mediawiki/php-1.47.0-wmf.10/LocalSettings.php(4): require('/srv/mediawiki/...')
fiwikiquote #3 /srv/mediawiki/php-1.47.0-wmf.10/includes/Setup.php(212): require_once('/srv/mediawiki/...')
fiwikiquote #4 /srv/mediawiki/php-1.47.0-wmf.10/maintenance/run.php(51): require_once('/srv/mediawiki/...')
fiwikiquote #5 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
fiwikiquote #6 {main}
fiwikiquote   thrown in /srv/mediawiki/php-1.47.0-wmf.10/includes/Config/EtcdConfig.php on line 207
Dreamy_Jazz added a comment.Jul 15 2026, 9:31 AM

And again for MediaModeration job:

simplewikiquote Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.11/includes/Config/EtcdConfig.php on line 180
simplewikiquote Warning: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.11/includes/Config/EtcdConfig.php on line 180
simplewikiquote Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.11/includes/Config/EtcdConfig.php:207
simplewikiquote Stack trace:
simplewikiquote #0 /srv/mediawiki/php-1.47.0-wmf.11/includes/Config/EtcdConfig.php(130): MediaWiki\Config\EtcdConfig->load()
simplewikiquote #1 /srv/mediawiki/wmf-config/CommonSettings.php(197): MediaWiki\Config\EtcdConfig->getModifiedIndex()
simplewikiquote #2 /srv/mediawiki/php-1.47.0-wmf.11/LocalSettings.php(4): require('/srv/mediawiki/...')
simplewikiquote #3 /srv/mediawiki/php-1.47.0-wmf.11/includes/Setup.php(212): require_once('/srv/mediawiki/...')
simplewikiquote #4 /srv/mediawiki/php-1.47.0-wmf.11/maintenance/run.php(52): require_once('/srv/mediawiki/...')
simplewikiquote #5 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
simplewikiquote #6 {main}
simplewikiquote   thrown in /srv/mediawiki/php-1.47.0-wmf.11/includes/Config/EtcdConfig.php on line 207
Dreamy_Jazz updated the task description. (Show Details)Jul 15 2026, 9:32 AM
Dreamy_Jazz added a project: SRE.Jul 15 2026, 10:00 AM
Clement_Goubert triaged this task as Medium priority.Jul 15 2026, 10:20 AM
Clement_Goubert edited projects, added: ServiceOps-Mediawiki, SRE Observability, MW-on-K8s; removed: SRE, Kubernetes.
Clement_Goubert moved this task from Inbox to Scheduled (this Q) on the ServiceOps board.
Clement_Goubert subscribed.
In T432110#12119181, @Dreamy_Jazz wrote:

It would also be nice if the alert for these failing jobs is fired once per job. For example, a failure on Sunday caused 4 alerts to be sent to our Slack channel until I was on my PC for Monday to check the failing job

We can set foreachwiki_ignore_errors => true so that an error on one wiki doesn't make the whole job fail. This will cause the job to not alert at all unless it *completely* fails to run iirc.

SRE Observability is there a way to avoid the alert repeating when sending to slack?

For the actual issue, this is T346971: Uncaught ConfigException: Failed to load configuration from etcd for which we haven't yet found a fix.

Dreamy_Jazz added a comment.Jul 15 2026, 10:50 AM
In T432110#12123223, @Clement_Goubert wrote:
In T432110#12119181, @Dreamy_Jazz wrote:

It would also be nice if the alert for these failing jobs is fired once per job. For example, a failure on Sunday caused 4 alerts to be sent to our Slack channel until I was on my PC for Monday to check the failing job

We can set foreachwiki_ignore_errors => true so that an error on one wiki doesn't make the whole job fail. This will cause the job to not alert at all unless it *completely* fails to run iirc.

There should be logging for these jobs in other ways for when the issue needs to be addressed, so that's probably fine to do

hnowlan added subscribers: andrea.denisse, hnowlan.Jul 15 2026, 2:35 PM

@andrea.denisse will have a look at whether we can group these alerts to reduce the volume.

hnowlan moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2026, 2:51 PM
Dreamy_Jazz added a comment.EditedTue, Jul 21, 10:27 AM

(Happened again three times this week so far, with 20 messages in Slack)

Dreamy_Jazz added a comment.Sat, Jul 25, 2:12 PM

(Again over the weekend with 7 alerts for the same etcd failure)

andrea.denisse changed the task status from Open to In Progress.Wed, Jul 29, 2:47 AM
andrea.denisse claimed this task.
andrea.denisse added a comment.Wed, Jul 29, 2:58 AM
In T432110#12156037, @Dreamy_Jazz wrote:

(Again over the weekend with 7 alerts for the same etcd failure)

Hi, what's the name of the channel?? I can't find a channel for MediaModeration.

andrea.denisse added a comment.Wed, Jul 29, 3:03 AM

I was unable to find the alert on the alerts repo, but it seems to come from here. includes/Config/EtcdConfig.php.

hnowlan removed andrea.denisse as the assignee of this task.Wed, Jul 29, 2:45 PM

These errors appear to be log lines that are making it to slack rather than alerts - I'm not sure if there's much that we can do to batch these if they are distinct failures for different instances. It would be helpful if we could see examples of the actual slack messages in questions as it's not clear how they're being created.

andrea.denisse unsubscribed.Wed, Aug 12, 7:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits