Relevant logs:

[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335867350Z wikidatawiki Fatal error: Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached Resolving timed out after 2000 milliseconds in /srv/mediawiki/php-1.47.0-wmf.21/includes/Config/EtcdConfig.php:207
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335873509Z wikidatawiki Stack trace:
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335876737Z wikidatawiki #0 /srv/mediawiki/php-1.47.0-wmf.21/includes/Config/EtcdConfig.php(130): MediaWiki\Config\EtcdConfig->load()
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335889579Z wikidatawiki #1 /srv/mediawiki/wmf-config/CommonSettings.php(197): MediaWiki\Config\EtcdConfig->getModifiedIndex()
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335913023Z wikidatawiki #2 /srv/mediawiki/php-1.47.0-wmf.21/LocalSettings.php(4): require('/srv/mediawiki/...')
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335921616Z wikidatawiki #3 /srv/mediawiki/php-1.47.0-wmf.21/includes/Setup.php(212): require_once('/srv/mediawiki/...')
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335928475Z wikidatawiki #4 /srv/mediawiki/php-1.47.0-wmf.21/maintenance/run.php(52): require_once('/srv/mediawiki/...')
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335931360Z wikidatawiki #5 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335937586Z wikidatawiki #6 {main}
[pod/update-special-pages-s8-29838540-fcj4r/mediawiki-main-app] 2026-09-25T05:00:09.335942255Z wikidatawiki   thrown in /srv/mediawiki/php-1.47.0-wmf.21/includes/Config/EtcdConfig.php on line 207

Looks like we had trouble talking to etcd.

Thanks Blake. @JMeybohm @Scott_French flagging if anything relevant/informative for the recent Etcd investigations? If yes can you please make it a subtask with the investigation

Thanks for flagging, @MLechvien-WMF - No, going by the rate of upstream health check failures, this looks like the same mystery Core DNS issue as led to T422955: Detect elevated rates of EtcdConfig fetch failures and T422967: Investigate DNS query improvements in MediaWiki-on-k8s.

As usual, I'll delete the Core DNS pods reporting errors (once I emerge from my current meeting).

Mentioned in SAL (#wikimedia-operations) [2026-09-28T21:24:52Z] <swfrench-wmf> deleting suspect coredns pods showing upstream resolution health check failures in eqiad - T439201

Pods have been deleted. I'll keep an eye on fetch failures for a few minutes. Although there were 4 pods in total that had shown at least some symptoms of failures, the coredns-85b4f68d95-52t6h appears to have been the primary culprit (highest rate of failures, and an near-immediate reduction in fetch failures upon deletion).

Edit: Confirmed a few hours later that the rate of fetch failures has subsided considerably, with the exception of a brief spike in the 00:54 UTC (Tuesday) minute. For future reference, one can distinguish the "slow etcd" and "slow DNS" cases by looking at the curl error detail appended to the ConfigException message, as seen in T439201#12363709 - e.g., Resolving timed out after 2000 milliseconds. In the (typically non-fatal, thanks to the APCu cached config) PHP-FPM case, the same can be seen in the EtcdConfig failed to fetch data warning emitted to the PHP-FPM error log (example).

In T439201#12371879, @Scott_French wrote:

Pods have been deleted. I'll keep an eye on fetch failures for a few minutes. Although there were 4 pods in total that had shown at least some symptoms of failures, the coredns-85b4f68d95-52t6h appears to have been the primary culprit (highest rate of failures, and an near-immediate reduction in fetch failures upon deletion).

What where the symptoms you where considering? Especially the replica you mentioned did log an elevated amount of i/o timeouts starting somewhere around Sept 21st alongside an increase of coredns_proxy_healthcheck_failures_total. I would guess that's what you are referring to?

Yes, exactly - once it's clear that the issue is resolution timeouts, the coredns_proxy_healthcheck_failures_total metric is what I look at first in order to identify a likely culprit.

Please check if DNS pods need deleting again. The pod fixing made a significant improvement in failures for calling out to hCaptcha APIs (server-to-server going through urldownloaders) and HTTP requests made entirely within production (calling inference models). But since the 1st of October we have seen that improvement reverse back to where it was before and also at the same time seen more EtcdConfig failures for our periodic scripts:

It does indeed look like we have a notable outlier:

coredns_proxy_healthcheck_failures_total{instance="10.67.168.133:9153", job="k8s-pods", k8s_app="kube-dns", kubernetes_namespace="kube-system", kubernetes_pod_name="coredns-85b4f68d95-r6jdq", pod_template_hash="85b4f68d95", prometheus="k8s", proxy_name="forward", site="eqiad", to="10.3.0.1:53"}

I have deleted the misbehaving pod (though it looks like it might have recovered a couple hours ago?)

It got better but then got worse again:

Not sure if that's reflected in your graph

I've deleted two more coredns pods in eqiad showing symptoms of healthcheck failures. @JMeybohm - In case it's useful for later analysis, their logs are in /root/ on deploy1003. As in previous instances of this, they're reporting UDP read timeouts (i/o timeout) toward 10.3.0.1:53.

Thanks for the fixes

Unfortunately a few hours after the delete this was back to a higher rate for external HTTP requests (https://logstash.wikimedia.org/goto/af3ccb77a61821a4e34a98d45dd01aec):

For internal HTTP requests this is also seen (https://logstash.wikimedia.org/goto/bd66e7b2535f2d216414f1290868a66b):
{F106813635}

Deleted coredns-85b4f68d95-96gz4 that was showing symptoms

Looking at the new pod I created immediately fail a healthcheck, I decided to peek at p99 latency per pod:

Then I figured that maybe the etcd and coredns issues are the same, and maybe caused by saturation on 1Gbps NIC, and the host it's running on is 1G, and discarding quite a bit

There may be an argument for tainting the 10Gbps NIC hosts and scheduling coredns workloads there.