Reassigning this to @Blake for safe keeping while a decision is made on how to track the remaining routing work on Q2 - e.g., whether to repurpose this task for a follow-on hypothesis vs. something else.

Transferring this over to @Blake, who will be picking up the Pretrain routing effort (thank you, Blake!).

Transferring this over to @Blake, who will be picking up the Pretrain routing effort (thank you, Blake!).

Following up, the REST Gateway page has been extended with a section on Upstream cluster selection for MediaWiki API requests, which explains the new functionality at a high level. That will need a small update when the feature is fully enabled (see T433752), but IMO is sufficient to explain the motivation, choice of strategy, and supported use case(s). I would not consider a general docs refresh as blocked on that.

I've finally had time to tweak (albeit minimally) https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Pretrain#Design_agreements in a way that aligns with the environment shape and routing design that are the focus of ST5.1.1 (and links out to the relevant implementation tasks). That could probably benefit for a more detailed take on why those particular choices were made (use well-established solutions for traffic routing, constrain special-casing to Pretrain itself as much as possible, avoid complexities inherent to using a non-production k8s cluster, etc.), but that may need to be added over time, and to some degree depends on target audience (i.e., some of these are in-the-weeds concerns of how of production environment works).

Revisiting this today, I noticed something odd in the alert history (4 week view). We can see a stretch of eqiad alerting through September 22nd, and then from there on, it's all intermittent alerting in codfw starting on the 26th. I then realized we never repooled codfw for etcd client reads following the power incident there (i.e., nearly all the traffic in codfw was etcd-mirror writes). codfw was repooled between 16:00 and 16:30 UTC today.

Transferring this back to @Clement_Goubert, who has kindly offered to shepherd the last change associated with T433752, at which point REST Gateway will fully support XWD routing.

https://gerrit.wikimedia.org/r/1346837 is now live in staging, and I've verified that:

• All configured XWD backends work as expected - i.e., successfully route to the appropriate upstream cluster.
• The rest-gateway test suite passes - i.e., is no more flaky than the "usual" number of off-by-ones in the rate-limiting specific tests (say, 1 in 50 test cases).

Unassigning, as I won't have time to work on this. I think this is still something worth doing, and am hopeful that the PoC linked in the task description can easily be revived.

Unassigning, as I won't have time to work on this. Adding @RLazarus for awareness, as something we could consider picking back up.

Unassigning, as I won't have time to work on this. This makes sense to do, but is clearly not a high priority for the moment.

Revisiting this task, I believe this is largely complete by way of the documentation updates in T367921#9914160 and subsequent updates to reflect the use of deb-ci package builds, which further simplify the procedure.

Just added https://wikitech.wikimedia.org/wiki/REST_Gateway#Upstream_cluster_selection_for_MediaWiki_API_requests, which explains the motivation behind this change and provides some basic intuition for how this works (with links to the relevant Envoy docs). We should update that as this work wraps up.

Alright, the PHP_ENGINE cookie support is now live, and I've tested it in staging using the same technique as in T433752#12287597. Once all rest-gateway routes that target mw-api-ext-ro have been moved over to the plugin (see https://gerrit.wikimedia.org/r/1346837 and https://gerrit.wikimedia.org/r/1346838), then enabling PHP_ENGINE diversion should be straightforward. I'll post a patch for that so y'all have it ready when the time comes.

I've deleted two more coredns pods in eqiad showing symptoms of healthcheck failures. @JMeybohm - In case it's useful, their logs are in /root/ on deploy1003. As in previous instances of this, they're reporting UDP read timeouts (i/o timeout) toward 10.3.0.1:53.

For confd specifically, this isn't really a question of etcd client behavior: the SRV record is resolved by confd itself during initialization - i.e., here in initConfig, called from main, with a simple net.LookupSRV. Even if confd did use the official (v2) etcd client's NewSRVDiscoverer (ref) for this purpose instead of the Go stdlib, it would still be a question of higher-level logic in confd detecting that the endpoints changed and swapping in a new client instance. In other words, it's more an application-level (confd) concern to solve, given that SRV -> endpoints resolution is external to the etcd client.

With https://gerrit.wikimedia.org/r/1335285 merged, envoy is now aware of the cluster specifier feature (along with prerequisites for X-Wikimedia-Debug routing), but it is not yet in use [0]. I'll follow up with @Clement_Goubert on the remaining patches in the series about how best to incrementally enable the new functionality across routes.

What I can now confirm is that the changeprop-reported (max) job run duration has changed from saturating at 5m to 7m. Which is to say (1) the envoy-side issue appears to be addressed, but (2) as you point out, there's still a (default) 7m timeout from the changeprop side.

@Urbanecm_WMF - The steam idle timeout change is live on jobrunners as of a bit before 17:20 UTC today. How frequently are reassignMenteesJobs produced? While I know I can reproduce the issue directly (T439577#12384791), it would be nice to confirm that things work end-to-end.

To recap, @Dzahn noticed that envoy had a 65s timeout on the RouteAction associated with the local listener. Indeed, in a scenario where the overall response hits that timeout, after already having sent headers (as here), we expect to see an abrupt connection termination.

Pulling a couple of data points together, this is reproducible internally when fetching directly from releases.discovery.wmnet (currently a CNAME to releases1003.eqiad.wmnet).

Thanks for all of your work here, @dancy! As expected, the test deployment of shellbox yesterday went smoothly.

Just realized I was still assigned to this. Un-assigning per T435426#12234869, since @jasmine_ has taken care of the puppet setup in https://gerrit.wikimedia.org/r/1329342.

Following up, it seems the stream idle timeout is the likely culprit - I was able to reproduce the issue simply using curl with a NullJob payload containing a usleep. Specifically:

• 300s is indeed the boundary between success vs. failure (504 served by envoy).
• Setting the RouteAction-level idle_timeout (mesh.idle_upstream_timeout) to match the overall request timeout resolves the issue.

Nice analysis, @Urbanecm_WMF.

@dancy - Thanks for performing this analysis, fixing the logstash query filter, and preparing the patch to update the thresholds.

Yes, exactly - once it's clear that the issue is resolution timeouts, the coredns_proxy_healthcheck_failures_total metric is what I look at first in order to identify a likely culprit.

Pods have been deleted. I'll keep an eye on fetch failures for a few minutes. Although there were 4 pods in total that had shown at least some symptoms of failures, the coredns-85b4f68d95-52t6h appears to have been the primary culprit (highest rate of failures, and an near-immediate reduction in fetch failures upon deletion).

Thanks for flagging, @MLechvien-WMF - No, going by the rate of upstream health check failures, this looks like the same mystery Core DNS issue as led to T422955: Detect elevated rates of EtcdConfig fetch failures and T422967: Investigate DNS query improvements in MediaWiki-on-k8s.

I believe there's nothing else tracked here, so I think we can resolve this. Thanks for the assistance with the investigation, all, and for the quick response to change endpoints, Sandra.

I've spent some time looking at this today, and I am ... stumped. @JMeybohm's observation in T438666 holds, where the instances of the alert correlate with a series of (slow) mutating requests. What's odd is that each one seems to correspond to a downward step in the latency SLI. As one might have guessed, these appear to be ipblock and (resulting) haproxy-provenance-map updates (using etcd-mirror logs on conf2005 as a history).

With https://gerrit.wikimedia.org/r/1331858 merged at the end of last week, the next step here is proceed with https://gerrit.wikimedia.org/r/1338000 and https://gerrit.wikimedia.org/r/1338030 to update all charts to the new module versions and then charlie out the changes. These were split simply due to the different sets of relevant modules.

Following up here: I'd propose we pick this up early in Q2, starting with the work in T435405: Investigate feasibility of adopting Zookeeper 3.8 on bookworm to upgrade to Zookeeper 3.8.

Great, thanks @JMeybohm. I'll be out through next week, but when I return, I'll start sketching out the upgrade procedure in a bit more detail.

If I'm understanding the very end of T437069#12290846 correctly, it sounds like the one potential workaround we have would be to add a @distro guard around the relevant line of the sources.list template to suppress bullseye-security, then rebuild the base image. That would of course come at the expense of some number of packages regressing, but is perhaps acceptable for this limited use case (i.e., as long as dependent images are never used in production).

Checking in, it looks like this is still broken in the same way as yesterday T437069#12297979 (InRelease Valid-Until has passed). It also doesn't look like a functioning alternative has been made available on archive.debian.org yet, so it's not as simple as switching the base image over to that and rebuilding.

BGP sessions look happy:

Thanks for the follow-up, @Mvolz. Yes, we can definitely help with making the private key available to your service via private puppet (details).

@Mvolz - Could you expand on the request here for ServiceOps? From a quick read of the task description, this sounds like a feature request for the respective applications (i.e., to compute and include the signature), rather than something dependent upon ServiceOps support. Thanks!

Thanks, folks. Per discussion in T421484#12278733, moving this to Scheduled.

My understanding per:

Using a similar strategy to yesterday (T433752#12282637), I was able to test both https://gerrit.wikimedia.org/r/1311964 and https://gerrit.wikimedia.org/r/1322140.

Huh ... Odd. Yeah, if I'm understanding these scap logs correctly, it looks like this query is producing this result, which clearly contains mw-web entries.

No issues encountered during a test deployment that exercised the new checks. Thanks for those!

Thanks for the detailed summary, @jasmine_!

I was able to do some basic testing in staging today after merging https://gerrit.wikimedia.org/r/1311963.

Works as expected:

@RKemper - Ah, that's great! Yes, if these workloads are able to access the mw-api-int-ro service directly, then that's vastly preferable. In which case, +1 to both points you raise as a path forward.