This is resolved, thank you!

I think the worst of this trend has been reversed by the revert of setting cutoff days to 1: https://grafana.wikimedia.org/goto/rwrkdsWDg?orgId=1

An added benefit to this work would be defining xhgui as an actual service - currently both Arc Lamp and xhgui do not migrate with the services switchover, and we have an explicit ask from the data persistence team to accommodate this across SRE.

In T410296#11389201, @Dbrant wrote:

^ after deploying the above patch, the 504 errors have all but disappeared. Not sure if this has any bearing on the unavailable replicas issues, but it's something!

Seems like these cases should be changed to query the page-analytics service directly on https://page-analytics.discovery.wmnet:30443/metrics/pageviews/[...]

Just as a datapoint - I roll-restarted mobileapps and it had an immediate impact on wikifeeds: https://grafana.wikimedia.org/goto/lmB4-hmvg?orgId=1

In T410296#11380452, @ssastry wrote:

This panel shows an increase in traffic to the wikifeeds_featured endpoint as well as wikifeeds_onthisday endpoints. And this panel shows an increase in upstream request timeout for those two endpoints.

So, my first guess is that it looks like there has been an increase in traffic two of these endpoints which may have existing inefficiencies which might need investigation and fixing vs something new happening to the services itself.

Wikifeeds logs quite heavily in general, but it's hard to determine signal. Looks like there has been a solid increase in internal 504s, but there isn't really any further context the error messages.

In T410198#11378605, @daniel wrote:

In T410198#11378496, @hnowlan wrote:

Could you supply some of these IP addresses for investigation? My gut feeling is that these are going to be health checks. Is this the api-gateway or the rest-gateway?

I haven't checked the API gateway. Here are the top IPs for the REST gateway (naive sort to find dupes, not by rec/sec):

10.192.12.30
10.192.14.10
10.192.29.14
10.192.36.6
10.192.4.20
10.192.40.11
10.192.41.6
10.192.5.30
10.192.8.26
172.16.1.41
172.16.19.172
172.16.4.75

From 172.16.19.172 alone we see 40 req/sec. That seems a lot for health checks...

Also I'd assume that most health checks shouldn't go through the gateway, right?

Could you supply some of these IP addresses for investigation? My gut feeling is that these are going to be health checks. Is this the api-gateway or the rest-gateway?

This issue was happening as a result of the migration of the action API to a common gateway within WMF infrastructure (work ticket: T408223, higher level reasoning/tracking: T406607). We're currently undergoing a slow rollout of wikis by group with the exception of enwiki, which means that all wikis are currently behind the gateway, along with 10% of requests for enwiki. The gateway by default itself imposes a timeout of 15 seconds, which was causing the issue seen here. We've since raised the timeout and the queries in this ticket are now succeeding. Apologies for the disruption.

In T408223#11377871, @dcausse wrote:

We received notifications from users that the search API which is configured to allow 50s timeouts to support costly search requests is now failing at 15s with an upstream request timeout (T410007). The user reported that the behavior started to change around nov 11th which is apparently when we started to roll out this new route on group2 wikis. I'm not 100% sure that this change is the cause of this new behavior but IIUC on all wikis except enwiki we now route api.php requests to the rest-gateway. If I'm not mistaken the rest-gateway has a default timeout of 15s which might explain this new behavior? Are there ways to vary this timeout based on the target action API?

I think we can close this ticket as we won't need a public cloud account for this work, can you confirm @tappof?

Invited! please let us know if you have any issues. Once you've created your account, please make sure you can log into https://app.oncall-optimizer.com/ and sync your calendar. Thanks!

Hi Arian, thanks for the ticket - could you let us know what username you would like for your account? Usually we'd go with something akin to abozorg-wmde

Merged! I see deployment in the user groups for itamar now.

In T408920#11341332, @Dzahn wrote:

@hnowlan Just checking, it should be wmde not wmf in this case.

Thanks for the clarification.

Hi Virginie, your account appears to already be a member of analytics-privatedata-users which should grant you Superset access. This access was added in T407605.

L3 signed, NDA applies. Key verified OOB.

Key verified out of band.

Blocked on approval from @mark.

Awaiting out of band verification of SSH key on Slack. Tagging @thcipriani as approver for deployment group.

Migration complete. Impact on rest-gateway was minimal, no scaling up required.

We're rolling out 10% of enwiki at the moment, and we will leave things there until next week.

We're now using mw-api-ext as appropriate for rest.php and mw-api-related APIs.

We've implemented rest-gateway-ro in multi-dc.lua, traffic flows moving as expected.

I think everything here has been taken care of.

In T405368#11211211, @aaron wrote:

I'm also seeing extra headers for all the transform POST endpoints that give HTML, e.g.:

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/html/Aleph from anon
REASON: mwapi vs gateway header differences
!+ x-frame-options: SAMEORIGIN
!+ x-xss-protection: 1; mode=block
~- server: mw-api-ext.codfw.main-6647cc6f69-59k4s
~+ server: mw-api-int.codfw.main-7c66456f9f-kg5cm
!+ referrer-policy: origin-when-cross-origin
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
!+ access-control-allow-methods: GET,HEAD
~- date: Wed, 24 Sep 2025 17:19:14 GMT
~+ date: Wed, 24 Sep 2025 17:19:13 GMT
!+ access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
!+ access-control-expose-headers: etag

The 50% change has been merged and is rolling out over the next 30 minutes or so. Please be aware of cache when testing as cached responses might limit the distribution of requests

We have a change ready for this that can be pushed at any point. Unfortunately at present the only easy way to identify between requests to the gateway and requests that don't hit it is the presence of the content-length or content-security-policy headers. The via header is stripped at the edge.

test2wiki's rest.php is now routed via the rest-gateway. This can be seen in the Via header supplied by the gateway

Reopening, moving to o11y backlog.

Resolving this issue for now, in order to track work elsewhere.

Removing the observability tag here as I don't think there's anything for us to do on this task - please re-add us if needs be.

Some notable jumps that line up exactly with these increases are a significant jump in mw-web 200s:

In theory this should have become critical at 1 week remaining - is the critical alert defined properly?

These images are now rendering correctly - hard to pinpoint why as this issue is quite old.

Since adding the resource changes in T392348, it looks like the 7000px version of this image now renders correctly.

Image is now rendering, most likely fixed by T381594.

Looks like the affected thumbs are working now - a big thanks to @AntiCompositeNumber for the fix.