@CDanis for the time being I made some really minor changes (eg prevent choosing both DCs, removed auto refresh), until this is picked up properly

thank you @MatthewVernon ! I am closing this in favour or T414967, as I have noticed more timeouts.

In T412947#11528726, @kostajh wrote:

Thanks for the notes and ideas, @jijiki. We'll have another pass at the implementation to address the concerns you've raised. For sense of urgency, is sometime this quarter OK, or does it need to be dealt with more urgently? (cc @OKryva-WMF @Rsilvola)

In T414085#11524639, @Joe wrote:

I like the idea of having a full-stack view of traffic problems, especially on the media-serving side of things, which I'd focus on more at first. I would actually create two separate dashboards for text and media-serving.

@daniel is there something actionable from our end?

In T410038#11515987, @MLechvien-WMF wrote:

@jijiki I'd recommend reframing this task to a problem statement, maybe with title 'high volume of unactionable alerts for oncall'?

And in the description or as comments, adding more color to size the problem: anecdotal examples from recent incidents, what do we see as the most 'unactionable' (wrong threshold, confusing name/description in the alert, missing runbook, irrelevant alerts).

For example, today there was a page for Mathoid, with two alert, one for the blackbox[[ https://grafana.wikimedia.org/goto/P-h7Dq4Dg?orgId=1 | test ]] and another for OpenAPI/Swagger endpoints being unhealthy , and then hen we got the ATS alert on top of that, since the real cause was a scraper. Additionally, honestly, I wouldn't remember off the top of my head that its URL path contains media/math/, which would have made pattern matching so much easier and faster.

In T412947#11480759, @kostajh wrote:

Since 2 Dec 2025, I have observed an odd increase in get misses from mc1047 and mc1042 have a 50+% rise in get misses, compared to the rest of the cluster

The automatic failover mode kicked in on December 3 (logs), so that doesn't explain the December 2 event.

By and large, identifying which memcached key is receiving the most misses is not straightforward, unless the application explicitly counts those misses. Moreover, when a cache miss occurs, the conversation with memcached will be:

get WANCache:dawiktionary:revision-slots::6272:16905|#|v
END

whereas a hit will appear as:

get WANCache:cawiki:page-content-model:123456|#|v
VALUE WANCache:cawiki:page-content-model:123456|#|v 4 71
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
END

To identify the key misses, I captured a tcpdump which I then processed with tshark to isolate the misses.

I am unsure if this is something worth pursuing, so for the time being I am not adding an team tags etc.

I have updated the task description, to reflect how we could potentially move this forward, based on discussions during the WE5+WE6 offsite

Bluntly closing, api_appservers do not exist anymore.

Hello! Since 1212204 was backported, it has been producing thousands of error messages https://logstash.wikimedia.org/goto/103ab3d23a65b901740f79fc62e71e9a