This is due to a hardware issue with one of the hosts involved in the replication chain to the wiki replicas: T394624: db1155 HW memory errors

normal expectations (web replicas: < 5 min; analytics: < 1 h)

[citation needed]

The server has been fixed and it is now slowly catching up.
@Voyagerim I would like to understand where these expectations come from:

This far exceeds normal expectations (web replicas: < 5 min; analytics: < 1 h)

I was one of the people that set up the wiki replicas years ago and I am pretty sure we didn't commit to any of this, because it is pretty much impossible to guarantee those numbers, even further, there are schema changes that can take days to replicate. So it is important to understand where those expectations come from, because there is some misalignment somewhere and it should be addressed.

Thanks

@Marostegui , expectations regarding replication latency thresholds - namely that web replicas should maintain a latency of less than 5 minutes and analytic replicas less than 1 hour - are not formal service level agreements (SLAs), but rather empirical observations.

The MediaWiki load balancer is configured to stop sending read requests to a replica if its latency exceeds 5 seconds. The maxlag parameter, introduced in MediaWiki 1.10, allows clients (e.g. bots) to check the replication latency before executing write requests. If the delay exceeds the specified threshold (typically 5 seconds), the server returns an error, prompting the client to repeat the request later.

Thanks for the feedback, I think the problem is solved.

In T394859#10845998, @Voyagerim wrote:

@Marostegui , expectations regarding replication latency thresholds - namely that web replicas should maintain a latency of less than 5 minutes and analytic replicas less than 1 hour - are not formal service level agreements (SLAs), but rather empirical observations.

I think it is important to differentiate between expectations and observations. especially with a service that is not considered mission critical. We are happy that they work well most of the time though, and we hope to keep that level of service, but again, those cannot be called expectations.

The MediaWiki load balancer is configured to stop sending read requests to a replica if its latency exceeds 5 seconds. The maxlag parameter, introduced in MediaWiki 1.10, allows clients (e.g. bots) to check the replication latency before executing write requests. If the delay exceeds the specified threshold (typically 5 seconds), the server returns an error, prompting the client to repeat the request later.

These hosts are not under MW load balancer, in fact, they do not have any load balancer in front of them (they use a DNS system to map IPs and sections but that is about it). These hosts aren't part of the normal production replicas, they live hanging behind a production replica, a host that filters the data (which we call sanitarium, which is in fact, the one that had issues).