Page MenuHomePhabricator

Wiki replicas replication lag for the s4 slice (commonswiki) is over 80 hours
Closed, DuplicatePublic

Description

The replication lag at https://tools.wmflabs.org/replag/ is currently >80 hours for S4 (commonswiki.{analytics,web}.db.svc.eqiad.wmflabs and testcommonswiki.{analytics,web}.db.svc.eqiad.wmflabs) - preventing up-to-date results appearing in Quarry. Possibly related to T232446 according to @bd808 at https://www.mediawiki.org/w/index.php?title=Topic:Vfdhv5hm5lea1tlh&topic_showPostId=vfdoj9mudfhiopj5&fromnotif=1#flow-post-vfdoj9mudfhiopj5 - although that ticket talks about S8 not S4.

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 23 2020, 8:05 AM

That's due to a compression going on on the sanitarium host as part as T232446: Compress new Wikibase tables
I think it will be done in around 24h or so.

There are also some tables being compressed on s8, so delay is also expected there (but it won't be that much)
Sorry for the inconveniences.

JJMC89 removed a project: Quarry.
Mike_Peel renamed this task from S4 replication lab on Wikimedia Clouds is over 70 hours to S4 replication lab on Wikimedia Clouds is over 80 hours.Jan 23 2020, 8:04 PM
Mike_Peel updated the task description. (Show Details)
bd808 renamed this task from S4 replication lab on Wikimedia Clouds is over 80 hours to Wiki replicas replication lag for the s4 slice (commonswiki) is over 80 hours.Jan 23 2020, 8:24 PM
bd808 added a comment.EditedJan 23 2020, 8:27 PM

I am going to merge this into T232446: Compress new Wikibase tables. My reasoning is that the lag is by-product of the compression process. Nothing can be done to correct the lag until the compression is completed. It will also go away by itself once the servers have the processing power available to resume replication.

Maintenance on sanitarium's master is over, so it has started to slowly catch up