MariaDB [enwiki_p]> select max(rev_timestamp) from revision;
+--------------------+
| max(rev_timestamp) |
+--------------------+
| 20200907080859     |
+--------------------+

(At time of writing, it is 2020-09-08T05:30:00Z.)

Worse, this does not seem to be improving - since I started investigating 15 minutes ago, the max rev timestamp has only moved from 20200907080856 to 20200907080859.

I noticed this because new users were not showing up in the editor interaction tool. Also, a thread on the enwiki village pump may be related.

There does not seem to be a problem with the replicas for any other wikis.

Here is the current data from heartbeat_p as of 2020-09-08T05:37:28Z:

MariaDB [heartbeat_p]> SELECT * FROM heartbeat
    -> ;
+-------+----------------------------+-------------+
| shard | last_updated               | lag         |
+-------+----------------------------+-------------+
| NULL  |                            |        NULL |
| s1    | 2020-09-08T05:37:28.001150 |      0.0000 |
| s2    | 2020-09-08T05:37:28.000970 |      0.0000 |
| s3    | 2020-09-08T05:37:28.001290 |      0.0000 |
| s4    | 2020-09-08T05:37:28.001150 |      0.0000 |
| s5    | 2020-09-08T05:37:28.000970 |      0.0000 |
| s6    | 2020-09-08T05:37:28.001190 |      0.0000 |
| s7    | 2020-09-08T05:37:28.001220 |      0.0000 |
| s8    | 2020-09-06T09:09:18.001260 | 160089.9987 |
+-------+----------------------------+-------------+
9 rows in set, 1 warning (0.00 sec)

Related Objects

Mentioned In: T212428: includes/Revision/RevisionStore.php: Main slot of revision (number) not found in database!
T262399: Edit Counter not working since since last 2 days
T262327: Untagged Uncategorized Articles not updating
Mentioned Here: T238966: Apply updates for MCR, actor migration, and content migration, to production wikis.
T254462: Compress enwiki InnoDB tables

Event Timeline

ST47 created this task.Sep 8 2020, 5:39 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 8 2020, 5:39 AM

bd808 added a project: DBA.Sep 8 2020, 4:53 PM

This is expected, there is maintenance going on on eqiad's master

For reference:
T254462: Compress enwiki InnoDB tables
And once the above is done, we'll also start with MCR schema changes,which means they'll also get lag: T238966: Apply updates for MCR, actor migration, and content migration, to production wikis.

There is nothing we can do to prevent that lag while doing this maintenance. I expect the lag to be there for a few more days.

Thank you Manuel! Apparently I just didn't know what to search for. What is "MCR"?

MCR stands for Multi Content Revision (https://mediawiki.org/wiki/Requests_for_comment/Multi-Content_Revisions)

It involves altering the huge revision table, which is massive on enwiki, and takes quite a few days - given that the cloud replicas already have lots of load, lag for a few days is to be expected unfortunately.

Wait, does this mean all Toolforge tools are frozen/broken as of 24 hours ago and will not receive any updates for the next few days? Anecdotally, that indeed appears to be the case. The MAX(enwiki.revision.rev_id) I find in WMCS is from 7 Sept 08:43 UTC which is more than 24 hours ago by now.

That seems UBN! imho as means a very large portion of tools is nearly completely unusable for editors.

I didn't think about this before, but do we not have a way for replicatation to continue from the current master in codfw instead?

Krinkle triaged this task as Unbreak Now! priority.Sep 8 2020, 9:44 PM

Krinkle added projects: cloud-services-team, Toolforge.

Restricted Application edited projects, added: cloud-services-team (Kanban); removed: cloud-services-team. · View Herald TranscriptSep 8 2020, 9:44 PM

Krinkle renamed this task from enwiki database replicas appear to be lagged and are falling further behind to enwiki database replicas (Toolforge and Cloud VPS) are more than 24h+ lagged.Sep 8 2020, 9:44 PM

Krinkle removed a project: Toolforge.

JJMC89 mentioned this in T262327: Untagged Uncategorized Articles not updating.Sep 8 2020, 10:02 PM

RhinosF1 subscribed.Sep 9 2020, 3:50 AM

@Krinkle when toolforge wiki replicas service was built, it was assumed that lag could happen anytime or even all the time.
In fact, for the last few months, lag has been present pretty much every day. The service is built on 3 hosts, that do multi source replication (that is, they have a single mysqld process that replicates all the traffic from production) and on top of that, they run a huge amount of heavy queries.
For the last few months, the servers have reached their limits and sometimes they cannot even cope with replication and they get behind on replication for a few days, and there is nothing we can do about it. We've tried to play with load balancer weights and lately it has improved a lot, but this is just a short-term solution.

We are working with WMCS to get a new architecture and more new servers to be able to keep the service up and running, as the actual model has reached its limit after almost 5 years running.

In T262239#6444637, @Krinkle wrote:

Wait, does this mean all Toolforge tools are frozen/broken as of 24 hours ago and will not receive any updates for the next few days? Anecdotally, that indeed appears to be the case. The MAX(enwiki.revision.rev_id) I find in WMCS is from 7 Sept 08:43 UTC which is more than 24 hours ago by now.

Yes, s1 will keep lagging for another 24h, and there is nothing we can do about it. When we operate on enwiki and its sizes, that is how much altering those tables take.

That seems UBN! imho as means a very large portion of tools is nearly completely unusable for editors.

There is nothing we can do about this, and the service is "a best effort" on that sense, lag can happen anytime given the nature of the infrastructure.

I didn't think about this before, but do we not have a way for replicatation to continue from the current master in codfw instead?

We can for certain operations, but for the operations that need to be done with replication we cannot do this, or even if we did, the problem would be the same as the operations have to come thru the replication thread, and if it takes 2 days to be done on the master, that's 2 days of lag until they arrive to the hosts, and then whatever time it takes to get done on the hosts themselves.

Moving the replicas to codfw isn't a super smooth process, it is very time consuming and very prone to errors as well. If we corrupt the data on the labsdb hosts, that means we need to recover 9TB of data, which means we need to depool another host (less capacity on the infra) and reclone it, which means less capacity on the service for a few hours/days and hence, more lag will show up.

It is a very complex and hard to operate infrastructure, and that's why, together with the WMCS team we are re-architecting the infra to make it more scalable, easy to operate and easy to recover in case of failure.

Hope this helps to clarify the situation.

Marostegui moved this task from Triage to In progress on the DBA board.Sep 9 2020, 5:38 AM

Flori4nK subscribed.Sep 9 2020, 8:02 AM

Nintendofan885 mentioned this in T262399: Edit Counter not working since since last 2 days.Sep 9 2020, 12:12 PM

JJMC89 merged a task: T262399: Edit Counter not working since since last 2 days.Sep 9 2020, 3:47 PM

JJMC89 added subscribers: Roller26, Nintendofan885.

bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.Sep 9 2020, 5:19 PM

bd808 moved this task from Inbox to Watching on the cloud-services-team (Kanban) board.

Lowering from UBN! to high. This is a real issue, but setting UBN! on a best effort support service is not highly useful.

taavi subscribed.Sep 9 2020, 6:58 PM

Nintendofan885 merged a task: T262508: XTools Statistics are not getting updated for the past couple of days.Sep 10 2020, 7:25 AM

Nintendofan885 added a subscriber: Rasnaboy.

BrownHairedGirl subscribed.Sep 10 2020, 1:23 PM

In T262239#6445454, @Marostegui wrote:

It is a very complex and hard to operate infrastructure, and that's why, together with the WMCS team we are re-architecting the infra to make it more scalable, easy to operate and easy to recover in case of failure.

Hope this helps to clarify the situation.

Thanks for that very helpful explanation. I understand that this is a slow process, and that disruption is inevitable while this necessary work is done. Some of my workflow relies on those tools,so it's good to know that I need to postpone some tasks.

Can you give any estimates of timelines for when service might resume?

In T262239#6450454, @BrownHairedGirl wrote:

In T262239#6445454, @Marostegui wrote:

It is a very complex and hard to operate infrastructure, and that's why, together with the WMCS team we are re-architecting the infra to make it more scalable, easy to operate and easy to recover in case of failure.

Hope this helps to clarify the situation.

Thanks for that very helpful explanation. I understand that this is a slow process, and that disruption is inevitable while this necessary work is done. Some of my workflow relies on those tools,so it's good to know that I need to postpone some tasks.

Thanks for understanding :-)

Can you give any estimates of timelines for when service might resume?

The service is effectively up, but with lag. I just took a quick look at the progress and I think it will start replicating again in around 24h, it will slowly catch up on replication, but given how loaded those hosts are, it will take maybe another day or two to be fully in sync with the master.
Keep in mind that this only affects enwiki and wikidatawiki. The rest of wikis are in sync with their respective masters.

Thank you

In T262239#6450462, @Marostegui wrote:

The service is effectively up, but with lag. I just took a quick look at the progress and I think it will start replicating again in around 24h, it will slowly catch up on replication, but given how loaded those hosts are, it will take maybe another day or two to be fully in sync with the master.
Keep in mind that this only affects enwiki and wikidatawiki. The rest of wikis are in sync with their respective masters.

I use only enwiki, so that's what concerns me.. So I'll assume that there's probably about three to five days in all before things have recovered ... with the possibility that it may take longer as people hammer the tools to catch up on postponed tasks.

I am glad that WMF is giving extra resources to the tools. They are hugely important for the productivity of many editors, and play a critical role in many maintenance tasks. Ensuring that they keep up with demand is a very important part of maintaining the 'pedia.

Thanks again for your good work.

Thank you for your understanding :-)

JJMC89 mentioned this in T212428: includes/Revision/RevisionStore.php: Main slot of revision (number) not found in database!.Sep 11 2020, 8:23 PM

It looks like the replication has catched up:

MariaDB [enwiki_p]> select max(rev_timestamp), replace(replace(replace(now(), '-', ''), ':', ''), ' ', '') as now from revision\G
*************************** 1. row ***************************
max(rev_timestamp): 20200913132253
               now: 20200913132255
1 row in set (0.01 sec)

MariaDB [wikidatawiki_p]> select max(rev_timestamp), replace(replace(replace(now(), '-', ''), ':', ''), ' ', '') as now from revision\G
*************************** 1. row ***************************
max(rev_timestamp): 20200913132402
               now: 20200913132403
1 row in set (0.00 sec)

For what is worth, the last maintenance that needs to happen on s1 is being run at the moment, I expect it to take 2 days or so to complete, so expect lag on labsdb hosts for enwiki.
If you want to follow its progress you can subscribe to: T238966

LSobanski subscribed.Sep 15 2020, 9:41 AM

In T262239#6458850, @Marostegui wrote:

For what is worth, the last maintenance that needs to happen on s1 is being run at the moment, I expect it to take 2 days or so to complete, so expect lag on labsdb hosts for enwiki.
If you want to follow its progress you can subscribe to: T238966

Thanks, Marostegui, but I'd need to do a lot of homework to make sense of T238966.

Can you give some sort of layperson's summary of how much longer this is likely to take? And when the tools will return to normal?

In T262239#6464980, @BrownHairedGirl wrote:

In T262239#6458850, @Marostegui wrote:

For what is worth, the last maintenance that needs to happen on s1 is being run at the moment, I expect it to take 2 days or so to complete, so expect lag on labsdb hosts for enwiki.
If you want to follow its progress you can subscribe to: T238966

Thanks, Marostegui, but I'd need to do a lot of homework to make sense of T238966.

Can you give some sort of layperson's summary of how much longer this is likely to take? And when the tools will return to normal?

Sure!
So right now the hosts are around 9h behind because they are executing a huge schema change on the revision table. I think it will take a few more hours to complete, so probably they will be lagging behind for around 24-30h.
This is the last big maintenance that needs to happen on enwiki.

Hope this helps.

Maintenance on s1 finished and labsdb hosts are in sync with enwiki master.

enwiki database replicas (Toolforge and Cloud VPS) are more than 24h+ laggedClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

enwiki database replicas (Toolforge and Cloud VPS) are more than 24h+ lagged
Closed, ResolvedPublic
Actions