Page MenuHomePhabricator

[Task] Make client ChangeHandler scale by batching updates
Open, HighPublic

Description

Per T104150: Investigation: Make sure client change handling scales: We should make the client change handling only handle a certain amount of changes (and pages) at once (per job run). If there are more changes to handle, another job needs to be added to queue.

This causes problem were the jobs time out, run to large queries and try to purge to many pages at once.

This mostly concerns ChangeHandler::handleChange which does getAffectedUsagesByPage and then tries to handle all of these at once. In order to address this T146838: Allow seeking in UsageLookup::getPagesUsing results needs to be resolved and afterwards ChangeHandler::handleChange needs to make use of that functionality and only do part of the updates at once.

Note: Fixing this might increase the amount of updates applied to a certain wiki by a huge amount, which could in turn negatively affect other parts of the infrastructure.

Event Timeline

hoo created this task.Aug 2 2015, 10:15 PM
hoo raised the priority of this task from to High.
hoo updated the task description. (Show Details)
hoo added subscribers: hoo, daniel, aude.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 2 2015, 10:15 PM
Jonas renamed this task from Make client change handling scale by batching updates to [Taks] Make client change handling scale by batching updates.Aug 13 2015, 2:21 PM
Jonas set Security to None.
aude renamed this task from [Taks] Make client change handling scale by batching updates to [Task] Make client change handling scale by batching updates.Aug 13 2015, 2:21 PM
hoo updated the task description. (Show Details)Oct 12 2015, 1:43 PM
RP88 added a subscriber: RP88.Nov 21 2015, 3:08 AM

Change 367328 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Move injection of RC records to a separate job.

https://gerrit.wikimedia.org/r/367328

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 23 2017, 4:02 PM

Change 367328 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Move injection of RC records to a separate job.

https://gerrit.wikimedia.org/r/367328

The patch for this is merged now. Should we make this resolved. It doesn't seem so.

@Ladsgroup The patch does not fully resolve this ticket, let's keep it open.

We should improve the ticket, though. It's not really clear what it's asking for. @hoo, can you write a bit more about what you had in mind? Currently, it's unclear what exactly needs to be done for this to be resolved.

Change 368224 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@wmf/1.30.0-wmf.10] Move injection of RC records to a separate job.

https://gerrit.wikimedia.org/r/368224

Change 368224 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.30.0-wmf.10] Move injection of RC records to a separate job.

https://gerrit.wikimedia.org/r/368224

hoo renamed this task from [Task] Make client change handling scale by batching updates to [Task] Make client ChangeHandler scale by batching updates.EditedJul 28 2017, 10:08 AM
hoo updated the task description. (Show Details)

@Ladsgroup The patch does not fully resolve this ticket, let's keep it open.
We should improve the ticket, though. It's not really clear what it's asking for. @hoo, can you write a bit more about what you had in mind? Currently, it's unclear what exactly needs to be done for this to be resolved.

I've appended the task description to add what I was thinking about here. I hope that it's clear enough now.