Page MenuHomePhabricator

updateIsActiveFlagForMentees.php can be heavily optimized
Closed, ResolvedPublic1 Estimated Story Points

Description

Currently, updateIsActiveFlagForMentees.php queries the database individually for each mentee it processes. For huge wikis, that can take a significant portion of time (see the four hours in T432959#12200953, for example). Most methods updateIsActiveFlagForMentees.php use support batching, at least:

  • UserIdentityLookup::newSelectQueryBuilder()
  • UserRegistrationLookup::getFirstRegistrationBatch() (technically a difference from getRegistration(), which we use, but shouldn't be significant, plus easy to implement getRegistrationBatch anyway)

Unfortunately, UserEditTracker::getLatestEditTimestamp doesn't have a batched version, and adding it would be a ton of work (at the least, ActorStore would need to have a batched version too). Given actor table is now locked in (and as we query it anyway), a direct query seems easiest at the moment.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change #1324267 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] tests: Add integration test for updateIsActiveFlagForMentees

https://gerrit.wikimedia.org/r/1324267

Change #1324269 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@master] UserRegistrationLookup: Implement getRegistrationBatch

https://gerrit.wikimedia.org/r/1324269

Change #1324270 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Speed updateIsActiveFlagForMentees up a bit

https://gerrit.wikimedia.org/r/1324270

Urbanecm_WMF triaged this task as Medium priority.

Change #1324267 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] tests: Add integration test for updateIsActiveFlagForMentees

https://gerrit.wikimedia.org/r/1324267

Change #1324269 merged by jenkins-bot:

[mediawiki/core@master] UserRegistrationLookup: Implement getRegistrationBatch

https://gerrit.wikimedia.org/r/1324269

Change #1324270 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Speed updateIsActiveFlagForMentees up a bit

https://gerrit.wikimedia.org/r/1324270

[urbanecm@deploy1003 ~]$ kubectl get jobs -l cronjob=growthexperiments-updateisactiveflagformentees
NAME                                                      STATUS     COMPLETIONS   DURATION   AGE
growthexperiments-updateisactiveflagformentees-29782662   Complete   1/1           28m        7d1h
growthexperiments-updateisactiveflagformentees-29792742   Complete   1/1           18m        119m
[urbanecm@deploy1003 ~]$

It used to be about 4 hours before those changes. Mission completed!