Page MenuHomePhabricator
Create Task
Maniphest T188381

Avoid joining three tables just to get the latest revision ID
Closed, ResolvedPublic
Actions

Description

WBQC uses WikiPageEntityMetaDataLookup to get the latest revision IDs for a list of entity IDs, but that lookup actually gets a lot more information than we need, joining three large tables (page, revision, text) in the process. We don’t need that – for just the revision IDs, page.page_latest should be enough.

Part of https://wikitech.wikimedia.org/wiki/Incident_documentation/20180226-WikibaseQualityConstraints.

Patch-For-Review:

Details

SubjectRepoBranchLines +/-
Handle redirects correctly in loadLatestRevisionIdsmediawiki/extensions/Wikibasemaster+90 -15
Rearrange PrefetchingWPEMDA::loadLatestRevisionIdsmediawiki/extensions/Wikibasemaster+10 -18
Optimize PrefetchingWPEMDA::loadLatestRevisionIdsmediawiki/extensions/Wikibasemaster+143 -23
Use WikiPageEntityMetaDataAcessor::loadLatestRevisionIdsmediawiki/extensions/WikibaseQualityConstraintsmaster+21 -34
Optimize WikiPageEntityMetaDataLookup::loadLatestRevisionIdsmediawiki/extensions/Wikibasemaster+61 -9
Extract two utility functionsmediawiki/extensions/Wikibasemaster+27 -12
Add WikiPageEntityMetaDataAccessor::loadLatestRevisionIdsmediawiki/extensions/Wikibasemaster+165 -0
Customize query in gerrit

Related Objects

Event Timeline

Lucas_Werkmeister_WMDE created this task.Feb 27 2018, 10:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 27 2018, 10:14 AM
Lucas_Werkmeister_WMDE added a subscriber: daniel.EditedFeb 27 2018, 10:18 AM

(Note: EntityRevisionLookup::getLatestRevisionId can also be used to get the latest revision ID for a single entity ID, and we actually used that prior to T182994: Get entity revision IDs in bulk. However, @daniel said that this is probably a legacy part of that interface, and the proper way to fix this task is to add a new method to WikiPageEntityMetaDataAccessor + …Lookup, not to extend EntityRevisionLookup.)

Lucas_Werkmeister_WMDE moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Feb 27 2018, 10:19 AM
Lucas_Werkmeister_WMDE added a project: Wikidata-Sprint-2018-02-28.Feb 27 2018, 1:30 PM
Lydia_Pintscher moved this task from incoming to in progress on the Wikidata board.Feb 27 2018, 1:35 PM
Lucas_Werkmeister_WMDE claimed this task.Feb 28 2018, 9:56 AM
Lucas_Werkmeister_WMDE moved this task from Backlog to In Progress on the Wikidata-Sprint-2018-02-28 board.
gerritbot subscribed.Feb 28 2018, 10:58 AM

Change 415258 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Add WikiPageEntityMetaDataAccessor::loadPageLatest

https://gerrit.wikimedia.org/r/415258

gerritbot added a project: Patch-For-Review.Feb 28 2018, 10:58 AM
gerritbot added a comment.Feb 28 2018, 1:00 PM

Change 415277 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Extract two utility functions

https://gerrit.wikimedia.org/r/415277

gerritbot added a comment.Feb 28 2018, 1:00 PM

Change 415278 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Optimize WikiPageEntityMetaDataLookup::loadPageLatest

https://gerrit.wikimedia.org/r/415278

Lucas_Werkmeister_WMDE moved this task from In Progress to Review on the Wikidata-Sprint-2018-02-28 board.Feb 28 2018, 1:00 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 2 2018, 11:10 AM
gerritbot added a comment.Mar 2 2018, 11:37 AM

Change 415258 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add WikiPageEntityMetaDataAccessor::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415258

gerritbot added a comment.Mar 2 2018, 2:50 PM

Change 415277 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Extract two utility functions

https://gerrit.wikimedia.org/r/415277

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 2 2018, 2:54 PM
thiemowmde triaged this task as Low priority.Mar 2 2018, 3:14 PM
thiemowmde added a project: Technical-Debt.
gerritbot added a comment.Mar 2 2018, 3:18 PM

Change 415865 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/WikibaseQualityConstraints@master] Use WikiPageEntityMetaDataAcessor::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415865

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 2 2018, 3:18 PM
gerritbot added a comment.Mar 2 2018, 5:31 PM

Change 415901 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] WIP: Optimize PrefetchingWPEMDA::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415901

Lucas_Werkmeister_WMDE added a comment.Mar 2 2018, 6:15 PM

I just saw this in WikiPageEntityRevisionLookup:

public function getLatestRevisionId( EntityId $entityId, $mode = self::LATEST_FROM_REPLICA ) {
    $rows = $this->entityMetaDataAccessor->loadRevisionInformation( [ $entityId ], $mode );
    $row = $rows[$entityId->getSerialization()];

    if ( $row && $row->page_latest && !$row->page_is_redirect ) {
        return (int)$row->page_latest;
    }

    return false;
}

Note that it’s only using page_latest if the page is not a redirect. We’re not currently doing that in loadLatestRevisionIds… is that a problem?

gerritbot added a comment.Mar 5 2018, 10:53 AM

Change 416410 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Wikibase@master] Rearrange PrefetchingWPEMDA::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/416410

Lucas_Werkmeister_WMDE added a project: Wikidata-Ministry-Of-Magic.Mar 6 2018, 10:42 AM
Lucas_Werkmeister_WMDE moved this task from Tasks to Needs Review on the Wikidata-Ministry-Of-Magic board.Mar 6 2018, 10:44 AM
ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)).Mar 6 2018, 1:01 PM
gerritbot added a comment.Mar 7 2018, 10:23 AM

Change 415278 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Optimize WikiPageEntityMetaDataLookup::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415278

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)); removed MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)).Mar 7 2018, 11:00 AM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 7 2018, 12:14 PM
gerritbot added a comment.Mar 7 2018, 4:39 PM

Change 415865 merged by jenkins-bot:
[mediawiki/extensions/WikibaseQualityConstraints@master] Use WikiPageEntityMetaDataAcessor::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415865

gerritbot added a comment.Mar 7 2018, 6:14 PM

Change 416997 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Handle redirects correctly in loadLatestRevisionIds

https://gerrit.wikimedia.org/r/416997

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 7 2018, 6:15 PM
gerritbot added a comment.Mar 8 2018, 9:45 AM

Change 415901 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Optimize PrefetchingWPEMDA::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/415901

gerritbot added a comment.Mar 8 2018, 10:16 AM

Change 416410 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Rearrange PrefetchingWPEMDA::loadLatestRevisionIds

https://gerrit.wikimedia.org/r/416410

Lucas_Werkmeister_WMDE moved this task from Needs Review to In Progress on the Wikidata-Ministry-Of-Magic board.Mar 8 2018, 10:57 AM
gerritbot added a comment.Mar 8 2018, 11:38 AM

Change 416997 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Handle redirects correctly in loadLatestRevisionIds

https://gerrit.wikimedia.org/r/416997

Lucas_Werkmeister_WMDE closed this task as Resolved.Mar 8 2018, 11:45 AM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Lydia_Pintscher moved this task from In Progress to Done on the Wikidata-Ministry-Of-Magic board.Mar 14 2018, 12:29 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:51 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 28 2020, 10:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits