Page MenuHomePhabricator
Create Task
Maniphest T353334

Batch page ID lookups in Wikibase entity search APIs
Closed, ResolvedPublic
Actions

Description

The action=wbsearchentities and action=query&list=wbsearch APIs (SearchEntities and QuerySearchEntities classes) currently make one database query per page ID they return. This should be batched.

Details

SubjectRepoBranchLines +/-
Use link batch in search APIsmediawiki/extensions/WikibaseREL1_39+48 -0
Make SearchEntitiesIntegrationTest an ApiTestCasemediawiki/extensions/WikibaseCirrusSearchREL1_39+9 -49
Use link batch in search APIsmediawiki/extensions/WikibaseREL1_40+48 -0
Make SearchEntitiesIntegrationTest an ApiTestCasemediawiki/extensions/WikibaseCirrusSearchREL1_40+9 -49
Use link batch in search APIsmediawiki/extensions/WikibaseREL1_41+48 -0
Make SearchEntitiesIntegrationTest an ApiTestCasemediawiki/extensions/WikibaseCirrusSearchREL1_41+9 -49
Use link batch in search APIsmediawiki/extensions/Wikibasewmf/1.42.0-wmf.9+48 -0
Make SearchEntitiesIntegrationTest an ApiTestCasemediawiki/extensions/WikibaseCirrusSearchwmf/1.42.0-wmf.9+9 -49
Use link batch in search APIsmediawiki/extensions/Wikibasemaster+48 -0
Make SearchEntitiesIntegrationTest an ApiTestCasemediawiki/extensions/WikibaseCirrusSearchmaster+9 -49
Customize query in gerrit

Related Objects
Search...

Event Timeline

Lucas_Werkmeister_WMDE created this task.Dec 13 2023, 3:38 PM
gerritbot added a comment.Dec 13 2023, 3:38 PM

Change 982818 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Use link batch in search APIs

https://gerrit.wikimedia.org/r/982818

gerritbot added a project: Patch-For-Review.Dec 13 2023, 3:38 PM
Lucas_Werkmeister_WMDE added a project: Wikidata Dev Team (Wikidata.org Slice).Dec 13 2023, 3:38 PM
Lucas_Werkmeister_WMDE moved this task from In Task Breakdown to Ready for Peer Review on the Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE mentioned this in T352877: Several efficiency errors in wbsearchentities API compound to long execution time and many database queries.
gerritbot added a comment.Dec 13 2023, 4:15 PM

Change 982850 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseCirrusSearch@master] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/982850

Maintenance_bot moved this task from Incoming to Ongoing on the wmde-wikidata-tech board.Dec 13 2023, 4:23 PM
gerritbot added a comment.Dec 18 2023, 5:37 PM

Change 982850 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@master] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/982850

gerritbot added a comment.Dec 18 2023, 5:39 PM

Change 982818 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Use link batch in search APIs

https://gerrit.wikimedia.org/r/982818

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.10; 2023-12-19).Dec 18 2023, 6:01 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 18 2023, 6:10 PM
gerritbot added a comment.Dec 19 2023, 2:32 PM

Change 984172 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseCirrusSearch@wmf/1.42.0-wmf.9] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/984172

gerritbot added a project: Patch-For-Review.Dec 19 2023, 2:32 PM

Change 984173 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.9] Use link batch in search APIs

https://gerrit.wikimedia.org/r/984173

gerritbot added a comment.Dec 19 2023, 2:51 PM

Change 984172 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@wmf/1.42.0-wmf.9] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/984172

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.9; 2023-12-12); removed MW-1.42-notes (1.42.0-wmf.10; 2023-12-19).Dec 19 2023, 3:00 PM
gerritbot added a comment.Dec 19 2023, 3:13 PM

Change 984173 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.9] Use link batch in search APIs

https://gerrit.wikimedia.org/r/984173

Stashbot added a comment.Dec 19 2023, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-19T15:13:41Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:984172|Make SearchEntitiesIntegrationTest an ApiTestCase (T353334)]], [[gerrit:984173|Use link batch in search APIs (T353334)]]

Stashbot added a comment.Dec 19 2023, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-19T15:15:22Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:984172|Make SearchEntitiesIntegrationTest an ApiTestCase (T353334)]], [[gerrit:984173|Use link batch in search APIs (T353334)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Dec 19 2023, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-19T15:22:31Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:984172|Make SearchEntitiesIntegrationTest an ApiTestCase (T353334)]], [[gerrit:984173|Use link batch in search APIs (T353334)]] (duration: 08m 49s)

gerritbot added a comment.Dec 19 2023, 3:24 PM

Change 984176 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseCirrusSearch@REL1_41] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/984176

gerritbot added a comment.Dec 19 2023, 3:24 PM

Change 984177 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_41] Use link batch in search APIs

https://gerrit.wikimedia.org/r/984177

gerritbot added a comment.Dec 19 2023, 3:41 PM

Change 984176 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@REL1_41] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/984176

Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to Ready for Development on the Wikidata Dev Team (Wikidata.org Slice) board.Dec 20 2023, 10:47 AM

Moving back to “ready for development” for backporting (currently semi-blocked on Disable Phan on release branches).

gerritbot added a comment.Jan 9 2024, 9:52 AM

Change 984177 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_41] Use link batch in search APIs

https://gerrit.wikimedia.org/r/984177

gerritbot added a comment.Jan 9 2024, 9:53 AM

Change 989122 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_40] Use link batch in search APIs

https://gerrit.wikimedia.org/r/989122

gerritbot added a comment.Jan 9 2024, 10:21 AM

Change 989123 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseCirrusSearch@REL1_40] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/989123

Lucas_Werkmeister_WMDE claimed this task.EditedJan 10 2024, 2:38 PM
Lucas_Werkmeister_WMDE moved this task from Ready for Development to In Development on the Wikidata Dev Team (Wikidata.org Slice) board.

(I should’ve reclaimed this earlier, I’ve been working on the backporting but there are some CI issues on REL1_41 REL1_40. This change and T354729 should hopefully resolve the issues.)

gerritbot added a comment.Jan 10 2024, 5:25 PM

Change 989123 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@REL1_40] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/989123

gerritbot added a comment.Jan 11 2024, 11:24 AM

Change 989122 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_40] Use link batch in search APIs

https://gerrit.wikimedia.org/r/989122

Maintenance_bot removed a project: Patch-For-Review.Jan 11 2024, 11:30 AM
gerritbot added a comment.Jan 11 2024, 2:53 PM

Change 989873 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_39] Use link batch in search APIs

https://gerrit.wikimedia.org/r/989873

gerritbot added a project: Patch-For-Review.Jan 11 2024, 2:53 PM
gerritbot added a comment.Jan 11 2024, 4:18 PM

Change 989882 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseCirrusSearch@REL1_39] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/989882

gerritbot added a comment.Jan 12 2024, 5:36 PM

Change 989882 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@REL1_39] Make SearchEntitiesIntegrationTest an ApiTestCase

https://gerrit.wikimedia.org/r/989882

gerritbot added a comment.Jan 15 2024, 9:54 AM

Change 989873 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_39] Use link batch in search APIs

https://gerrit.wikimedia.org/r/989873

Lucas_Werkmeister_WMDE closed this task as Resolved.Jan 15 2024, 9:55 AM
Lucas_Werkmeister_WMDE moved this task from In Development to Done on the Wikidata Dev Team (Wikidata.org Slice) board.

Alright, this is finally backported to all supported branches.

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2024, 10:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL