Page MenuHomePhabricator
Create Task
Maniphest T319291

[L] Reduce Cirrusdoc API payload
Closed, ResolvedPublic
Actions

Description

The current payload returned by cirrusDoc is currently quite large, and we are trying to find way to make its size smaller.

One possible solution would be to reduce the returned information from CirrusDoc. Currently this payload include lots of information that are not currently used by Search Preview.

Currently we just need the following fields:

  • heading -> for the sections
  • title, redirect.title, category, heading, text, auxiliary_text, file_text, source_text... -> for the snippets

Further information in this slack thread: https://wikimedia.slack.com/archives/CKZ5CPBQX/p1664883272109109

AC:

  • Expose from which field an article snippets comes from (wikitext or stripped version)
  • Define arguments in CirrusSearch\Api\QueryCirrusDoc to allow us to retrieve just a couple of relevant fields
  • Change the UI to request just the data it needs

Details

SubjectRepoBranchLines +/-
Reduce CirrusDoc request payloadmediawiki/extensions/SearchVuemaster+2 -1
Reduce Cirrusdoc API payloadmediawiki/extensions/CirrusSearchmaster+46 -9
Customize query in gerrit

Related Objects

Event Timeline

SimoneThisDot created this task.Oct 4 2022, 12:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2022, 12:34 PM
CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.Oct 4 2022, 12:46 PM
CBogen added a project: SDAW-Search-Improvements (Milestone 2: QuickView MVP).
gerritbot added a comment.Oct 4 2022, 2:08 PM

Change 838165 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/CirrusSearch@master] [POC] Expose field from which text is highlighted

https://gerrit.wikimedia.org/r/838165

gerritbot added a project: Patch-For-Review.Oct 4 2022, 2:08 PM
matthiasmullie renamed this task from [SPIKE] Cirrusdoc API investigation to reduce payload to Cirrusdoc API investigation to reduce payload.Oct 4 2022, 2:09 PM
matthiasmullie renamed this task from Cirrusdoc API investigation to reduce payload to Reduce Cirrusdoc API payload.Oct 4 2022, 2:16 PM
matthiasmullie updated the task description. (Show Details)
matthiasmullie subscribed.

Had a quick look; looks possible to expose the field. I updated the ticket description, removed "SPIKE" & I think we're ready to work on this.

CBogen renamed this task from Reduce Cirrusdoc API payload to [L] Reduce Cirrusdoc API payload.Oct 5 2022, 4:29 PM
CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Oct 5 2022, 4:40 PM
SimoneThisDot claimed this task.Oct 10 2022, 2:26 PM
SimoneThisDot moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.
gerritbot added a comment.Oct 11 2022, 10:38 AM

Change 841465 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo):

[mediawiki/extensions/CirrusSearch@master] Reduce Cirrusdoc API payload

https://gerrit.wikimedia.org/r/841465

SimoneThisDot moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Oct 13 2022, 8:36 PM
gerritbot added a comment.Oct 13 2022, 8:59 PM

Change 841465 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Reduce Cirrusdoc API payload

https://gerrit.wikimedia.org/r/841465

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.6; 2022-10-17).Oct 13 2022, 9:00 PM
SimoneThisDot moved this task from Code Review to Doing on the Structured-Data-Backlog (Current Work) board.Oct 14 2022, 9:28 AM
SimoneThisDot updated the task description. (Show Details)
SimoneThisDot updated the task description. (Show Details)
gerritbot added a comment.Oct 17 2022, 2:33 PM

Change 843507 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo):

[mediawiki/extensions/SearchVue@master] Reduce CirrusDoc request payload

https://gerrit.wikimedia.org/r/843507

SimoneThisDot moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Oct 17 2022, 2:33 PM
matthiasmullie moved this task from Code Review to Needs QA on the Structured-Data-Backlog (Current Work) board.Oct 21 2022, 7:40 AM
matthiasmullie updated the task description. (Show Details)
matthiasmullie mentioned this in T316397: [L] Quick view - Show the expanded search result snippet.Oct 21 2022, 7:43 AM
gerritbot added a comment.Oct 21 2022, 7:45 AM

Change 843507 merged by jenkins-bot:

[mediawiki/extensions/SearchVue@master] Reduce CirrusDoc request payload

https://gerrit.wikimedia.org/r/843507

matthiasmullie mentioned this in rESVUbb94e233bc1a: Reduce CirrusDoc request payload.Oct 21 2022, 7:46 AM
ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.7; 2022-10-24); removed MW-1.40-notes (1.40.0-wmf.6; 2022-10-17).Oct 21 2022, 8:00 AM
SimoneThisDot added a comment.Feb 2 2023, 10:22 PM

I can confirm that "Change the UI to request just the data it needs" has been completed and live since October.

The other open AC will be covered by the "expanding snippets" ticket

Etonkovidova closed this task as Resolved.Feb 15 2023, 11:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL