Page MenuHomePhabricator
Create Task
Maniphest T384514

Reduce Commons' search index delta size
Closed, ResolvedPublic
Actions

Description

In T372912#10336684, the Search team highlighted that large search index updates can be ~40x bigger than usual for Commons, and asked if this can be understood and optimized.
The intuition is that big updates happen when lots of weighted tag scores are different.

Possible solutions:

  • compute the delta after stripping the score from weighted tags
  • round the score to reduce variation at compute time, i.e., round(x / 10) * 10 or (1 + floor(x / 10)) * 10 - we decided to opt for this one
  • the delta computation is left intact

Then, talk to Search to ingest the initial big delta.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
new lead image score rounding formularepos/structured-data/image-suggestions!53mfossatiimprove-roundingmain
reduce the variation of relevance scoresrepos/structured-data/image-suggestions!50mfossatiT384514main
Customize query in GitLab

Related Objects
Search...

Event Timeline

mfossati created this task.Jan 22 2025, 5:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2025, 5:39 PM
mfossati added a parent task: T340437: [EPIC] Image suggestions data pipelines maintenance .Jan 22 2025, 5:40 PM
mfossati mentioned this in T380389: [SPIKE] Investigate search index delta variations.
mfossati updated the task description. (Show Details)Feb 6 2025, 11:21 AM
mfossati added a parent task: T378005: [SPIKE] ALIS code base review.Feb 7 2025, 5:18 PM
mfossati edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.
MarkTraceur moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.Feb 10 2025, 5:22 PM
mfossati changed the task status from Open to In Progress.Feb 18 2025, 11:17 AM
mfossati claimed this task.
mfossati moved this task from Ready for Estimation to Doing on the Structured-Data-Backlog (Current Work) board.
CodeReviewBot added a project: Patch-For-Review.Feb 21 2025, 5:55 PM

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/50

reduce the variation of relevance scores

mfossati moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Feb 21 2025, 6:16 PM
mfossati moved this task from Code Review to Doing on the Structured-Data-Backlog (Current Work) board.Mar 3 2025, 1:59 PM
CodeReviewBot added a comment.Mar 4 2025, 12:52 PM

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/53

new lead image score rounding formula

CodeReviewBot added a comment.Mar 4 2025, 2:38 PM

mfossati merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/50

reduce the variation of relevance scores

mfossati moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Mar 4 2025, 3:04 PM
mfossati added a subscriber: matthiasmullie.

@matthiasmullie the new formula doesn't seem to work better than the previous one, see https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/53.

CodeReviewBot added a comment.Mar 14 2025, 9:40 AM

mlitn merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/53

new lead image score rounding formula

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2025, 10:30 AM
matthiasmullie moved this task from Code Review to Verify on Production on the Structured-Data-Backlog (Current Work) board.Mar 14 2025, 1:32 PM

We ended up implementing a rounded log instead. It's not as efficient as the one we were initially considering, but it's much better at keeping the relevant nuance in the scores. If it turns out to insufficiently compress the delta, we can simply increase the log base to decrease the spread.

matthiasmullie claimed this task.Mar 14 2025, 1:33 PM
mfossati closed this task as Resolved.Apr 8 2025, 9:19 AM
mfossati added a subscriber: dcausse.

Commons delta row counts of last two scheduled runs:

select count(*) from analytics_platform_eng.image_suggestions_search_index_delta where snapshot='2025-03-24' and wikiid='commonswiki';
112205

select count(*) from analytics_platform_eng.image_suggestions_search_index_delta where snapshot='2025-03-17' and wikiid='commonswiki';
122904

CC @dcausse .

Looks good, closing.

dcausse added a comment.Apr 8 2025, 9:38 AM

@mfossati thanks! Do these two snapshots use the same dumps? if yes we might perhaps wait for a run that uses different dump and see?

dcausse added a comment.Apr 8 2025, 9:41 AM

Additionally if we're confident that we'll always have less than 500k tags/week we might consider unblocking T372912: Migrate image recommendation to use page_weighted_tags_changed stream

mfossati added a subscriber: Cparle.Apr 9 2025, 9:09 AM
In T384514#10720680, @dcausse wrote:

@mfossati thanks! Do these two snapshots use the same dumps?

Yes, they both use the usual inputs. FYI @Cparle is working towards consuming new weekly inputs in T389516: [L] Update ALIS to use wmf_content.mediawiki_content_history_v1 instead of wmf.*.

if yes we might perhaps wait for a run that uses different dump and see?

Sounds good.

In T384514#10720681, @dcausse wrote:

Additionally if we're confident that we'll always have less than 500k tags/week we might consider unblocking T372912: Migrate image recommendation to use page_weighted_tags_changed stream

I suggest to let the new logic run for a while and keep an eye on the numbers before confirming.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits