[SPIKE] Investigate search index delta variations
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mfossati
	Nov 20 2024, 4:43 PM

Description

T372912: Migrate image recommendation to use page_weighted_tags_changed stream will change how image suggestions send updates to search indices. In T372912#10336830 we discussed ways to optimize them.

The main question here is to understand delta variations, which usually impact Commons. See T372912#10336730.

NOTE: we compute scores for Commons weighted tags. The intuition is that big updates happen when lots of scores are different. How important are those scores for real-world search queries? The easiest optimization could just be to drop them if they're not used.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	add Commons delta threshold	repos/data-engineering/airflow-dags!974	mfossati	T380389	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T340437 [EPIC] Image suggestions data pipelines maintenance
		Resolved		mfossati	T380389 [SPIKE] Investigate search index delta variations

Event Timeline

mfossati created this task.Nov 20 2024, 4:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2024, 4:43 PM

mfossati added a parent task: T340437: [EPIC] Image suggestions data pipelines maintenance .Nov 20 2024, 4:43 PM

mfossati mentioned this in T372912: Migrate image recommendation to use page_weighted_tags_changed stream.

mfossati updated the task description. (Show Details)Nov 20 2024, 5:55 PM

mfossati updated the task description. (Show Details)

dcausse awarded a token.Nov 21 2024, 8:30 AM

dcausse subscribed.

Filter out Commons while we figure out the importance of its weighted tags.

mfossati edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Dec 9 2024, 4:55 PM

mfossati moved this task from Incoming to Ready for Development on the Structured-Data-Backlog (Current Work) board.

KStoller-WMF subscribed.Dec 9 2024, 7:30 PM

mfossati changed the task status from Open to In Progress.Dec 11 2024, 10:52 AM

mfossati claimed this task.

mfossati moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/46

Skip Commons delta if its row count is above a given threshold

mfossati opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/974

add Commons delta threshold

Moving to code review, but looking at Commons weighted tags usage in the meanwhile.

mfossati merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/46

Skip Commons delta if its row count is above a given threshold

mfossati merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/974

add Commons delta threshold

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2024, 5:30 PM

mfossati moved this task from Code Review to Doing on the Structured-Data-Backlog (Current Work) board.Dec 18 2024, 9:36 PM

We're focusing here on the following weighted tags that go to the Commons search index:

image.linked.from.wikidata.p18/QID|SCORE
image.linked.from.wikidata.p373/QID|SCORE
image.linked.from.wikipedia.lead_image/QID|SCORE

where QID is a Wikidata item and SCORE is computed in commonswiki_file.py.

These tags seem to be used in 2 ways:

implicitly - when searching the File namespace on any wiki with the WikibaseMediaInfo extension enabled, which should currently translate to when using MediaSearch or Special:Search on Commons
explicitly - via custommatch:depicts_or_linked_from=QID in a Commons search query

Example:

implicit - search query = ramones

...

{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikidata.p18\/Q483407",
			"boost": 2.342153943085914
		}
	}
},
{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikidata.p373\/Q483407",
			"boost": 4.386547106798469
		}
	}
},
{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikipedia.lead_image\/Q483407",
			"boost": 4.795040181639707
		}
	}
}

...

explicit - search query = custommatch:depicts_or_linked_from=Q483407, where Q483407 is the Wikidata item for Ramones.

...

{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikidata.p18\/Q483407",
			"boost": 1.984794590275781
		}
	}
},
{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikidata.p373\/Q483407",
			"boost": 5.739424158364518
		}
	}
},
{
	"match": {
		"weighted_tags": {
			"query": "image.linked.from.wikipedia.lead_image\/Q483407",
			"boost": 3.74983393205065
		}
	}
}


...

Note that these queries yield slightly different results.

Almost no search queries on Commons contain custommatch:depicts_or_linked_from:

def collect_searches(spark):
    initial_query = """SELECT http, params
    FROM event.mediawiki_cirrussearch_request
    WHERE database='commonswiki' AND params IS NOT NULL
    """
    ddf = spark.sql(initial_query)
    filtered = (
        ddf
        .where(
            ddf.params.title.contains('Special:Search') | ddf.params.title.contains('Special:MediaSearch')
        )
        .where(
            ddf.http.request_headers.referer.contains('index.php')
        )
    )

    return filtered

from wmfdata.spark import create_session
spark = create_session(app_name='commons-weighted-tags', type='yarn-large')
ddf = collect_searches(spark)
kw = 'custommatch:depicts_or_linked_from='
wt = ddf.where(ddf.params.search.contains(kw))
wt.count()

4

Moving to needs design, discussion with engineers needed.
FYI @Sneha you can safely ignore this ticket.

Follow-up ticket: T384514: Reduce Commons' search index delta size.

[SPIKE] Investigate search index delta variationsClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

[SPIKE] Investigate search index delta variations
Closed, ResolvedPublic
Actions

Related Objects
Search...