Scores

Light improvement, but to be taken with a grain of salt.

current

F1 Score | 0.65874139069724
Precision@1 | 0.94964028776978
Precision@3 | 0.92439024390244
Precision@10 | 0.88924661944623
Precision@25 | 0.85384615384615
Precision@50 | 0.82200772200772
Precision@100 | 0.77702285361334
Recall | 0.60978165938865
Average precision | 0.51507706084643

synonyms

F1 Score | 0.66441314553991
Precision@1 | 0.95316159250585
Precision@3 | 0.93892215568862
Precision@10 | 0.90568475452196
Precision@25 | 0.87420460107685
Precision@50 | 0.8398285268901
Precision@100 | 0.79800498753117
Recall | 0.60344533515265
Average precision | 0.51973424941461

Response times

They're mostly ok, but I'm a little disappointed in the worst 5% response times with synonyms.
Curiously, they're worse than when I ran this locally.
I have another quick idea, though - will give that a quick shot tomorrow.

current

average: 371ms
median: 319ms
10%: 266ms
90%: 421ms
95%: 498ms
99%: 1491ms

synonyms

average: 522ms
median: 354ms
10%: 289ms
90%: 630ms
95%: 984ms
99%: 4231ms

Change 757906 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/WikibaseMediaInfo@master] Further narrow down which synonyms to use, and cap them

https://gerrit.wikimedia.org/r/757906

I submitted another patch to optimize synonyms.
Here's a rough overview of how synonyms are (to be) used:

Profile currently in use:

• we only use wikidata entities to match via P180/P6243
• no synonyms at all

Synonyms profile currently in master:

• all of the above default existing mediasearch stuff
• gather English label & aliases of the entities best matching the search term (where we have a score > 0.5, based on their similarity to the search term & their position)
• remove duplicate terms after normalizing (lowercasing, removing redundant whitespace & punctuation)
• remove terms that are a superset of another term (e.g. "wikimedia commons" will be stripped if "commons" is already being searched for) because they would yield no additional results that would not already be found

Synonyms profile in patch in CR:

• all of the above existing mediasearch + synonyms stuff
• remove synonyms that are too short (<= 1 byte; i.e. all latin characters and single digits); they're too generic (expensive and of little value)
• remove synonyms where the normalized form deviates too much from the original term with whitespace/punctuation (e.g. "C#" !== "c") for not being relevant/specific/accurate enough
• remove synonyms that are too similar to other terms already being search (e.g. "cat" & "cats", or "house cat" and "housecat"): they're often adding no extra value (the term would already be matched via the other version once stemmed), or very little compared to searching another completely different variant
• sort synonyms by score and "differentness", and remove all but the top 5: if one or multiple of the terms match a massive amount of documents, it'd leave a massive amount of documents to be scored; at least there's now an upper limit, and we're still getting the most valuable synonyms

I ran all 3 profiles through our set of 3000 labels terms and here's what I've got in terms of result "accuracy" & response times.
Note that the results are probably underestimated for synonyms (because our labeled data was retrieved via the original profile: we're unable to tell which additional good matches synonyms bring in). But even in this case, we're seeing a minor improvement in precision.
Note that the response times are probably an overestimation for all 3 - there's some extra latency because I'm running these through an SSH tunnel.

I suggest we merge the new patch: it seems to have no measurable negative impact on the results of the synonyms vs the current version, but it does further improve response times (and puts a hard cap on the extent to which they can increase)

Current (no synonyms)

This last one is odd; the other 2 profiles were faster, even though they can't possibly *be* faster; probably caused by external factors, which may also explain why my results with the current synonyms profile last week were so unexpectedly high)

Synonyms currently in master:

Synonyms after latest patch in CR:

Change 757906 merged by jenkins-bot:

[mediawiki/extensions/WikibaseMediaInfo@master] Further narrow down which synonyms to use, and cap them

https://gerrit.wikimedia.org/r/757906

In T299488#7657512, @matthiasmullie wrote:

Response times

They're mostly ok, but I'm a little disappointed in the worst 5% response times with synonyms.
Curiously, they're worse than when I ran this locally.
I have another quick idea, though - will give that a quick shot tomorrow.

Note: I since discovered that the normalizeFulltextScores trick which was supposed to be disabled (T296631) was accidentally re-enabled.
This was already known to have a significant impact on synonyms.
That's most why the earlier results were unexpectedly worse.
After disabling again, and with these new optimizations, I suspect we'll find that synonyms will no longer have a pronounced impact on response times.

Ran numbers again.

• Precision has improved a few percents
• We can't assess changes in recall, which is where the biggest impact should be
• I can't no longer spot an obvious performance issue; they are all within the same range - extremes (both low and high) seem inconsistent but similar enough between both profiles

Current (no synonyms)

Response times (limit 1) #1

Response times (limit 1) #2

Response times (limit 500) #1

Response times (limit 500) #2

Scores

With synonyms

Response times (limit 1) #1

Response times (limit 1) #2

Response times (limit 500) #1

Response times (limit 500) #2

Scores