Tune the new search profile created in T286563:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Cparle | T283869 [EPIC] Experiment with incorporating Image Matching Algorithm data into commons search index | |||
| Resolved | Cparle | T286565 [L] Compare accuracy of MediaSearch using the new weighted_fields data to data returned from Image Matching Algorithm |
First iteration of this adds a query on each of the new weighted_tags fields for each wikidata item we have found corresponding to the search term (the same set of wikidata ids we use for statement matching)
See https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseMediaInfo/+/732956/4/
Analysis:
Control
| F1 Score | 0.65022753730554 |
| Precision@10 | 0.8750981932443 |
| Precision@25 | 0.84593190998269 |
| Precision@50 | 0.81521739130435 |
| Precision@100 | 0.78602620087336 |
| Recall | 0.58093797276853 |
| Average precision | 0.4848566817268 |
Using weighted tags iteration 1
| F1 Score | 0.65741220346006 |
| Precision@10 | 0.86902927580894 |
| Precision@25 | 0.840820854132 |
| Precision@50 | 0.80975185023944 |
| Precision@100 | 0.77704576976422 |
| Recall | 0.59890524726312 |
| Average precision | 0.49781306577307 |
So either the new data is not useful, or I'm not using it in a useful way in the new search profile. Created T296309 to try a different approach
Iteration 2 (see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseMediaInfo/+/732956/6)
Using weighted tags iteration 2
| F1 Score | 0.643027356782 |
| Precision@1 | 0.95616438356164 |
| Precision@3 | 0.9271978021978 |
| Precision@10 | 0.89629068887207 |
| Precision@25 | 0.85527034407428 |
| Precision@50 | 0.82389380530973 |
| Precision@100 | 0.77945205479452 |
| Recall | 0.5817523283891 |
| Average precision | 0.48841066947357 |
So a 1% increase in precision@25, which is probably good enough to make use of in production. Might try one more iteration though ...
Iteration 3
Using weighted_tags iteration 3
| F1 Score | 0.68837871507386 |
| Precision@1 | 0.97822931785196 |
| Precision@3 | 0.95238095238095 |
| Precision@10 | 0.91096271563717 |
| Precision@25 | 0.86794019933555 |
| Precision@50 | 0.834179357022 |
| Precision@100 | 0.78722237455443 |
| Recall | 0.65648336727766 |
| Average precision | 0.57191605106942 |
So a little over 2% improvement for precision@25, with all other scores improved too. Bit disappointing tbh, but enough to proceed with weighted_tags on production