[M] Analyse results from the synonyms patch
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Cparle
	Sep 9 2021, 11:10 AM

Description

Once the synonyms patch is deployed to production I think we ought to:

modify AnalyzeResults.php so that it gathers response times from api calls
run AnalyzeResults.php with the patch turned on and off (after having run it once more, to prepopulate cache), and paste the results in a comment - also gather data for response times when the patch is on/off and paste here (e.g. median response time, average response time of slowest 10% of calls, average response time of fastest 10% of calls)
discuss results and determine whether to remove the feature flag from T258053 or whether more work is necessary and if so, what work is necessary

Related Objects

Mentioned In: T293106: MediaSearch with synonyms turned on sometimes takes MUCH longer than without
Mentioned Here: T258053: [L] Use English labels/aliases of matching wikidata entities to expand the wikitext, title, caption & categories search terms

Event Timeline

Cparle created this task.Sep 9 2021, 11:10 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2021, 11:10 AM

Cparle updated the task description. (Show Details)Sep 10 2021, 9:08 AM

Cparle updated the task description. (Show Details)

Aklapper added a project: SDAW-MediaSearch.Sep 12 2021, 7:06 PM

Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptSep 12 2021, 7:06 PM

CBogen updated the task description. (Show Details)Sep 13 2021, 4:43 PM

CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.

CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.

Possible follow on work to be discussed:

gather and rate new data relevant to synonyms (e.g. data for non-English languages) and store it in the labeled data repository with the tag "synonyms"
run AnalyzeResults.php with the patch turned on and off just on the data tagged with "synonyms", and paste the results in a comment

matthiasmullie updated the task description. (Show Details)Sep 29 2021, 4:38 PM

CBogen renamed this task from Analyse results from the synonyms patch to [M] Analyse results from the synonyms patch.Sep 29 2021, 4:38 PM

CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Sep 29 2021, 5:02 PM

Cparle claimed this task.Sep 30 2021, 10:37 AM

Cparle moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.

... actually, never mind - going to try again from toolforge to reduce noise due to network variance

Here are the results of the analysis run on toolforge

Synonyms

F1 Score	0.63958625148152
Precision@10	0.88936170212766
Precision@25	0.8572319201995
Precision@50	0.82246199117214
Precision@100	0.79898516783763
Recall	0.56042296072508
Average precision	0.47271739186655

Min response time	82ms
Max response time	10205ms
Average response time	367ms
Median response time	249ms
Average fastest 10%	116ms
Average slowest 10%	1383ms
Average 2nd slowest 10%	487ms

Control (without synonyms)

F1 Score	0.64963041182682
Precision@10	0.87753510140406
Precision@25	0.84721424468696
Precision@50	0.81352087114338
Precision@100	0.78035520115984
Recall	0.58180442595044
Average precision	0.4853565651381

Min response time	76ms
Max response time	2084ms
Average response time	211ms
Median response time	180ms
Average fastest 10%	99ms
Average slowest 10%	456ms
Average 2nd slowest 10%	373ms

Ok so the upshot of the above is some queries take a very long time - e.g. searching for "wikimedia commons" with the synonyms patch on causes a timeout.

Cparle mentioned this in T293106: MediaSearch with synonyms turned on sometimes takes MUCH longer than without.Oct 12 2021, 3:38 PM

Cparle closed this task as Resolved.Oct 20 2021, 11:40 AM

[M] Analyse results from the synonyms patchClosed, ResolvedPublicActions