Page MenuHomePhabricator

Add "source" to A/B test schema for DYM suggestions
Closed, ResolvedPublic

Description

Analysis of the A/B test for Glent Method 0 (M0) has proven to be a bit difficult because of the large variability in user behavior between the buckets, which is dominated by the performance of the phrase suggester suggestions.

In order to be better able to see the impact of M0 (and M‍1 and M‍2 in the future), add "source" to the logging schema so that for any given suggestion shown to a user, we know which algorithm it comes from. This will allow us to evaluate phrase suggester perfomance against M0 performance per query, rather than in the aggregate (where the phrase suggester currently dominates).

Event Timeline

TJones created this task.Nov 13 2019, 7:31 PM

Change 554605 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] [WIP] Apply FallbackMethod changes from atomic units

https://gerrit.wikimedia.org/r/554605

EBernhardson triaged this task as Medium priority.

Change 566849 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] search satisfaction: Track profile providing search results and query suggestions

https://gerrit.wikimedia.org/r/566849

Change 554605 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Apply FallbackMethod changes from atomic units

https://gerrit.wikimedia.org/r/554605

Change 566849 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] search satisfaction: Track profile providing search results and query suggestions

https://gerrit.wikimedia.org/r/566849

Change 576404 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576404

Checked recorded logs, desired data was not included due to a mismatch in variable names across repositories Above patch should fix it

Change 576404 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576404

Change 576446 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576446

Change 576447 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576447

Change 576446 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576446

Change 576447 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Match fallback config key with the one used in cirrus

https://gerrit.wikimedia.org/r/576447

Gehel closed this task as Resolved.Mar 11 2020, 3:21 PM