Page MenuHomePhabricator

Deploy A/B test to test relaxing the retrieval query filter
Closed, ResolvedPublic

Description

I'd like to test 2 configs:

  • 3+ terms with 80% on stems and 66% on plain 80% on stems, expected ZRR change: -4.6% (1k sample enwiki search queries)
  • 4+ terms with 80% on stems and 66% on plain 80% on stems, expected ZRR change: -2.1% (1k sample enwiki search queries)

Event Timeline

Change 382468 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@master] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382468

Change 382470 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Deploy recall A/B on enwiki

https://gerrit.wikimedia.org/r/382470

debt triaged this task as Medium priority.Oct 5 2017, 4:42 PM
debt subscribed.

Will these tests run concurrently? How long will they run? :)

Based on our chat during standup, we'll run both these tests at the same time, using different buckets. We'll probably start this next Monday, after the code rides the train this week.

Change 382468 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382468

Change 382470 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Prepare profiles for the recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382470

Mentioned in SAL (#wikimedia-operations) [2017-10-16T13:45:19Z] <hashar@tin> Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)

Change 384529 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var

https://gerrit.wikimedia.org/r/384529

Change 384669 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.3] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/384669

Change 384529 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var

https://gerrit.wikimedia.org/r/384529

Change 384703 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var (take 2)

https://gerrit.wikimedia.org/r/384703

Change 384703 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var (take 2)

https://gerrit.wikimedia.org/r/384703

Mentioned in SAL (#wikimedia-operations) [2017-10-17T13:59:00Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:384703|[cirrus] Fix typo in UserTesting config var (take 2) (T177502)]] (duration: 00m 45s)

Change 384669 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.3] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/384669

Mentioned in SAL (#wikimedia-operations) [2017-10-18T13:22:58Z] <zfilipin@tin> Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:384669|[cirrus] Turn on recall A/B test on enwiki (T177502)]] (duration: 00m 50s)

debt claimed this task.