Page MenuHomePhabricator

Deploy A/B test to test relaxing the retrieval query filter
Closed, ResolvedPublic

Description

I'd like to test 2 configs:

  • 3+ terms with 80% on stems and 66% on plain 80% on stems, expected ZRR change: -4.6% (1k sample enwiki search queries)
  • 4+ terms with 80% on stems and 66% on plain 80% on stems, expected ZRR change: -2.1% (1k sample enwiki search queries)

Event Timeline

dcausse created this task.Oct 5 2017, 2:54 PM

Change 382468 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@master] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382468

Change 382470 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Deploy recall A/B on enwiki

https://gerrit.wikimedia.org/r/382470

debt triaged this task as Normal priority.Oct 5 2017, 4:42 PM
debt added a subscriber: debt.

Will these tests run concurrently? How long will they run? :)

debt added a comment.Oct 10 2017, 5:16 PM

Based on our chat during standup, we'll run both these tests at the same time, using different buckets. We'll probably start this next Monday, after the code rides the train this week.

Moving back to in progress - this test is ready to go and will be turned on during the week of Oct 16, 2017.

Change 382468 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382468

Change 382470 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Prepare profiles for the recall A/B test on enwiki

https://gerrit.wikimedia.org/r/382470

Mentioned in SAL (#wikimedia-operations) [2017-10-16T13:45:19Z] <hashar@tin> Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)

Change 384529 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var

https://gerrit.wikimedia.org/r/384529

Change 384669 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.3] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/384669

Change 384529 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var

https://gerrit.wikimedia.org/r/384529

Change 384703 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var (take 2)

https://gerrit.wikimedia.org/r/384703

Change 384703 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Fix typo in UserTesting config var (take 2)

https://gerrit.wikimedia.org/r/384703

Mentioned in SAL (#wikimedia-operations) [2017-10-17T13:59:00Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:384703|[cirrus] Fix typo in UserTesting config var (take 2) (T177502)]] (duration: 00m 45s)

Change 384669 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.3] [cirrus] Turn on recall A/B test on enwiki

https://gerrit.wikimedia.org/r/384669

Mentioned in SAL (#wikimedia-operations) [2017-10-18T13:22:58Z] <zfilipin@tin> Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:384669|[cirrus] Turn on recall A/B test on enwiki (T177502)]] (duration: 00m 50s)

debt closed this task as Resolved.Oct 19 2017, 8:01 PM
debt claimed this task.