Let's use this ticket to turn off the A/B 2-way test for interleaved results. Expected time to have this test run is 7 days.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T174064 [FY 2017-18 Objective] Implement advanced search methodologies | |||
| Resolved | EBernhardson | T161632 [Epic] Improve search by researching and deploying machine learning to re-rank search results | |||
| Resolved | EBernhardson | T162369 Evaluate rescore windows for learning to rank | |||
| Resolved | None | T174066 [Q1 2017-18 Objective] Perform load and A/B tests on new models (interleaved search results) | |||
| Resolved | EBernhardson | T150032 Add support for interleaved results in 2-way A/B test | |||
| Resolved | debt | T171212 Interleaved results A/B test: turn on | |||
| Resolved | EBernhardson | T171213 Interleaved results A/B test: check that data is flowing the way we expect | |||
| Resolved | debt | T171214 Interleaved results A/B test: turn off test | |||
| Resolved | mpopov | T171215 Interleaved results A/B test: analysis of data | |||
| Declined | EBernhardson | T171984 Turn on test of LTR with standard AB buckets and an interleaved bucket. |
Event Timeline
Comment Actions
Change 372153 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] Disable cirrus MLR ab test
Comment Actions
Change 372169 had a related patch set uploaded (by Thcipriani; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.13] Disable cirrus MLR ab test
Comment Actions
Change 372170 had a related patch set uploaded (by Thcipriani; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.14] Disable cirrus MLR ab test
Comment Actions
Change 372169 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.13] Disable cirrus MLR ab test
Comment Actions
Change 372170 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.14] Disable cirrus MLR ab test
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-08-16T18:29:01Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:372170|Disable cirrus MLR ab test]] T171214 (duration: 00m 51s)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-08-16T18:30:32Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:372169|Disable cirrus MLR ab test]] T171214 (duration: 00m 50s)
Comment Actions
This was turned off on Aug 16, 2017; we are currently awaiting analysis to begin/complete in T171215.
Comment Actions
Change 372153 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Disable cirrus MLR ab test
Comment Actions
Re-opening as the sampling rates were set much lower than expected. Test now starts on aug 17th and will run through aug 24. The previous run of the test has valid data to analyze, but it only collected 1k sessions per bucket. This next run of the same test should collect 7k sessions per bucket if turned off on aug 24.
Comment Actions
Change 374394 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] Turn off Cirrus MLR AB Test
Comment Actions
Change 374394 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Turn off Cirrus MLR AB Test
Comment Actions
Change 374432 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.15] Turn off Cirrus MLR AB Test
Comment Actions
Change 374432 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.15] Turn off Cirrus MLR AB Test
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-08-28T23:13:34Z] <ebernhardson@tin> Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)