Re-run AB test for Hebrew Wikipedia (has > 1% of search traffic) with new model
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	debt
	Dec 11 2017, 6:46 PM

Description

We want to re-run a test on Hebrew (follow-up from T175771) that has an updated model.

Details

Subject	Repo	Branch	Lines +/-
Remove cirrus AB test config for hewiki	operations/mediawiki-config	master	+0 -24
Turn off cirrus AB test on hewiki	mediawiki/extensions/WikimediaEvents	wmf/1.31.0-wmf.17	+1 -7
Turn off cirrus AB test on hewiki	mediawiki/extensions/WikimediaEvents	wmf/1.31.0-wmf.16	+1 -7
Turn off cirrus AB test on hewiki	mediawiki/extensions/WikimediaEvents	master	+1 -7
Turn on second mlr test for hewiki	mediawiki/extensions/WikimediaEvents	wmf/1.31.0-wmf.12	+7 -4
Turn on second mlr test for hewiki	mediawiki/extensions/WikimediaEvents	wmf/1.31.0-wmf.11	+7 -4
Turn on second mlr test for hewiki	mediawiki/extensions/WikimediaEvents	master	+7 -4
Setup MLR AB test for hewiki	operations/mediawiki-config	master	+13 -60

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T174064 [FY 2017-18 Objective] Implement advanced search methodologies
Resolved	EBernhardson	T161632 [Epic] Improve search by researching and deploying machine learning to re-rank search results
Resolved	EBernhardson	T182616 Re-run AB test for Hebrew Wikipedia (has > 1% of search traffic) with new model
Resolved	• chelsyx	T183024 Analysis of hewiki's A/B test (> 1% of search traffic with a new model)

Event Timeline

debt created this task.Dec 11 2017, 6:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2017, 6:46 PM

debt added a parent task: T161632: [Epic] Improve search by researching and deploying machine learning to re-rank search results.Dec 11 2017, 6:46 PM

Change 397973 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397973

Change 397582 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Setup MLR AB test for hewiki

https://gerrit.wikimedia.org/r/397582

EBernhardson claimed this task.Dec 12 2017, 10:34 PM

EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.

Change 397582 merged by jenkins-bot:
[operations/mediawiki-config@master] Setup MLR AB test for hewiki

https://gerrit.wikimedia.org/r/397582

Mentioned in SAL (#wikimedia-operations) [2017-12-13T00:05:38Z] <ebernhardson@tin> Synchronized wmf-config/: SWAT: T182616: Setup MLR AB test for hewiki (duration: 01m 10s)

Change 397973 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397973

Change 397984 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.11] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397984

Change 397985 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.12] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397985

Change 397984 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.11] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397984

Change 397985 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.12] Turn on second mlr test for hewiki

https://gerrit.wikimedia.org/r/397985

Mentioned in SAL (#wikimedia-operations) [2017-12-13T00:44:41Z] <ebernhardson@tin> Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)

Mentioned in SAL (#wikimedia-operations) [2017-12-13T00:49:48Z] <ebernhardson@tin> Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)).Dec 13 2017, 1:00 AM

Moving this to the new column - we're gathering data over the next several weeks (over the holidays) before we pick this work up again.

• chelsyx subscribed.Jan 2 2018, 11:51 PM

Since the holiday traffic that we've been gathering is 'different' in a lot of ways from our 'normal' (non-holiday) traffic, we'll basically keep this test going for another couple of weeks and discard the last couple of weeks that we've gathered.

We'll set the new dates as January 2 - 16th, 2018 to do collection and then it's onto analysis.

Change 404591 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] Turn off hewiki AB test

https://gerrit.wikimedia.org/r/404591

Change 404592 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Remove cirrus AB test config for hewiki

https://gerrit.wikimedia.org/r/404592

Change 404591 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Turn off cirrus AB test on hewiki

https://gerrit.wikimedia.org/r/404591

Change 404594 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.16] Turn off cirrus AB test on hewiki

https://gerrit.wikimedia.org/r/404594

Change 404595 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.17] Turn off cirrus AB test on hewiki

https://gerrit.wikimedia.org/r/404595

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)); removed MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)).Jan 16 2018, 11:00 PM

Change 404594 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.16] Turn off cirrus AB test on hewiki

https://gerrit.wikimedia.org/r/404594

Change 404595 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.31.0-wmf.17] Turn off cirrus AB test on hewiki

https://gerrit.wikimedia.org/r/404595

Mentioned in SAL (#wikimedia-operations) [2018-01-17T00:08:48Z] <ebernhardson@tin> Synchronized php-1.31.0-wmf.17/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: T182616 Turn off cirrus AB test on hewiki (duration: 01m 14s)

Mentioned in SAL (#wikimedia-operations) [2018-01-17T00:10:18Z] <ebernhardson@tin> Synchronized php-1.31.0-wmf.16/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: T182616 Turn off cirrus AB test on hewiki (duration: 01m 12s)

Change 404592 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove cirrus AB test config for hewiki

https://gerrit.wikimedia.org/r/404592

Mentioned in SAL (#wikimedia-operations) [2018-01-17T00:28:51Z] <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: T182616 Remove cirrus AB test config for hewiki (duration: 01m 09s)