Page MenuHomePhabricator

Variant tests: roll out variant C/D
Open, Needs TriagePublic

Description

On October 19, we will enable variant C and D for newly created accounts. 40% of new accounts will get variant C, 40% will get variant D, and 20% will be in the control group that does not get the homepage or help panel. (Currently, 80% gets variant A and 20% is in the control group.) For existing accounts, nothing will change, they will continue to get variant A.

On October 26, we will remove variant A, and make variant D the default. Accounts that existed before October 19 (and newer accounts who manually enable the homepage) will get variant D.

Before Oct 19After Oct 19After Oct 26
Old accounts in treatment groupVar AVar AVar D
Old accounts who manually enabledVar AVar AVar D
Old accounts in control groupNothingNothingNothing
New accountsVar AVar C/DVar C/D
New accounts who manually enableVar AVar AVar D
New accounts in control groupNothingNothingNothing

Event Timeline

Catrope created this task.Oct 14 2020, 11:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2020, 11:05 PM

Change 634117 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments: Enable variant C/D for new users

https://gerrit.wikimedia.org/r/634117

Catrope set Due Date to Oct 19 2020, 6:00 PM.Oct 14 2020, 11:08 PM
MMiller_WMF renamed this task from Roll out variant C/D to Variant tests: roll out variant C/D.Oct 14 2020, 11:09 PM
MMiller_WMF moved this task from Incoming to Code Review on the Growth-Team (Current Sprint) board.
MMiller_WMF added subscribers: kostajh, Etonkovidova, Tgr and 4 others.

Change 634088 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/GrowthExperiments@master] Revert "Revert "Make variant D the default, and remove variant A""

https://gerrit.wikimedia.org/r/634088

kostajh assigned this task to Catrope.Oct 19 2020, 11:45 AM

Change 634117 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Enable variant C/D for new users

https://gerrit.wikimedia.org/r/634117

Mentioned in SAL (#wikimedia-operations) [2020-10-19T18:20:55Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Enable variant C/D for new users (T265556) (duration: 00m 56s)

Stage 1 is done, the patch for stage 2 has passed code review, so now we just need to wait for Oct 26 and deploy stage 2 then.

Change 635048 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments: Set default variant to D on trwiki

https://gerrit.wikimedia.org/r/635048

Change 635048 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Set default variant to D on trwiki

https://gerrit.wikimedia.org/r/635048

Mentioned in SAL (#wikimedia-operations) [2020-10-19T19:01:16Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Set default variant to D on trwiki (T243445, T265556) (duration: 00m 56s)

Change 635371 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments: Make variant D the default everywhere

https://gerrit.wikimedia.org/r/635371

Change 634088 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Revert "Revert "Make variant D the default, and remove variant A""

https://gerrit.wikimedia.org/r/634088

Change 635595 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Revert "Revert "Make variant D the default, and remove variant A""

https://gerrit.wikimedia.org/r/635595

Change 635669 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments: Make variant D the default on all wikis

https://gerrit.wikimedia.org/r/635669

Change 635669 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Make variant D the default on all wikis

https://gerrit.wikimedia.org/r/635669

Mentioned in SAL (#wikimedia-operations) [2020-10-26T18:10:51Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Make variant D the default on all wikis (T265556) (duration: 00m 58s)

Change 635595 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Revert "Revert "Make variant D the default, and remove variant A""

https://gerrit.wikimedia.org/r/635595

Mentioned in SAL (#wikimedia-operations) [2020-10-26T18:47:53Z] <catrope@deploy1001> Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/: Make variant D the default, remove variant A (T265372, T265556) (duration: 00m 58s)

Change 635371 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Remove variant setting override

https://gerrit.wikimedia.org/r/635371

Mentioned in SAL (#wikimedia-operations) [2020-10-26T18:59:37Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Remove variant setting override (no-op) (T265556) (duration: 00m 57s)

I just checked several production wikis and my legacy users all have Variant D. And accounts that didn't have it, when I flip it on in preferences, they get Variant D. That's all as expected.

@Etonkovidova -- could you please verify?

Etonkovidova added a comment.EditedFri, Oct 30, 11:58 PM

I just checked several production wikis and my legacy users all have Variant D. And accounts that didn't have it, when I flip it on in preferences, they get Variant D. That's all as expected.

@Etonkovidova -- could you please verify?

Yes, the legacy users (manually enabled users) have Variant D.

All seems to be in place, according to the table in the task description. I filed T267273: [arwiki] 'skip' Welcome survey redirects user to Special:UserLogin - the issue happens quite consistently.