Page MenuHomePhabricator

Re-launch related pages a/b test on mobile enwiki
Closed, ResolvedPublic1 Estimated Story Points

Description

Launch related pages a/b test on mobile enwiki as follows:

  • feature enabled for 98% of users
  • feature disabled for 2% of users

Note: the instrumentation sampling rate should stay as-is, i.e. 1% of all eligible distinct user sessions (remembering that RelatedArticles is currently limited to mobile devices).

TODO

Sign off notes

  • Test will run for 19 days. Please fill in boxes to show which days they are

Deploy date Jun 27 2017
Test start date Jul 1 2017______ (2 days after to allow event logging groups to stabilise) (i've made this July to make it easier to filter data)
Test end date Jul 19 2017 ______ (19 days after start date)

Event Timeline

ovasileva renamed this task from Re-launch related pages a/b test to Re-launch related pages a/b test on mobile enwiki.Jun 7 2017, 4:24 PM
ovasileva updated the task description. (Show Details)
ovasileva set the point value for this task to 1.
ovasileva added a subscriber: Tbayer.

Change 357666 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Relaunch related pages A/B test to 98% of users

https://gerrit.wikimedia.org/r/357666

Change 357666 merged by jenkins-bot:
[operations/mediawiki-config@master] Relaunch related pages A/B test to 98% of users

https://gerrit.wikimedia.org/r/357666

Mentioned in SAL (#wikimedia-operations) [2017-06-07T23:28:03Z] <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Relaunch related pages A/B test to 98% of users on enwiki (T167310) (duration: 00m 44s)

Jdlrobson removed a project: Patch-For-Review.
Jdlrobson subscribed.

the a/b test is back on but I'd give it a few days for the A/B groups to stablise but hopefully this time things will work out nicely. It would be good to confirm things look good by rerunning queries before signing this off.

Sounds good, will retest tuesday, 6/13

This is not blocked on train as it needs a swat anyway. Whenever we can find a swat window this can be done. :)

@Jdlrobson: The updates to the bucketing/removal of per-pageview sampling of events hasn't been fully deployed yet, right?

@Jdlrobson: The updates to the bucketing/removal of per-pageview sampling of events hasn't been fully deployed yet, right?

See description. That is what we will be swatting. Note, this task will fix itself with the n+1 train if we do nothing, but we would like to move it forward given the issues so far with this.

See description. That is what we will be swatting. Note, this task will fix itself with the n+1 train if we do nothing, but we would like to move it forward given the issues so far with this.

Ah! I thought those changes had made it into the most recent deployment branch.

Tilman - the new changes are live. Can you run a query tomorrow to check if the bucketing is working as you would like it to be?

How does this look @Tbayer ?
Any more work needed...?

Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)

See https://phabricator.wikimedia.org/T167236#3414174 - the data is looking more sensible. Next steps are outlined in T169948
@Tbayer please reopen if you need anything else.