Page MenuHomePhabricator

Re-launch related pages a/b test on mobile enwiki
Closed, ResolvedPublic1 Story Points

Description

Launch related pages a/b test on mobile enwiki as follows:

  • feature enabled for 98% of users
  • feature disabled for 2% of users

Note: the instrumentation sampling rate should stay as-is, i.e. 1% of all eligible distinct user sessions (remembering that RelatedArticles is currently limited to mobile devices).

TODO

Sign off notes

  • Test will run for 19 days. Please fill in boxes to show which days they are

Deploy date Jun 27 2017
Test start date Jul 1 2017______ (2 days after to allow event logging groups to stabilise) (i've made this July to make it easier to filter data)
Test end date Jul 19 2017 ______ (19 days after start date)

Details

Related Gerrit Patches:
operations/mediawiki-config : masterRelaunch related pages A/B test to 98% of users

Event Timeline

ovasileva created this task.Jun 7 2017, 3:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2017, 3:30 PM
ovasileva updated the task description. (Show Details)Jun 7 2017, 4:14 PM
ovasileva renamed this task from Re-launch related pages a/b test to Re-launch related pages a/b test on mobile enwiki.Jun 7 2017, 4:24 PM
ovasileva updated the task description. (Show Details)
ovasileva triaged this task as High priority.Jun 7 2017, 4:27 PM
ovasileva set the point value for this task to 1.
ovasileva added a subscriber: Tbayer.
Restricted Application added a subscriber: Dereckson. · View Herald TranscriptJun 7 2017, 5:02 PM
Jdlrobson moved this task from To Do to Doing on the Readers-Web-Kanbanana-Board-Old board.
ovasileva updated the task description. (Show Details)Jun 7 2017, 5:10 PM

Change 357666 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Relaunch related pages A/B test to 98% of users

https://gerrit.wikimedia.org/r/357666

Change 357666 merged by jenkins-bot:
[operations/mediawiki-config@master] Relaunch related pages A/B test to 98% of users

https://gerrit.wikimedia.org/r/357666

Mentioned in SAL (#wikimedia-operations) [2017-06-07T23:28:03Z] <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Relaunch related pages A/B test to 98% of users on enwiki (T167310) (duration: 00m 44s)

Jdlrobson removed a project: Patch-For-Review.
Jdlrobson added a subscriber: Jdlrobson.

the a/b test is back on but I'd give it a few days for the A/B groups to stablise but hopefully this time things will work out nicely. It would be good to confirm things look good by rerunning queries before signing this off.

Sounds good, will retest tuesday, 6/13

phuedx removed ovasileva as the assignee of this task.Jun 23 2017, 9:36 AM

This is not blocked on train as it needs a swat anyway. Whenever we can find a swat window this can be done. :)

@Jdlrobson: The updates to the bucketing/removal of per-pageview sampling of events hasn't been fully deployed yet, right?

@Jdlrobson: The updates to the bucketing/removal of per-pageview sampling of events hasn't been fully deployed yet, right?

See description. That is what we will be swatting. Note, this task will fix itself with the n+1 train if we do nothing, but we would like to move it forward given the issues so far with this.

See description. That is what we will be swatting. Note, this task will fix itself with the n+1 train if we do nothing, but we would like to move it forward given the issues so far with this.

Ah! I thought those changes had made it into the most recent deployment branch.

phuedx updated the task description. (Show Details)Jun 26 2017, 11:47 AM

^ I've updated the description to reflect the current instrumentation sampling rate in the config.

I'm moving this to Blocked as it's unclear whether the conversation on T167236: userSessionToken in RelatedArticles schema does not seem to survive beyond one pageview (starting at T167236#3380485) has been resolved.

Tilman - the new changes are live. Can you run a query tomorrow to check if the bucketing is working as you would like it to be?

How does this look @Tbayer ?
Any more work needed...?

Jdlrobson closed this task as Resolved.Jul 6 2017, 11:41 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)

See https://phabricator.wikimedia.org/T167236#3414174 - the data is looking more sensible. Next steps are outlined in T169948
@Tbayer please reopen if you need anything else.