Page MenuHomePhabricator

A/B test language switcher alternative
Closed, ResolvedPublic2 Story Points

Description

Details

Related Gerrit Patches:

Related Objects

Event Timeline

dr0ptp4kt raised the priority of this task from to High.
dr0ptp4kt updated the task description. (Show Details)
dr0ptp4kt updated the task description. (Show Details)
dr0ptp4kt set Security to None.
dr0ptp4kt added subscribers: Jhernandez, Nemo_bis, Jdlrobson and 5 others.
dr0ptp4kt updated the task description. (Show Details)Jan 19 2016, 12:30 AM
dr0ptp4kt updated the task description. (Show Details)Jan 19 2016, 12:49 AM

A few comments

  • What is the question we want to get answered in the A/B test? How will we know if one version is better than the other?.Can we work this out before committing to an A/B test?
  • A/B tests are usually best carried out on minor changes, not a bundle of changes (we're adding headings, different ordering of preferred languages, language count label, language codes and wikidata descriptions in one - it's likely one of those things could be very effective but one not so). There is a danger that if B performs better than A this might overlook something that A does very well. B doing well does not necessarily mean adding Wikidata descriptions is a good thing (and maybe adding Wikidata descriptions reduces clicks due to providing further information which might be seen as a good thing)
  • The beta mode of the mobile site has historically been not useful for A/B tests. The audience is tiny.
    • To this point, we should check how many users in beta are currently using the existing language switcher. Do we have enough to test a feature on existing users and get valuable feedback? I would guess not. We should aim to grow beta's multilingual audience if that's something we want to do. (I would have looked it up but that table takes over an hour to query)
dr0ptp4kt updated the task description. (Show Details)Feb 8 2016, 5:18 PM
dr0ptp4kt updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)Feb 8 2016, 5:20 PM
bd808 edited a custom field.Feb 8 2016, 5:22 PM
dr0ptp4kt updated the task description. (Show Details)Feb 13 2016, 3:18 PM
dr0ptp4kt updated the task description. (Show Details)Feb 13 2016, 3:21 PM

... wikidata descriptions in one ...

There are no wikidata descriptions in the structured language overlay.

bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-66-Harry is Tired board.

@dr0ptp4kt,

Instrumentation sample-in rate increased by a factor of 10 from current 13-February-2016 levels for the duration of the A/B test, in order that version B will have a sufficiently large number of events to characterize its usage.

Current instrumentation levels are:

'beta' => 0.1,
'stable' => 0.0001,

Are you suggesting we multiply both numbers by 10?

Change 271263 had a related patch set uploaded (by Bmansurov):
Change the language overlay configuration

https://gerrit.wikimedia.org/r/271263

Change 271264 had a related patch set uploaded (by Bmansurov):
Enable the structured language overlay and increase the instrumentation rate

https://gerrit.wikimedia.org/r/271264

dr0ptp4kt added a comment.EditedFeb 17 2016, 4:36 PM

@dr0ptp4kt,

Instrumentation sample-in rate increased by a factor of 10 from current 13-February-2016 levels for the duration of the A/B test, in order that version B will have a sufficiently large number of events to characterize its usage.

Current instrumentation levels are:

'beta' => 0.1,
'stable' => 0.0001,

Are you suggesting we multiply both numbers by 10?

Just the stable number. Let's SWAT the config in tandem with fully enhanced modal (tap count, generalization) landing on the prod cluster.

Once we've got enough data (weekend plus at least a full weekday) we can lower the sampling back down. I will create a task in anticipation of that.

Change 271263 merged by jenkins-bot:
Change the language overlay configuration

https://gerrit.wikimedia.org/r/271263

This is in the Ready for Signoff column, but there are several acceptance criteria to be fulfilled. In the interim, is there a URL or set of URLs in a lower environment to observe the structured overlay with the tweaks for the language tap count and the generalized language?

Should we move this task to To Do in anticipation of the SWAT of the mediawiki-config changes?

The config change [1] patch hasn't been merged yet. I'll create one for the beta labs too. I'll also post a URL to test once that's merged.

[1] https://gerrit.wikimedia.org/r/#/c/271264/

Change 271443 had a related patch set uploaded (by Bmansurov):
Enable the structured language overlay and increase the instrumentation rate

https://gerrit.wikimedia.org/r/271443

Change 271443 merged by jenkins-bot:
Labs: Enable the structured language overlay and increase the instrumentation rate

https://gerrit.wikimedia.org/r/271443

bmansurov added a comment.EditedFeb 18 2016, 12:15 AM

The labs patch has been merged. It will take about 10 mins before we can see the change. The new shiny peachy overlay can be found at http://en.m.wikipedia.beta.wmflabs.org/wiki/Language_test?mobileaction=beta#/languages . Feel free to add/remove languages to real articles on beta labs to test different features.

Cool. Seemed to be working for me on the beta cluster. Nice.

Okay, next up on this task is getting the master code to production and the config change SWAT'd.

I guess technically speaking master will start to roll out to the production wikis Tuesday, 23-February, but it won't be on Wikipedias until Thursday, 25-February.

Would it be safe to SWAT https://gerrit.wikimedia.org/r/#/c/271264 on Tuesday, 23-February so that we can get started on queries on the non-Wikipedia while I'm in the office? I'm OoO 25-February to 29-February, inclusive. Granted, we'd just need to make sure our queries on data don't rely upon 23-, 24-, and part of 25-February for mobile web Wikipedias because the A/B test wouldn't actually be in force on mobile web Wikipedias until the train delivers the updated MobileFrontend on Thursday, 25-February. But I'm mainly asking if it's safe because, even though my cursory read of the code and config seems to suggest it's fine, I wanted your finer judgment.

Yes, that sounds right. I've added the patch to the deployment queue on Tuesday, the 23rd. https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=316108&oldid=315973

Excellent. Thanks.

@bmansurov I noticed a bug. When navigating to the language overlay route directly (e.g. via the link you gave), the close button does not work. Not sure if this is a general overlay bug yet or just language switcher, but probably worth looking into before deploying.

Yes, it's a known feature. There are various reasons it works the way it does. I think there is a discussion somewhere here in phabricator about it.

Yes, it's a known feature. There are various reasons it works the way it does. I think there is a discussion somewhere here in phabricator about it.

Seems more like a known "feature." ;) I'll take a look around for the discussion and comment there; thanks for the info.

Jdlrobson moved this task from Backlog to Tasks on the MobileFrontend board.Feb 18 2016, 6:25 PM

Change 271264 merged by jenkins-bot:
Enable the structured language overlay and increase the instrumentation rate

https://gerrit.wikimedia.org/r/271264

The config change has been deployed. You can test the new overlay on WP at https://en.m.wikipedia.org/wiki/Book?mobileaction=beta#/languages when the changes roll out.

bmansurov removed bmansurov as the assignee of this task.Feb 25 2016, 1:20 PM
dr0ptp4kt closed this task as Resolved.Mar 10 2016, 7:24 PM
dr0ptp4kt updated the task description. (Show Details)
dr0ptp4kt moved this task from Tracking to Done on the Reading-Web-Sprint-67-If, Then, Else...? board.