- Ensure event instrumentation indicates which version is being used in the modal! Using T123932: Instrument mobile web language switching user workflow
- Ensure tap count is relayed correctly for first time users with LocalStorage. T126818: Send correct event logging languageButtonTappedBucket value when client LocalStorage capable and first use of switcher
- Ensure version B has final enhancement. T126868: In mobile language switcher prioritize available general language if user's specific client side variant isn't available
- 100% of users see the new version B in beta
- Instrumentation sample-in rate increased by a factor of 10 from current 13-February-2016 levels for the duration of the A/B test, in order that version B will have a sufficiently large number of events to characterize its usage.
- A/B test in mobile web stable - 90% should get version A (old), 10% should get version B (new)
- Analysis by product owner
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • JKatzWMF | T121919 [GOAL] Improve in-article language switching on mobile web I | |||
| Resolved | None | T126236 [Release] Verify alternative language switcher working on prod cluster in beta channel | |||
| Resolved | None | T123980 A/B test language switcher alternative | |||
| Resolved | • bmansurov | T123384 Build language overlay design alternative | |||
| Resolved | • bmansurov | T126868 In mobile language switcher prioritize available general language if user's specific client side variant isn't available | |||
| Resolved | • bmansurov | T126818 Send correct event logging languageButtonTappedBucket value when client LocalStorage capable and first use of switcher |
Event Timeline
Comment Actions
A few comments
- What is the question we want to get answered in the A/B test? How will we know if one version is better than the other?.Can we work this out before committing to an A/B test?
- A/B tests are usually best carried out on minor changes, not a bundle of changes (we're adding headings, different ordering of preferred languages, language count label, language codes and wikidata descriptions in one - it's likely one of those things could be very effective but one not so). There is a danger that if B performs better than A this might overlook something that A does very well. B doing well does not necessarily mean adding Wikidata descriptions is a good thing (and maybe adding Wikidata descriptions reduces clicks due to providing further information which might be seen as a good thing)
- The beta mode of the mobile site has historically been not useful for A/B tests. The audience is tiny.
- To this point, we should check how many users in beta are currently using the existing language switcher. Do we have enough to test a feature on existing users and get valuable feedback? I would guess not. We should aim to grow beta's multilingual audience if that's something we want to do. (I would have looked it up but that table takes over an hour to query)
Comment Actions
... wikidata descriptions in one ...
There are no wikidata descriptions in the structured language overlay.
Comment Actions
Instrumentation sample-in rate increased by a factor of 10 from current 13-February-2016 levels for the duration of the A/B test, in order that version B will have a sufficiently large number of events to characterize its usage.
Current instrumentation levels are:
'beta' => 0.1, 'stable' => 0.0001,
Are you suggesting we multiply both numbers by 10?
Comment Actions
Change 271263 had a related patch set uploaded (by Bmansurov):
Change the language overlay configuration
Comment Actions
Change 271264 had a related patch set uploaded (by Bmansurov):
Enable the structured language overlay and increase the instrumentation rate
Comment Actions
Just the stable number. Let's SWAT the config in tandem with fully enhanced modal (tap count, generalization) landing on the prod cluster.
Once we've got enough data (weekend plus at least a full weekday) we can lower the sampling back down. I will create a task in anticipation of that.
Comment Actions
This is in the Ready for Signoff column, but there are several acceptance criteria to be fulfilled. In the interim, is there a URL or set of URLs in a lower environment to observe the structured overlay with the tweaks for the language tap count and the generalized language?
Comment Actions
Should we move this task to To Do in anticipation of the SWAT of the mediawiki-config changes?
Comment Actions
The config change [1] patch hasn't been merged yet. I'll create one for the beta labs too. I'll also post a URL to test once that's merged.
Comment Actions
Change 271443 had a related patch set uploaded (by Bmansurov):
Enable the structured language overlay and increase the instrumentation rate
Comment Actions
Change 271443 merged by jenkins-bot:
Labs: Enable the structured language overlay and increase the instrumentation rate
Comment Actions
The labs patch has been merged. It will take about 10 mins before we can see the change. The new shiny peachy overlay can be found at http://en.m.wikipedia.beta.wmflabs.org/wiki/Language_test?mobileaction=beta#/languages . Feel free to add/remove languages to real articles on beta labs to test different features.
Comment Actions
Cool. Seemed to be working for me on the beta cluster. Nice.
Okay, next up on this task is getting the master code to production and the config change SWAT'd.
I guess technically speaking master will start to roll out to the production wikis Tuesday, 23-February, but it won't be on Wikipedias until Thursday, 25-February.
Would it be safe to SWAT https://gerrit.wikimedia.org/r/#/c/271264 on Tuesday, 23-February so that we can get started on queries on the non-Wikipedia while I'm in the office? I'm OoO 25-February to 29-February, inclusive. Granted, we'd just need to make sure our queries on data don't rely upon 23-, 24-, and part of 25-February for mobile web Wikipedias because the A/B test wouldn't actually be in force on mobile web Wikipedias until the train delivers the updated MobileFrontend on Thursday, 25-February. But I'm mainly asking if it's safe because, even though my cursory read of the code and config seems to suggest it's fine, I wanted your finer judgment.
Comment Actions
Yes, that sounds right. I've added the patch to the deployment queue on Tuesday, the 23rd. https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=316108&oldid=315973
Comment Actions
@bmansurov I noticed a bug. When navigating to the language overlay route directly (e.g. via the link you gave), the close button does not work. Not sure if this is a general overlay bug yet or just language switcher, but probably worth looking into before deploying.
Comment Actions
Yes, it's a known feature. There are various reasons it works the way it does. I think there is a discussion somewhere here in phabricator about it.
Comment Actions
Seems more like a known "feature." ;) I'll take a look around for the discussion and comment there; thanks for the info.
Comment Actions
Change 271264 merged by jenkins-bot:
Enable the structured language overlay and increase the instrumentation rate
Comment Actions
The config change has been deployed. You can test the new overlay on WP at https://en.m.wikipedia.org/wiki/Book?mobileaction=beta#/languages when the changes roll out.