Page MenuHomePhabricator

Deploy mobile language button placement improvement to a larger-medium wiki on Tuesday week two of sprint
Closed, ResolvedPublic1 Estimate Story Points

Description

Enable the new mobile web language switcher affordance (button at top, keeping button at bottom with scroll to top thing for the time being) on ru.wikipedia.org.

Acceptance Criteria:

  • All subtasks resolved.
  • Ensure event logging reflects the new button placement

Details

Related Gerrit Patches:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jhernandez renamed this task from Deploy mobile language switching improvement to 1 large wikis to Deploy mobile language switching improvement to a subset of wikis.Jun 29 2016, 4:21 PM
Jhernandez updated the task description. (Show Details)
Jhernandez updated the task description. (Show Details)Jun 29 2016, 4:24 PM
Jhernandez renamed this task from Deploy mobile language switching improvement to a subset of wikis to Deploy mobile language button placement improvement to a subset of wikis.Jun 29 2016, 4:27 PM
Jhernandez updated the task description. (Show Details)Jun 29 2016, 4:33 PM

@Tbayer, presently the stable channel is at 1/1000 sampling. In order to observe the impact of positioning the language switcher button at the top of the article on a handful of wikis, do you recommend increasing the sampling rate? If so, do you mind if we increase the sampling rate on those wikis only? And at what rough sampling rates would you want for each of the three classes of wiki (small, medium, large) ?

@Jdlrobson I understand you have data about wiki sizes, can you recommend 2 small 2 medium and 1 large wiki to deploy this to that are different from the ones we're using for the performance work?

@JKatzWMF and @dr0ptp4kt to flesh this out and discuss technical blockers before the team reviews for estimation.

dr0ptp4kt updated the task description. (Show Details)Jul 11 2016, 4:51 PM

@dr0ptp4kt Need to define which wikis

i really would strongly advise deploying to all wikis.
Especially given T139794 Show users that language button has moved will behave differently on different wikis.

@Jdlrobson, suppose we picked only one wiki on which to do this instead of 5. What are the pros and cons of doing one wiki, followed by all VS. all wikis all at once?

One wiki

Pros:

  • There is a small chance we might hit problems with cached HTML but imo these can be caught through vigorous testing before a launch.
  • In the event of an issue any major problems can be contained to one wiki/rolled back

Cons:

  • Inconsistent/confusing ux for multilingual users where they use one wiki where it is enabled and one wiki where it is not. We will be pointing to the new language button on one but they would have to use it below on the other
  • Two SWAT deploys at least rather than 1

All wikis

Pros:

  • Any bugs will bubble up quicker given more users will be exposed to feature and through Village pump/mailing list/Twitter
  • Consistent UX across all wikis

Cons:

  • If something catastrophic happened e.g. API requests melted the server - would be hard to recover from.

Assessment

My assessment of the risk of something bad happening are very low. Moving the language button to the top will generate in more clicks but will it lead to an overwhelming amount of API calls? I think not. I think the consistent UX across wikis is very important (but @Nirzar should decide about that).

Given we've been in beta for some time I'm really not concerned about minor issues. If we find them I'd argue we can squash them quicker with a follow up SWAT.

dr0ptp4kt renamed this task from Deploy mobile language button placement improvement to a subset of wikis to Deploy mobile language button placement improvement to a larger-medium wiki.Jul 20 2016, 4:34 PM
dr0ptp4kt updated the task description. (Show Details)

I think it is very important to go with All wikis option.

Inconsistent/confusing ux for multilingual users where they use one wiki where it is enabled and one wiki where it is not. We will be pointing to the new language button on one but they would have to use it below on the other

This feature is for multilingual users, we can't afford to have this negative point about small wikis.

@Nirzar, to clarify your point, doing this for one wiki for one week would be unacceptable. Do I have that right?

@dr0ptp4kt yes. since the user will switch language to other wiki and the button will be in different place without any understanding of why.

If that's the consensus I'm ok with it.

My personal opinion is that we should production-test this on a wiki given we can't reproduce locally the production infrastructure with the caching conditions.

As a note, rollback is not that easy/clean if the feature (which changes server html) has been deployed for some time. Then even if we revert the patches, there may be cached HTML pages that were generated by the initial deploy, but now will be acted on by the reverted resources. To properly roll back we would need to purge, which is not a good idea (but better on one than 25).

It's also not that big of a change, so I understand where the confidence is coming from.

As I said, fine with whatever, both pros and cons are valid on their own.

One compromise is 1 deployment on a Wednesday followed by all others on the Thursday. Really problems with cached pages should have been thought about and tested on the beta cluster (we can test it there if we plan to prior to merging)

dr0ptp4kt renamed this task from Deploy mobile language button placement improvement to a larger-medium wiki to Deploy mobile language button placement improvement to a larger-medium wiki on Wednesday first day of sprint.Jul 25 2016, 4:10 PM
dr0ptp4kt renamed this task from Deploy mobile language button placement improvement to a larger-medium wiki on Wednesday first day of sprint to Deploy mobile language button placement improvement to a larger-medium wiki on Wednesday.Jul 25 2016, 4:16 PM

Copying in verbiage from another task:

@Jhernandez we can verify the cached HTML issue locally. I just verified this works as expected myself.
The verification step is as follows:

You can also do similar tests by checking out older revisions e.g.

dr0ptp4kt updated the task description. (Show Details)Jul 30 2016, 2:47 AM
dr0ptp4kt renamed this task from Deploy mobile language button placement improvement to a larger-medium wiki on Wednesday to Deploy mobile language button placement improvement to a larger-medium wiki on Tuesday.Aug 1 2016, 4:21 PM
dr0ptp4kt updated the task description. (Show Details)
dr0ptp4kt renamed this task from Deploy mobile language button placement improvement to a larger-medium wiki on Tuesday to Deploy mobile language button placement improvement to a larger-medium wiki on Tuesday week two of sprint.Aug 1 2016, 4:28 PM

I was making a case of enabling this on Hebrew wikipedia instead of Russian since it's a group 1 wiki, but then discovered a RTL, so roll out is blocked again.

Hebrew Wikipedia is a fine candidate, too. Sounds like T142016: Beta: Cannot access languages on main page and T142044: PointerOverlay misbehaves on RTL need to be in place in conjunction with or ahead of this one-wiki change. Feel free to update the Description with Hebrew Wikipedia, or do both Hebrew and Russian.

Change 302999 had a related patch set uploaded (by Jdlrobson):
Promote language switcher to top of page in Russian Wikipedia

https://gerrit.wikimedia.org/r/302999

I'm going to SWAT the fix to PointerOverlay today and then this
on Monday

Change 302999 merged by jenkins-bot:
Promote language switcher to top of page in Russian Wikipedia

https://gerrit.wikimedia.org/r/302999

Mentioned in SAL [2016-08-08T23:19:28Z] <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Promote language switcher to top of page on ruwiki (T138961) (duration: 00m 59s)

phuedx reassigned this task from Jdlrobson to dr0ptp4kt.Aug 9 2016, 9:53 AM

Some simple non-SQL join'd funnel analysis now that the change is coming into fuller force on ru.m.wikipedia.org is below. The following was with less than an hour remaining on 20160809. It's not worth drawing any conclusions right now other than that the eventing seems to look fairly healthy.

mysql:research@s6-analytics-slave.eqiad.wmnet [log]> select left(timestamp,8) ts, event_languageButtonVersion, event_event, count(*)
    -> from MobileWebLanguageSwitcher_15302503
    -> where timestamp > '201608'
    -> and webHost = 'ru.m.wikipedia.org'
    -> group by ts, event_event, event_languageButtonVersion
    -> order by ts, event_event, event_languageButtonVersion;
+----------+-----------------------------+--------------------------+----------+
| ts       | event_languageButtonVersion | event_event              | count(*) |
+----------+-----------------------------+--------------------------+----------+
| 20160801 | NULL                        | exitModal                |       25 |
| 20160801 | NULL                        | languageButtonImpression |     2713 |
| 20160801 | bottom-of-article           | languageButtonTap        |       30 |
| 20160801 | top-of-article              | languageButtonTap        |        1 |
| 20160801 | NULL                        | languageListLoaded       |       43 |
| 20160801 | NULL                        | pageLoaded               |     7005 |
| 20160801 | NULL                        | startLanguageSearch      |        1 |
| 20160802 | NULL                        | exitModal                |       28 |
| 20160802 | NULL                        | languageButtonImpression |     2689 |
| 20160802 | bottom-of-article           | languageButtonTap        |       42 |
| 20160802 | top-of-article              | languageButtonTap        |        1 |
| 20160802 | NULL                        | languageListLoaded       |       45 |
| 20160802 | NULL                        | pageLoaded               |     6977 |
| 20160802 | NULL                        | startLanguageSearch      |        1 |
| 20160803 | NULL                        | exitModal                |       16 |
| 20160803 | NULL                        | languageButtonImpression |     2736 |
| 20160803 | bottom-of-article           | languageButtonTap        |       23 |
| 20160803 | top-of-article              | languageButtonTap        |        1 |
| 20160803 | NULL                        | languageListLoaded       |       26 |
| 20160803 | NULL                        | pageLoaded               |     7011 |
| 20160804 | NULL                        | exitModal                |       27 |
| 20160804 | NULL                        | languageButtonImpression |     2739 |
| 20160804 | bottom-of-article           | languageButtonTap        |       41 |
| 20160804 | NULL                        | languageListLoaded       |       41 |
| 20160804 | NULL                        | pageLoaded               |     7019 |
| 20160805 | NULL                        | exitModal                |       20 |
| 20160805 | NULL                        | languageButtonImpression |     2653 |
| 20160805 | bottom-of-article           | languageButtonTap        |       26 |
| 20160805 | top-of-article              | languageButtonTap        |        6 |
| 20160805 | NULL                        | languageListLoaded       |       34 |
| 20160805 | NULL                        | pageLoaded               |     6874 |
| 20160806 | NULL                        | exitModal                |       22 |
| 20160806 | NULL                        | languageButtonImpression |     2924 |
| 20160806 | bottom-of-article           | languageButtonTap        |       25 |
| 20160806 | top-of-article              | languageButtonTap        |        3 |
| 20160806 | NULL                        | languageListLoaded       |       36 |
| 20160806 | NULL                        | pageLoaded               |     7635 |
| 20160807 | NULL                        | exitModal                |       31 |
| 20160807 | NULL                        | languageButtonImpression |     3227 |
| 20160807 | bottom-of-article           | languageButtonTap        |       32 |
| 20160807 | top-of-article              | languageButtonTap        |        1 |
| 20160807 | NULL                        | languageListLoaded       |       39 |
| 20160807 | NULL                        | pageLoaded               |     8156 |
| 20160807 | NULL                        | startLanguageSearch      |        1 |
| 20160808 | NULL                        | exitModal                |       23 |
| 20160808 | NULL                        | languageButtonImpression |     2826 |
| 20160808 | bottom-of-article           | languageButtonTap        |       24 |
| 20160808 | top-of-article              | languageButtonTap        |        1 |
| 20160808 | NULL                        | languageListLoaded       |       34 |
| 20160808 | NULL                        | pageLoaded               |     7272 |
| 20160808 | NULL                        | startLanguageSearch      |        3 |
| 20160809 | NULL                        | exitModal                |       17 |
| 20160809 | NULL                        | languageButtonImpression |     4841 |
| 20160809 | bottom-of-article           | languageButtonTap        |       11 |
| 20160809 | top-of-article              | languageButtonTap        |       11 |
| 20160809 | NULL                        | languageListLoaded       |       31 |
| 20160809 | NULL                        | pageLoaded               |     7564 |
+----------+-----------------------------+--------------------------+----------+
dr0ptp4kt closed this task as Resolved.Aug 10 2016, 12:51 PM

Here's data from about 12 hours on 20160810 (UTC). Signing off.

| 20160810 | NULL                        | exitModal                |        6 |
| 20160810 | NULL                        | languageButtonImpression |     2330 |
| 20160810 | top-of-article              | languageButtonTap        |        7 |
| 20160810 | NULL                        | languageListLoaded       |        9 |
| 20160810 | NULL                        | pageLoaded               |     2962 |

Did impressions of traffic go up?
Do we need to think about adjusting sampling rate? (Looks like it's getting clicked a lot more now?)

The ratio of impressions to page loads has gone up, as expected. It looks like the button tap to page load ratio is in the same ballpark right now, which is acceptable. Even if the tap ratio stays about the same, we've made it a lot easier for users to get to it. Granted, this is at the expense of future options for buttons in the page action bar, but it's okay.

I believe we can keep the sampling rate as is. It's true "unnecessary" button impression events will be fired, but for the time being this is the least complicated approach. Eventually we'll want to remove the eventing for these funnels generally anyway.

The ratio of impressions to page loads has gone up, as expected. It looks like the button tap to page load ratio is in the same ballpark right now, which is acceptable.

This is what I saw when I ran an informal query on beta--there didn't seem to be an impact on language clicks, which is surprising since we moved something from the bottom to the top of the page. 3 hypotheses:

  • Since we moved it to the top, we simplified the button from: "read in another language" to the much simpler: [symbol]. It might be that it will take some time for people to learn what [symbol] means.
  • It also might be that there is 0 volume impact because the people who would switch were willing to crawl through the mud to do it....and we simply made it a lot easier for them.
  • The last hypothesis I have is negative, which is that some large % of the time people realize they want to switch only when they reach the end of the article and are unsatisfied. If this is the case, they now have to scroll up and we have made their lives harder. I do not think this is what is happening. The lack of volume change (and our instinct) does not support this. In hindsight, more clarity would have come if we added this question to the design research we did or talked more with target users.