Is old treatment languages in the sidebar? Is new treatment language in the header?

Readers Web will be A/B testing the existing treatment of the language switcher and a new treatment being worked on as part of Desktop Improvements. The initial cohort will be all users (not just logged-in users).

Note, given the old and new treatment require HTML changes I think an A/B test could only feasibly be run here for logged in users. Is that a problem? Is that an explicit requirement? Is the A/B test bucketing planned based on page title?

i.e. we'll have to ship both features in a working state and then reveal one of them to the user.

I don't think shipping both is possible as the language list makes heavy use of IDs which gadgets may or may use and rendering both would create elements with duplicate IDs...

If we're prepared to break the non-JS version of the button we could adapt the existing implementation to always render in the sidebar and just render the button but this would require a considerable rearchitecture of the existing code and our feature flags and that might be workable (I wish I'd known about an anon A/B test earlier!).

In T275807#6864971, @Jdlrobson wrote:

Is old treatment languages in the sidebar? Is new treatment language in the header?

Yes.

Readers Web will be A/B testing the existing treatment of the language switcher and a new treatment being worked on as part of Desktop Improvements. The initial cohort will be all users (not just logged-in users).

Note, given the old and new treatment require HTML changes I think an A/B test could only feasibly be run here for logged in users. Is that a problem? Is that an explicit requirement? Is the A/B test bucketing planned based on page title?

It would be preferable to run the test across all users, as we currently have little information on language switching among anons.

i.e. we'll have to ship both features in a working state and then reveal one of them to the user.

I don't think shipping both is possible as the language list makes heavy use of IDs which gadgets may or may use and rendering both would create elements with duplicate IDs...

If we're prepared to break the non-JS version of the button we could adapt the existing implementation to always render in the sidebar and just render the button but this would require a considerable rearchitecture of the existing code and our feature flags and that might be workable (I wish I'd known about an anon A/B test earlier!).

Would it be possible to estimate how difficult this approach would be? Also, to confirm, this would allow us to test for both treatments as indicated above?

Definitely feasible, provided we're okay with a performance penalty (delay) in show languages demonstrated below:

A/B test will also need to consider cached HTML and the fact that the language button is tied to the new indicators placement.

In T275807#6864971, @Jdlrobson wrote:

Is old treatment languages in the sidebar? Is new treatment language in the header?

Readers Web will be A/B testing the existing treatment of the language switcher and a new treatment being worked on as part of Desktop Improvements. The initial cohort will be all users (not just logged-in users).

Note, given the old and new treatment require HTML changes I think an A/B test could only feasibly be run here for logged in users.

Not necessarily. We can, for example, ship both features in a working state but in the control state until the A/B test is enabled. We trade off a higher degree of accuracy in our analysis (including being able to draw conclusions about the behaviour of anons in this context) against performance and complexity for 3 weeks (2 weeks for the A/B test + 1 week for tidy up).

Is the A/B test bucketing planned based on page title?

Great question. Using page random as the unit of variance would mean that we don't have to ship both treatments at the same time, thereby reducing the performance cost of the A/B test, but at the cost of a slightly more fraught rollout as you have less granular control over when the A/B test is enabled for everyone (the A/B test is enabled on the page when it's invalidated in our edge cache(s)).

i.e. we'll have to ship both features in a working state and then reveal one of them to the user.

I don't think shipping both is possible as the language list makes heavy use of IDs which gadgets may or may use and rendering both would create elements with duplicate IDs...

This seems worthy of investigation. Should this be moved back into Needs Analysis?

Nice work on the proof of concept!

In T275807#6872967, @Jdlrobson wrote:

<snip /> and the fact that the language button is tied to the new indicators placement.

Could you expand on this?

Not necessarily. We can, for example, ship both features in a working state but in the control state until the A/B test is enabled. We trade off a higher degree of accuracy in our analysis (including being able to draw conclusions about the behaviour of anons in this context) against performance and complexity for 3 weeks (2 weeks for the A/B test + 1 week for tidy up).

Yes, we can ship both features in a working state, but we would need to consider how this is enabled with potential performance implications and complexity as highlighted in T275807#6872967

This seems worthy of investigation. Should this be moved back into Needs Analysis?

Provided we build one of the experiences in JavaScript we can avoid this issue. If we want to make the change in HTML this would delay things as we'd need to transition the new language button to use unique IDs and not be compatible with existing gadgets.

<snip /> and the fact that the language button is tied to the new indicators placement.
Could you expand on this?

There is a config flag $wgVectorLanguageInHeader which currently moves the language button to the header however it also changes the position of indicators in the DOM. I don't think this should impact the A/B test, however if it does, we will want to disentangle the indicator change from the button, preferably by shipping it prior to the A/B test and leaving some time for users to get used to it.

Thanks for the summary, @Jdlrobson.

I think there's another option between #1 and #2, which involves shipping both treatments in a visible loading state, which is a tried-and-tested method to give quick feedback to the user that something is happening thereby improving perceived performance.

See also T268504#6796185 for some discussion.

In T275807#6919266, @Jdlrobson wrote:

We talked today about implementing option 1 and adding timing metrics to get a sense of the impact before considering 2.

Next steps: putting option 1 on the beta cluster and testwiki to test the delay

Change 667716 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/Vector@master] A/B test for language button for anonymous users

https://gerrit.wikimedia.org/r/667716

Change 673151 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/Vector@master] Inform anonymous A/B test by tracking time from DOM interactive

https://gerrit.wikimedia.org/r/673151

Following up on this now we're actively investigating how to run this A/B test on logged out users:

In T275807#6864971, @Jdlrobson wrote:

i.e. we'll have to ship both features in a working state and then reveal one of them to the user.

I don't think shipping both is possible as the language list makes heavy use of IDs which gadgets may or may use and rendering both would create elements with duplicate IDs...

I quick global search in the gadget, gadget talk, and gadget definition namespaces yields no results for 'p-lang'. However, a code search for the same ID yields a host of codebases.

Is there a list of IDs that you had in mind?

Change 673151 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Inform anonymous A/B test by tracking time from navigationStart

https://gerrit.wikimedia.org/r/673151

In T275807#6934786, @phuedx wrote:

Following up on this now we're actively investigating how to run this A/B test on logged out users:

In T275807#6864971, @Jdlrobson wrote:

i.e. we'll have to ship both features in a working state and then reveal one of them to the user.

I don't think shipping both is possible as the language list makes heavy use of IDs which gadgets may or may use and rendering both would create elements with duplicate IDs...

I quick global search in the gadget, gadget talk, and gadget definition namespaces yields no results for 'p-lang'. However, a code search for the same ID yields a host of codebases.

Is there a list of IDs that you had in mind?

#p-lang, #p-lang-label

This might not be a problem in the wild, but it's also possible for arbitary IDs to be added via hook eg..

$wgHooks['SidebarBeforeOutput'][] = function ( $a, &$portlets ) {
	$portlets['LANGUAGES'][]= [ 'id' => 'klingon', 'text' => 'klingon', 'lang' => 'kling'];
};

Moved task to upcoming.
Next steps:

• @phuedx will backport on 24th March
• After offsite we will review data and make a decision about this task.
• Estimate if we decide to go ahead

Change 674382 had a related patch set uploaded (by Phuedx; owner: Jdlrobson):
[mediawiki/skins/Vector@wmf/1.36.0-wmf.36] Inform anonymous A/B test by tracking time from navigationStart

https://gerrit.wikimedia.org/r/674382

Change 674382 merged by jenkins-bot:
[mediawiki/skins/Vector@wmf/1.36.0-wmf.36] Inform anonymous A/B test by tracking time from navigationStart

https://gerrit.wikimedia.org/r/674382

Mentioned in SAL (#wikimedia-operations) [2021-03-25T11:44:08Z] <ladsgroup@deploy1002> Synchronized php-1.36.0-wmf.36/skins/Vector/resources: [[gerrit:674382|Inform anonymous A/B test by tracking time from navigationStart (T275807)]] (duration: 01m 09s)

Here's a graph of the lower bound, p50, and p75 of the Vector.ready timer in Graphite: https://graphite.wikimedia.org/S/r

Option 1 and its disadvantages

For clarity, I'm defining option 1 as having the server make the html for one of the language widgets but that widget would be hidden until our javascript either reveals it or moves it/transforms it into the other language widget based on the A/B bucket determined by our javascript.

One of the main downsides to this approach is that the language links in the sidebar would no longer be immediately visible to the user when that part of the page is painted by the browser (the status quo). From my anecdotal experience, modern Vector's first paint includes the rendering of the language links most of the time. With option 1, the languages would only be visible after the relevant javascript executes.

What our instrumentation measured

Given this, our instrumentation measured the number of miliseconds from the start of navigation (e.g. after the user enters en.wikipedia.org into their address bar and presses enter) to the point at which the JS in Vector starts to execute. The duration represents the earliest that the javascript would execute and could make the widget visible to the user.

What our instrumentation did not measure

It's important to keep in mind that our instrumentation did not collect data about the point at which the user first sees part of the site (first paint) or the point the user starts to interact with the site or would want to try to change languages (which would be much harder to measure). In other words, an unknown amount of the time represented by these graphs is time the user is waiting for the part of the page containing the language widget to render. That unknown amount of time is going to be a factor regardless of the option we choose for this ticket.

p50, p75 results

Having said that, the results suggest that 50% of users would see the language treatment within ~1.7 seconds after the start of navigation, and 75% of users would see the language treatment within ~2.5 seconds after the start of navigation. IMO, because these times are pretty fast, this population of people won't be affected as much by the delay caused by option 1.

p95, p99 results

The p95 is more variable than the others ranging anywhere from 5 seconds - 45 seconds, but most of the time staying in the 5 - 10 second range. The p99 is variable as well with even larger extremes. The p95, p99 population of people I'm much more concerned about as they could experience a significant amount of time without being able to see/interact with the language widget, However, I'm very curious what the first paint time is for this population and how that compares to the start of JS execution time we measured in our instrumentation. If the first paint time is within several seconds to the JS time, the delay may not matter as much vs. if the delta is > 10 or 20 seconds.

Conclusion

Given these results, I'm most concerned about the p95, p99 crowd of people and I think some more data (e.g. first paint info or some other reference point) could be helpful to make a more informed call, but I understand we have time constraints. We have several options. Perhaps we can schedule another meeting to talk about the next steps as a group?

cc @ovasileva, @Jdlrobson

Conclusion

Given these results, I'm most concerned about the p95, p99 crowd of people and I think some more data (e.g. first paint info or some other reference point) could be helpful to make a more informed call, but I understand we have time constraints. We have several options. Perhaps we can schedule another meeting to talk about the next steps as a group?

cc @ovasileva, @Jdlrobson

Thank you for reviewing this @nray! I agree that the p95 and p99 would be of largest concern here. Scheduled a meeting early next week to discuss in more detail.

In T275807#7001249, @nray wrote:

What our instrumentation did not measure

It's important to keep in mind that our instrumentation did not collect data about the point at which the user first sees part of the site (first paint)

See https://grafana.wikimedia.org/d/4SfKp_XGk/t275807?viewPanel=2&orgId=1 for a plot of NavigationTiming's firstPaint metric for the desktop platform (i.e. Vector) for all users alongside our vector.ready metric. Note well that you can select p50, p75, p95, and p99 for both metrics.

… or the point the user starts to interact with the site or would want to try to change languages (which would be much harder to measure).

The UniversalLanguageSelector instrument does record the number of milliseconds from when the instrument begins executing. Note well that this is not directly comparable to the above because they don't share the same time origin. That said, the p50, p75, and p95 percentiles of this metric are all typically an order of magnitude larger than the above (and, e.g. the same percentiles NavigationTiming's loadEventEnd metric):

+-----+---------+----------+-----------+-----------+
| day | median  |   p_75   |   p_95    |   p_99    |
+-----+---------+----------+-----------+-----------+
|   1 |   15475 | 34628.25 | 112324.25 | 215615.18 |
|   2 |   16468 |    34298 | 102582.55 | 295264.75 |
|   3 |   16072 |  33065.5 |  153204.1 | 550102.43 |
|   4 |   17584 |    37809 |    172180 | 1162738.4 |
|   5 | 17095.5 |    34890 |  209532.5 |  930779.7 |
|   6 |   16449 |  28576.5 |  100583.1 |  512025.9 |
|   7 |   15384 |  29142.5 |   91348.9 | 342737.72 |
|   8 |   15606 |  30855.5 |  141465.6 | 837037.16 |
|   9 |   17354 | 33196.25 |  148581.5 | 421404.14 |
|  10 |   16922 |  33505.5 |  150478.4 | 572283.76 |
|  11 | 17637.5 | 34359.25 |    112454 | 338986.52 |
|  12 |   16955 |    31440 |  123101.8 | 596427.12 |
|  13 |   17379 |  31495.5 |  108310.4 | 523139.82 |
|  14 |   17050 | 32645.25 | 152746.15 | 893564.69 |
|  15 |   16016 |    30851 |  125581.2 | 493262.36 |
|  16 | 16031.5 |  33114.5 |  117720.2 | 477142.57 |
|  17 |   17020 | 35059.75 | 125467.15 | 326368.08 |
|  18 | 16237.5 |  32233.5 | 118852.15 | 470346.86 |
|  19 | 15490.5 |  31041.5 | 103469.75 | 513156.09 |
|  20 | 18681.5 | 33026.75 |  96995.05 |  356272.1 |
+-----+---------+----------+-----------+-----------+

select
  day,
  round( percentile( cast( event.timetochangelanguage as bigint ), 0.5 ), 2 ) as median,
  round( percentile( cast( event.timetochangelanguage as bigint ), 0.75 ), 2 ) as p_75,
  round( percentile( cast( event.timetochangelanguage as bigint ), 0.95 ), 2 ) as p_95,
  round( percentile( cast( event.timetochangelanguage as bigint ), 0.99 ), 2 ) as p_99
from
  event.universallanguageselector
where
  -- https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/672742, included in the -wmf.37 branch was deployed on 1st April 2021.
  year = 2021
  and month = 04
  and event.action = 'language-change'
group by
  day
order by
  day asc
limit 10000
;

Change 667716 abandoned by Jdlrobson:

[mediawiki/skins/Vector@master] A/B test for language button for anonymous users

Reason:

https://gerrit.wikimedia.org/r/667716