How does this fit in with the existing instrumentation https://meta.wikimedia.org/wiki/Schema:UniversalLanguageSelector ?
We probably want to talk to the language engineering team regardless of whether we're using this or not as we'd need to disable it if building something new and we'd need to amend the existing schema to add bits that are missing.

A quick glance suggests a lot of what we need is in the existing instrumentation except for the A/B test.

Additional questions/comments raised during today's Stand Up meeting:

• Can we make our instrument, if we have to make one, compatible with the existing one

The following are my unedited notes from today's investigation. I'll be tidying them up and hopefully making an easy-to-consume tl;dr tomorrow.

• [[SPIKE] Instrument language button](https://phabricator.wikimedia.org/T268504)
  • This is actually two tasks: one for any changes to the existing instrument and one for setting up the A/B test
  • Approach it in two parts: 1) instrument; and 2) A/B test
  • Instrument:
    • clicks to the new button
      • The UniversalLanguageSelector instrument already logs a settings-open event with the context field set to a unique token representing the location of the UI element the user interacted with (see the notes for the context property)
    • clicks to languages from ULS
      • This is already done in the UniversalLanguageSelector instrument
    • (nice to have) number of new users accessing language switching functionality
      • The UniversalLanguageSelector instrument logs the user ID for the token property, if the user is logged in (see this line in log()). In principle, we should be able to fetch the user's registration date from the replicas during analysis.

If we want to use the bucketised edit count for the user as a proxy for newness, then we'll have to update the instrument in the same way that we're updating the SearchSatisfaction schema in https://phabricator.wikimedia.org/T272991

• (nice to have) initial and final language
  • This is possible with a trivial change to the UniversalLanguageSelector instrument.

Currently, the instrument logs the final language (see interfaceLanguageChange(), noting that the logParams.interfaceLanguage property overrides that set in one sent in log()).

I propose we rename the property set in interfaceLanguageChange() from logParams.interfaceLanguage to logParams.finalLanguage, update the UniversalLanguageSelector schema, and bump the version accordingly.

• • (nice to have) time on page prior to switching to a different language
• A/B test:
  • Allow logged-in (logged-out users nice to have) to be bucketed into either the new treatment or the control
    • It's simple enough to vary treatments based on the user's ID
    • Since we're looking at language switching, we should use the user's "central ID", i.e.

<?php

$lookup = \CentralIdLookup::factoryNonLocal();
$id = null;
if ( $lookup ) {
  $id = $lookup->centralIdFromLocalUser( $user );
}
// The central ID lookup failed?
if ( !$id ) {
  $id = $user->getId();
}

See this Slack thread for context.

• As usual, expanding the initial cohort to all users introduces complexity:
  • We'll need to deliver the page in the Maybe State – the list of languages and the ULS button are both hidden – until we know which treatment we should show to the user. Since all JavaScript is delivered, parsed, and executed asynchronously, the user will observe either treatment being enabled. The effect will be acutely obvious when the list of 300+ languages suddenly appears in the sidebar. However, I should note that the sidebar isn't expanded by default for anonymous users

AC: Allow logged-in (logged-out users nice to have) to be bucketed into either the new treatment or the control

The ULS itself doesn't care where the UI element is so we won't have to make changes to the ULS codebase.

Logged-in users only

If we restrict the initial cohort to logged-in users, then we can vary the treatment on the server based on their ID since it identifies a user for longer than we're running the experiment.

The bucketing code would look something like:

$isExperimentEnabled = (bool)$config->get( Constants::LANGUAGE_SWITCHER_EXPERIMENT_ENABLED );
$treatment = Constants::LANGUAGE_SWITCHER_EXPERIMENT_TREATMENT_CONTROL;

if ( $isExperimentEnabled && $user->isRegistered() ) {
  $lookup = \CentralIdLookup::factoryNonLocal();
  $id = null;
  if ( $lookup ) {
    $id = $lookup->centralIdFromLocalUser( $user );
  }

  // The central ID lookup failed?
  if ( !$id ) {
    $id = $user->getId();
  }

  $treatment = $id % 2 === 0
    ? Constants::LANGUAGE_SWITCHER_EXPERIMENT_TREATMENT_CONTROL
    : Constants::LANGUAGE_SWITCHER_EXPERIMENT_TREATMENT_A
    ;
}

Logged-in and -out users

We can't bucket logged-out users on the server, which introduces a number of complications:

1. We'll have to deliver the page in an "undefined" state until we can make a determination. Since all JavaScript is loaded asynchronously so as not to block the initial rendering of the page, there'll be a small delay until either treatment is available to the user, i.e. there's a small risk that a user who has previously expanded the sidebar, might see a list of 300+ languages suddenly appearing – I've judged the risk as small because the sidebar is collapsed by default for anonymous users with fresh sessions (and sessions are highly likely to be refreshed after seven days).

2. Following on from the above, the ULS have to be changed to handle the UI element not being present when it loads.

At the time of writing, the ULS binds click event handlers to specific UI elements. If we made the ULS rely on those click events bubbling to the body element, then it would be agnostic of when the UI element was added to the page, i.e.

// L382
$( document.body ).on(
  'click',
  '.uls-trigger',
  clickHandler // Defined further up in initInterface()
);

3. We'd bucket users based on their session ID (a per-origin, client-side-only identifier stored in Local Storage). If the user is logged-in, then they're likely to receive different treatments on different Wikipedias.

Instrumentation: Clicks to new button

At the time of writing, when the user clicks on the UI element, then the UniversalLanguageSelector instrument logs a settings-open event with the context property set to one of personal, interlanguage, menu, preferences, representing the location of the UI element.

Those values for the context property are hardcoded in parts of initInterface() in ext.uls.interface.js so we could:

• Just add another value, which is certainly the path of least resistance. However, it would increase the complexity of the function, couple the Vector and UniversalLanguageSwitcher codebases, and would be another patch to revert when all of this done; or
• Change the ULS to accept a hint for the context parameter from the UI element

<button class="uls-trigger" data-uls-source="article-header">...</button>

// L382
$ulsTrigger.on(
  'click',
  { source: $ulsTrigger.data( 'uls-source' ) }
  clickHandler // Defined further up in initInterface()
);

thereby keeping the Vector skin and UniversalLanguageSwitcher extension decoupled and allowing us to simplify the latter's codebase slightly.

Instrumentation: Number of new users accessing language switching functionality

The UniversalLanguageSelector sets the event's token property to the user's name if the user is logged in or the session ID if the user is logged out (see log() in ext.uls.eventlogger.js). In principle, we should be able to fetch the user's registration date from the replicas by joining on event.universallanguageselector.event.token = user.user_name.

Instrumentation: Initial and final language

At the time of writing, the UniversalLanguageSelector instrument sets the interfaceLanguage property to the final language for language-change events. However, for all other events, the interfaceLanguage property is set to the initial language.

We could infer the initial language from the domain of the intake URL, which is logged as the event.meta.domain field.

I'm by no means a domain expert but the above seems fragile, especially when logging both the initial and final languages will require two trivial changes:

1. Add the selectedLanguage property to the UniversalLanguageSelector@latest schema in the schemas-event-secondary repo, i.e.

properties:
  event:
    properties:
      selectedLanguage:
        type: string
        description: The language code of the selected interface language.

2. Update interfaceLanguageChange() in ext.uls.eventlogger.js to set the selectedLanguage property, i.e.

function interfaceLanguageChange( language ) {
  log( {
    action: 'language-change',
    context: 'interface',
    selectedLanguage: language
  } );
}

Instrumentation: Time on page prior to switching to a different language

By "time on page" do we mean:

1. Time the page was open;
2. Time the page was visible;
3. Time active on the page; or
4. Something else?

I ask because #1, #2, #3 are all possible but are increasingly complex to implement. All three require us to choose some origin to start measuring from, e.g. DOM interactive, first paint, First Contentful Paint, or Largest Contentful Paint, the first two of which we used in the ReaderDepth instrument. Stephane Bisson took a different tack with the InukaPageView instrument and chose the time the instrument began executing because this can occur before DOM interactive for large pages (or smaller pages on slow/unreliable networks) in modern browsers (see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/551259 for detailed discussion). In all of the following, I've opted to follow Stephane's example.

1. Time the page was open (XS)

var start = mw.now();

function getTimeOnPage() {
  return mw.now() - start;
}

function interfaceLanguageChange( language ) {
  log( {
    /* ... */
    timeOnPage: getTimeOnPage()
  } );
}

2. Time the page was visible (S-M)

While this is more complex than #1, it's something that we've implemented before in the ReadingDepth instrument and Stephane Bisson implemented for the InukaPageView instrument.

// Stephane Bisson's implementation in WikimediaEvents/ext.wikimediaEvents/InukaPageView.js (see
// https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/551259), which I believe
// was based on Readers Web's implementation in the ReadingDepth instrument.

var start = mw.now(),
  hiddenAt = null, timeHidden = 0;

function onHide() {
  if ( !hiddenAt ) {
    hiddenAt = mw.now();
  }
}
function onShow() {
  if ( hiddenAt ) {
    timeHidden += mw.now() - hiddenAt;
    hiddenAt = null;
  }
}

if ( document.hidden ) {
  onHide();
}

document.addEventListener( 'visibilitychange', () => {
  if ( document.hidden ) {
    onHide();
  } else {
    onShow();
  }
} );

function getTimeSinceLoaded() {
  return mw.now() - timePaused;
}

// ...

Instrument: Clicks to language list in control group

Update: 2021/02/09

@LGoto, @ovasileva, @nshahquinn-wmf, @MNeisler, @Nikerabbit, @Jdlrobson, and I met yesterday to discuss the proposed changes to the ULS and its instrumentation. We agreed that tracking clicks on items in the language list should be added to the instrument.

This will require a small change to the ULS instrumentation in order to set the context property of the language-change UniversalLanguageSelector events:

function interfaceLanguageChange( language, source ) {
  log( {
    /* ... */
    language: language,
    context: source || 'interface'
  } );
}

/* ... */

document.body.addEventListener( 'click', ( event ) => {
  var el = event.target;
  if ( !el.classList.contains( 'interlanguage-link-target' ) ) {
    return;
  }

  mw.hook( 'mw.uls.interface.language.change' ).fire(
    el.attributes.getNamedItem( 'hreflang' ),
    'language-list'
  );
} );

Add a provenance parameter to the URL of each item, e.g. wprov=dipw1, will allow us to know how many pageviews are a result of users clicking on those links, the initial domain and the final domain:

select
    uri_host as final_domain,
    parse_url( referer, 'HOST' ) as initial_domain
from
    wmf.webrequest
where
    x_analytics_map['wprov'] = 'dipw1'
;

Adding the provenance parameter to the links can be done on both the server- and client-side (see the sample code below). However, if the initial cohort includes logged-out users, then we must do so on the client-side as since we can't bucket logged-out users on the server.

On the client-side:

Array.from( document.querySelectorAll(  '.interlanguage-link-target' ) )
  .forEach( el => {
    el.href = el.href + ( el.href.strpos( '?' ) === -1 ? '?' : '' ) + 'wprov=dipw1'
  } )
);

On the server-side:

namespace Vector;

class Hooks {
  public static function onSkinTemplateGetLanguageLink( array &languageLink ) {
    $featureManager = self::getFeatureManager();

    if ( $featureManager->isEnabled( Constants:: LANGUAGE_SWITCHER_EXPERIMENT_ENABLED ) ) {
      $languageLink[ 'href' ] = wfAppendQuery(
        $languageLink[ 'href' ],
        [
          'wprov' => 'dipw1',
        ]
      );
    }
  }
}

In T268504#6804647, @LGoto wrote:

Questions from Sam: Is it worth thinking about the affect on the a/b test? Is there a design treatment that is an ambiguous state we send to everyone?

These questions were regarding T268504#6796185.

@alexhollender and I discussed this on Thursday and agreed that hiding the language list in the sidebar and hiding the language switcher until the user is bucketed is the simplest solution from both a design and engineering perspective.

Sorry, I ran out of time to look at this today. Will plan on reviewing tomorrow

Thank you @phuedx for the great detailed notes!

In T268504#6810416, @phuedx wrote:

@alexhollender and I discussed this on Thursday and agreed that hiding the language list in the sidebar and hiding the language switcher until the user is bucketed is the simplest solution from both a design and engineering perspective.

Could you elaborate on how you envision this working for no-js users just so I'm on the same page? Were you envisioning the server would render the language list in the sidebar (the status quo), the language button in the body header but with an empty .vector-menu-content-list list, and we would use the client-js class to hide both of these before the JS executes? If the user is bucketed in the body header group on the client then ULS would use the list of languages in the sidebar for its data?

In T268504#6796185, @phuedx wrote:

AC: Allow logged-in (logged-out users nice to have) to be bucketed into either the new treatment or the control

Logged-in and -out users

We can't bucket logged-out users on the server, which introduces a number of complications:

1. We'll have to deliver the page in an "undefined" state until we can make a determination. Since all JavaScript is loaded asynchronously so as not to block the initial rendering of the page, there'll be a small delay until either treatment is available to the user, i.e. there's a small risk that a user who has previously expanded the sidebar, might see a list of 300+ languages suddenly appearing – I've judged the risk as small because the sidebar is collapsed by default for anonymous users with fresh sessions (and sessions are highly likely to be refreshed after seven days).

If we do end up bucketing anon users though, would we use the same client-side code to bucket logged-in users? (where I think the sidebar is open by default, and I would think 300+ languages suddenly appearing in the sidebar would be more noticeable)?

In T268504#6817487, @nray wrote:

In T268504#6810416, @phuedx wrote:

@alexhollender and I discussed this on Thursday and agreed that hiding the language list in the sidebar and hiding the language switcher until the user is bucketed is the simplest solution from both a design and engineering perspective.

Could you elaborate on how you envision this working for no-js users just so I'm on the same page? Were you envisioning the server would render the language list in the sidebar (the status quo), the language button in the body header but with an empty .vector-menu-content-list list, and we would use the client-js class to hide both of these before the JS executes? If the user is bucketed in the body header group on the client then ULS would use the list of languages in the sidebar for its data?

Yes. This is how I'd envisioned it. I imagine that it'd look like the following:

.mw-portlet-lang, .vector-language-switcher {
  /* Initial state for all users in the A/B test. */
  display: none;
}

.no-js .mw-portlet-lang {
  /* If the A/B test can't run, then the user should see the control treatment. */
  display: block;
}

Obviously, when it comes time to deploy our language switcher treatment (if it performs well), then we'd want to render the list on the server to enable it for users without JS enabled.

In T268504#6817487, @nray wrote:

In T268504#6796185, @phuedx wrote:

AC: Allow logged-in (logged-out users nice to have) to be bucketed into either the new treatment or the control

Logged-in and -out users

We can't bucket logged-out users on the server, which introduces a number of complications:

1. We'll have to deliver the page in an "undefined" state until we can make a determination. Since all JavaScript is loaded asynchronously so as not to block the initial rendering of the page, there'll be a small delay until either treatment is available to the user, i.e. there's a small risk that a user who has previously expanded the sidebar, might see a list of 300+ languages suddenly appearing – I've judged the risk as small because the sidebar is collapsed by default for anonymous users with fresh sessions (and sessions are highly likely to be refreshed after seven days).

If we do end up bucketing anon users though, would we use the same client-side code to bucket logged-in users? (where I think the sidebar is open by default, and I would think 300+ languages suddenly appearing in the sidebar would be more noticeable)?

👍

@phuedx Thank you. The rest of your posts made sense to me and were well articulated. Please consider this a "+2" from me 😀

Thanks @phuedx for looking into this and your notes. Overall the proposed instrumentation changes and AB test setup plan look good to me. I just have a few follow-up notes and questions:

Instrumentation: Number of new users accessing language switching functionality
The UniversalLanguageSelector sets the event's token property to the user's name if the user is logged in or the session ID if the user is logged out (see log() in ext.uls.eventlogger.js). In principle, we should be able to fetch the user's registration date from the replicas by joining on event.universallanguageselector.event.token = user.user_name.

I clarified with @ovasileva today that we will be using the user's registration date as the easiest way to identify users that have not likely used the language switcher before. I did a quick check and confirmed this will be feasible using the MariaDB replicas or mediawiki_user_history data set.

Instrumentation: Initial and final language
At the time of writing, the UniversalLanguageSelector instrument sets the interfaceLanguage property to the final language for language-change events. However, for all other events, the interfaceLanguage property is set to the initial language.
We could infer the initial language from the domain of the intake URL, which is logged as the event.meta.domain field.

Could the initial language also be inferred from the event.ContentLanguage field for language-change events? Either way, I agree with the addition of the selectedLanguageproperty to clarify the final language selection for events.

Additional instrumentation recommendations:

Session ID. We'll need a session id to calculate clicks to the language button per session and track other per session activity. It looks like the ULS schema does not currently have one.
editCountBucket (Nice to have). Not necessary but would be nice to have in case we want to review any trends by user experience.
isAnon (Nice to have). The only way to currently decipher logged-in vs anon users with the current schema is the event.token field, which records the user name for logged in users or session token for anonymous users. It's feasible to run a query against this field to find values that match the session token pattern or by joining on replicas users table but this might result in some error and make the analysis more complicated. If possible, I'd recommend adding a boolean type isAnon field so we can easily distinguish logged-in users included in the AB test. Note: We will still need to keep the event.token field so we can identify the user's registration date by joining with the replicas user table.

In T268504#6821401, @MNeisler wrote:

Instrumentation: Initial and final language
At the time of writing, the UniversalLanguageSelector instrument sets the interfaceLanguage property to the final language for language-change events. However, for all other events, the interfaceLanguage property is set to the initial language.
We could infer the initial language from the domain of the intake URL, which is logged as the event.meta.domain field.

Could the initial language also be inferred from the event.ContentLanguage field for language-change events? Either way, I agree with the addition of the selectedLanguageproperty to clarify the final language selection for events.

I'm not sure how I missed event.ContentLanguage!

Additional instrumentation recommendations:

Session ID. We'll need a session id to calculate clicks to the language button per session and track other per session activity. It looks like the ULS schema does not currently have one.
editCountBucket (Nice to have). Not necessary but would be nice to have in case we want to review any trends by user experience.
isAnon (Nice to have). The only way to currently decipher logged-in vs anon users with the current schema is the event.token field, which records the user name for logged in users or session token for anonymous users. It's feasible to run a query against this field to find values that match the session token pattern or by joining on replicas users table but this might result in some error and make the analysis more complicated. If possible, I'd recommend adding a boolean type isAnon field so we can easily distinguish logged-in users included in the AB test. Note: We will still need to keep the event.token field so we can identify the user's registration date by joining with the replicas user table.

Great points. We'll need to add the corresponding fields in the same as outlined in T268504#6797071. I'd estimate this as an S. It's not an XS as there are a couple of moving parts and we'll need to bring in the getUserEditCountBucket function from either the Popups and QuickSurveys codebase.

Update: 2021/02/15

Once the UniversalLanguageSelector instrument is moved into MediaWiki-extensions-WikimediaEvents, we'll have access to mw.wikimediaEvents.getUserEditCountBucket( editCount: number ). With this in mind, I'm updating my estimate to an XS.

I checked with @Amire80, and he does not have any dashboards or automated reports based on the UniversalLanguageSelector data stream. So even if you accidentally break backwards compatibility, there shouldn't be anything downstream that will break 😊

Thanks, @nshahquinn-wmf!