@Edtadros Would you be able to put some brain power to this to determine if there are specific flows within the context of "Readers Growth" we should be specifically hitting for this potential change.

Ultimately it's removing the m., which should work ok but might require specific mobile browser testing to ensure features still work as designed. I'll let you and Elena say whether that's true or not.

Primary tracking task for any work to be done should be created here on the primary ticket: https://phabricator.wikimedia.org/T403638#11159914

Here are highlights of the conversation that happened on Slack, complemented with my takeaways.

Client error dashboard

• @Etonkovidova points out the error_context.skin field, which can have minerva as a value for mobile. She confirms that assuming it won't go away, it should be enough for her purposes
• @Jdlrobson-WMF states that:
  • relying on Minerva will also give you desktop Minerva users which account for less than 1% of logged in users. For example on desktop Minerva more menu links are present so you would likely see more clicks there
  • logged-out users always get Minerva on mobile and Vector on desktop, unless they explicitly click on the Desktop link at the bottom of a Wiki page
  • there are lots of people on mobile devices using Vector 2022
  • we can use HTTP user agents to filter desktop browsers

Takeaways

It sounds like the mobile domain removal has a relatively low risk, provided that the skin field is kept.
However, relying on Minerva can:

• add a < 1% noise of logged-in desktop users
• ignore mobile users on Vector 2022

Instrumentation & analytics

• @jwang's feedback:
  • the distinction between mobile and desktop is important
  • a field telling whether MobileFrontend is running would enable filtering out the edge cases above
  • HTTP user agent analysis isn't ideal, as the value is often null or requires manual mapping of ambiguous browser families to mobile or desktop

Since Minerva alone isn't a reliable indicator for mobile users, she sees 2 directions:

• reduce noise by excluding users who didn't actually see the control or treatment version they were assigned to
• bucket mobile or desktop users at the beginning of the funnel

Takeaways

• If we check whether MobileFrontend is running within the team's ReaderExperiments MediaWiki extension, we will effectively restrict the target audience. In other words, we’ll be able to gate our features to mobile users only, which can be similar to bucketing at the beginning of the funnel. This seems to be the most reliable way to conduct A/B tests. See also T404233: Image Browsing: Gate the feature properly
• agent.client_platform_family coupled with mediawiki.skin from web_base or web_ui_actions schemas should be enough to capture mobile users, despite a tiny amount of exotic cases, see this Slack message