Page MenuHomePhabricator

[SPIKE] Investigate practicality of making sampling rate consistent across mobile talk page schemas
Closed, ResolvedPublic

Description

T301026 proposes various approaches for ensuring peoples' usage of the mobile talk page overlay and mobile "read as wiki page" view can be compared.

In order to make the comparison mentioned above, we'll need to ensure the various schema these two talk page views are instrumented with can be related.

A requirement for doing the above is making sure that the following four schemas are sampling events at the same rates and using the same methods :

  • VisualEditorFeatureUse
  • EditAttemptStep
  • MobileWebUIActions
  • Talk_Page_Edit

This task involves the work of investigating how the rates at which each of the four schema above sample events can be made the same.

Open Question(s)

  • 1. Is it feasible to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?
  • 2. If the answer to "1." is "yes," what work would be required to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?
    • Implementation will happen in T309260.
  • 3. See question in T303654#7959099.

Done

  • Answers to the ===Open Question(s) above are documented

Related Objects

Event Timeline

It's subtly distinct from sampling rate -- we also need to make sure we synchronize the sampling method. If MobileWebUIActions and EditAttemptStep were both sampling at 1/16 that'd still not be helpful if they were sampling a different 1/16.

It's subtly distinct from sampling rate -- we also need to make sure we synchronize the sampling method. If MobileWebUIActions and EditAttemptStep were both sampling at 1/16 that'd still not be helpful if they were sampling a different 1/16.

Great spot, @DLynch. I've updated the task description to incorporate what you noted above. Please edit the task description if any further adjustment is needed to accurately represent the nuanced you noted in T303654#7771288.

Conveniently: everything about whether or not to log in both desktop and mobile DiscussionTools (assuming nothing triggering oversampling is present), and also the UIActions schemas, goes through mw.eventLog.inSample. Because of how this works, using a global "pageload token" as the sampling ID, any events with the same sampling rate on the same pageload are going to be sampled the same.

*If* we synchronize sampling rate configs, everything should line up. There will be things you can do that will jog them out of alignment, mostly by causing editing stats oversampling. However, if Megan excludes editing sessions with is_oversample from the analysis, it should work. (I think. She might want to comment on whether that'll have any statistical effect.)

(To remind myself when I read this later: This doesn't apply to WikiEditor or desktop VisualEditor. They use the EditAttemptStep session ID as the sampling ID.)

However, if Megan excludes editing sessions with is_oversample from the analysis, it should work. (I think. She might want to comment on whether that'll have any statistical effect.)

@DLynch would it be accurate for me to think the next step on this task is for @MNeisler to confirm that A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas and B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

@DLynch @ppelberg See responses and follow-up questions below.

@DLynch would it be accurate for me to think the next step on this task is for @MNeisler to confirm that A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas and B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas

The plan described in T303654#7856027 sounds reasonable to me. To clarify, do we anticipate this synchronization to cause any changes to the UIActions schema's current sampling rates? If so, we should keep the Web team in the loop of any changes as it may impact any planning or in-progress analyses using that instrumentation.

FYI current UIActionSchema Sampling Rates (unless changed recently):

B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

The only issue I see with removing is_oversample events is that it will reduce the number of DiscussionTool sessions available to review within a certain timeframe (In previous analyses, we've included oversampled DiscussionTool events to help increase the sample size available for review). As a result, we may be need to collect data for a longer time period in order to obtain a representative sample, especially for analyses of smaller-sized partner wikis where there are likely to be fewer mobile talk page events. This may impact the timing for T298065 and T298062, which both have metrics that require us joining talk page views from the UIActions schemas to editAttemptStep.

(To remind myself when I read this later: This doesn't apply to WikiEditor or desktop VisualEditor. They use the EditAttemptStep session ID as the sampling ID)

Does this mean that the sampling rates configs WikiEditor or desktop VisualEditor will not be synchronized with the UIActions schemas? In other words, will we only be able to link desktop and mobile DiscussionTool edit events to UIActions schemas?

To clarify, do we anticipate this synchronization to cause any changes to the UIActions schema's current sampling rates?

I don't take a position on that. We'd need to set all their rates to the same value, but it's all the same to me which one is chosen as the base rate. I suspect that's a decision for you and Peter, based on how long it'd take to accumulate enough data to make decisions?

Does this mean that the sampling rates configs WikiEditor or desktop VisualEditor will not be synchronized with the UIActions schemas? In other words, will we only be able to link desktop and mobile DiscussionTool edit events to UIActions schemas?

Yes. Those are both sampled from a different value, and so will be sampled at the same rate but not in sync with the UIActions on that pageload. In the case of WikiEditor we outright can't sync them up without some fairly large rewrites to how the page session token is generated, because the init event happens server-side before that token even exists. (GrowthExperiments has done a little bit in this direction already, I believe, but that's entirely restricted to Special:Homepage and we'd need something more general purpose that'd feed into other existing schemas.)

We could rewrite VE's sampling to also go off of the page token, if we wanted, and it'd be pretty trivial. It might cause a minor difference in the meaning of sampled data, as currently every individual edit session has a chance to be sampled separately, and we'd switch that so that multiple sessions on a pageload would always be sampled together. (But I strongly suspect that most sessions wind up being on their own pageloads.)

MNeisler triaged this task as Medium priority.May 24 2022, 4:13 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Per what @DLynch, @MNeisler, and I talked about last week (18 May):

  • 1. Is it feasible to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?

Yes, it is feasible to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method.

  • 2. If the answer to "1." is "yes," what work would be required to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?

To make it so the schemas listed above sample events at the same rate, the Editing Team will need to:

  • A) Increase the EditingAttemptStep schema's sampling rate from 6.25% (1 in 16 events) to match the sampling rate of the MobileWebUIActions and DesktopWebUI schemas' sampling rates, both of which are currently set to sample 20% (1 in 5) events.
  • B) "Turn on" sampling within the MobileWebUIActions and DesktopWebUI schemas at the Editing Team's nine mobile partner Wikipedias (T294609): ja.wiki, ar.wiki, fr.wiki, ko,wiki, vi.wiki, he.wiki, bn.wiki, zh.wiki, and ht.wiki

Note: work to do A) and B) will happen in T309260.

@DLynch, today a question surfaced for you in the conversation @MNeisler and I were having about what you shared in T303654#7939647:

  • Would it be accurate for us to think that if we do NOT change "...how the page session token is generated..." or "...rewrite VE's sampling to also go off of the page token..." then we would still be able to calculate and compare the following?
    • The rates at which people who land on a desktop or mobile talk page and engage with it in some way? Where "engage" in this context means, clicking or tapping any of the following: 1) any Add topic link/button that appears on the page, 2) any Reply link/button that appears on the page, and any Subscribe link/button that appears on the page.

Note: if the answer to the question above is "yes" then I do not think we need to prioritize making additional changes to how page session tokens are generated or how VE's sampling logic works. Reason being: in the A/B test we have planned (T302358), we will be comparing the test and control groups based on how they each engage with DiscussionTool affordances (read: buttons/links to add a new discussion topic, respond to something someone else has said, or subscribe to an ongoing conversation).

@ppelberg Yes. None of those require a change to the sampling method, as they don't involve VE/WikiEditor. Those can all be logged via EditAttemptStep or the UIActions schemas.

@ppelberg Yes. None of those require a change to the sampling method, as they don't involve VE/WikiEditor. Those can all be logged via EditAttemptStep or the UIActions schemas.

Understood. Thank you, @DLynch.

The information you shared means ===Open question #3 is answered and with it, this task can be resolved.