Page MenuHomePhabricator

[SPIKE] Instrument opt-in/opt-out mechanism for desktop improvements project
Closed, ResolvedPublicSpike

Description

Background

In T237561 we determined the way using which we would like to implement the opt-in/opt-out mechanism for the desktop improvements project. In this task, we would like to outline how we would measure the mechanism

Acceptance Criteria

Please outline the instrumentation required to answer the following questions:

  • What is the retention of the feature (i.e. how many users opt-out on test wikis over time, how many users opt-in on test wikis over time)
  • What is the average duration of opt-in status (so we can see which features might have more effects than others)
  • How often do users view the appearance tab in preferences? What percentage of these users continue to interact (whether opt-in or opt-out) with the desktop improvements option
  • How do these changes vary based on the number of edits a given user has??
  • note: we would also like to monitor what effect the change has on overall pageviews

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptJan 10 2020, 11:41 AM
ovasileva triaged this task as High priority.Jan 10 2020, 11:41 AM
Niedzielski added subscribers: pmiazga, phuedx.

Assigning to @phuedx (cc @pmiazga) to document his thoughts on this task.

Fortunately, all preference changes are already instrumented and that instrumentation is resilient to the addition/deletion of user preferences.

The WikimediaEvents extension is responsible for emitting a PrefUpdate event for every preference that the user changes.

  • What is the retention of the feature (i.e. how many users opt-out on test wikis over time, how many users opt-in on test wikis over time)
  • What is the average duration of opt-in status (so we can see which features might have more effects than others)

PrefUpdate events include timestamp, user ID, property name, and property value. Therefore, we can construct queries to determine:

  • Opt-in/out rate
  • Average/median/95 and 99th percentile duration of opt-in status

a) How often do users view the appearance tab in preferences? b) What percentage of these users continue to interact (whether opt-in or opt-out) with the desktop improvements option

a) To answer this question exactly would require adding new instrumentation to the Special:Preferences page.

b) We can determine the number of logged-in users that visited Special:Preferences in a given period by querying the wmf.webrequest Hive table; and we can determine the number of users that changed a preference in a given period by querying the event.PrefUpdate Hive table.

AIUI the answer to b is the lower bound answer to a.

  • How do these changes vary based on the number of edits a given user has??

Since PrefUpdate events include user ID, we can determine the number of edits that the user has made during analysis.

However, since the user's edit count increases over time, we might want to update the instrumentation to include the bucketed user edit count. My recommendation would be to implement the edit count bucketing code in MediaWiki Core so that we can remove the independent implementations in the QuickSurveys and Popups codebases. It would be prudent to run the change by Product Analytics before the fact and also to announce the change after the fact.

ovasileva closed this task as Resolved.Jan 17 2020, 12:58 PM

Created T243071: Update PrefUpdate schema to include edit count to track adding edit count buckets. We can pass on answering and creating new instrumentation for "How often do users view the appearance tab in preferences?". Resolving this task. Thanks @phuedx!