Page MenuHomePhabricator

Determine opt-out rate for desktop improvements feature on test wikis
Closed, ResolvedPublic

Description

Background

We would like to know the number of logged-in users that have opted out from the desktop improvements projects from the beginning of the deployments on the test wikis (target is less than 40%)

Acceptance criteria

  • Report on the percentage of logged-in users that have opted out from the new version of vector:
  • per test wiki
  • per user edit bucket
  • Add a chart reflecting this to the desktop improvements dashboard

Note: due to the issue of accidental opt-ins/opt-outs, we will have to only consider the percentage of opt-outs after the date of deployment on each wiki.

Event Timeline

MNeisler triaged this task as Medium priority.Aug 11 2020, 4:31 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

@ovasileva - Can you confirm the deployment dates for each of the test wikis?

@MNeisler - sure, here you go:

Deployment schedule

projectWed. July 22Tues. July 28Wed. Aug 5
euwiki (Basque Wikipedia)x
fawiki (Persian Wikipedia)x
frwiki (French Wikipedia)x
frwiktionary (French Wiktionary)x
hewiki (Hebrew Wikipedia)x
ptwikiversity (Portuguese Wikiversity)x

I created a draft dashboard in Superset to review the opt in and opt out rates for the new desktop improvement features with filters to review by test wiki and edit count bucket. The data in the dashboard comes from PrefUpdate and uses the event.value field to track if the user selected legacy or modern skin versions and only includes events that occurred after the deployment date for each wiki. Even with these changes, there appear to be several issues with how these events are recorded in PrefUpdate that are causing data to appear off from what is expected. For example, we're seeing a really large number of modern vector selections for users with 0 edits, which might be in part due to T260867. Also, the number of opt-in and opt-out numbers don't seem to match what is expected based on which skin was provided as default.

Instead of using PreUpdate, the team is instead looking into creating a new schema that will only track the opt-ins and opt-outs of desktop improvements in T261842. Once that task is complete, I'll recreate the dashboard and create a report on the desktop improvement feature opt-out rate.

ovasileva raised the priority of this task from Medium to High.Sep 8 2020, 1:40 PM

Currently looking into providing current vector skin version selections using the the mediawiki user_properties table pending new instrumentation to track opt-outs.

@ovasileva

Here's are the current nondefault user VectorSkinVersion preferences as of 12 October 2020. Data comes from the mediawiki user_properties table.

Some clarifying notes:

  • Data reflects the current state and does not account for users that may have opt'd in and opt'd out multiple times since deployment.
  • Data recorded in the user properties tables reflects all current nondefault user preferences. If a user has not made any changes to their vector skin version then there is no skin preference recorded.
  • Only accounts for logged-in users.
  • Based on the context that the new vector skin was deployed as the default setting for all logged-in users on the wikis below, each of the values below can be interpreted as follows: (I ran a check using my user profile to confirm):
    • Legacy: These are users that have currently opt'd out of the modern version.
    • Modern: This number does not reflect the total number of users that are using the modern skin; only the users that made changes to their default preferences (i.e. These are users that opt'd out and then opt'd back in to the modern version).
    • Unknown: There were a few VectorSkinVersion values set to 0 (instead of 1 [legacy] or 2[modern]). I need to further investigate what those values indicate.
    • Opt-Out Rate: Calculated by dividing the total number of users with legacy preference set by the total number of all registered users on the wiki. It might be useful to look at opt-out rate among just active users as well. I can look into if needed.

Overall

SkinNumber of Users
Legacy128242
Modern [1]98334
Unknown29

By Target Wiki

WikiSkinNumber of usersOpt-Out Rate[2]
EuWikiLegacy20671.79%
Modern1852
Total Registered Users115786
FaWikiLegacy214422.22%
Modern22046
Unknown2
Total Registered Users963101
FrWikiLegacy715351.83%
Modern49185
Unknown19
Total Registered Users3905482
FrWiktionaryLegacy46841.60%
Modern3899
Unknown6
Total Registered Users291868
HeWikiLegacy280344.09%
Modern20890
Unknown2
Total Registered Users685306
PtWikiversityLegacy4801.62%
Modern462
Total Registered Users29455

[1] These are users that opt'd out and then opt'd back into the modern version. (Note: This number does not reflect the total number of users that are using the modern skin; only the users that made changes to their default preferences).
[2] Calculated by dividing the total number of users with their vector version preference changed to 'legacy' by the total number of all registered users on the wiki.

See notebook for further details.

MNeisler lowered the priority of this task from High to Medium.Oct 13 2020, 4:56 PM
MNeisler moved this task from Doing to Next 2 weeks on the Product-Analytics (Kanban) board.

Per discussions with @ovasileva today, I'll also review the opt-out rate among active editors. Moving this to medium priority based on context of other analysis task items currently.

I reviewed the opt-out rate among active editors (users that had 5 or more content edits overall in the last year from September 2019 to September 2020).

Approach: The opt out rate was calculated by finding the percent of active editors for each wiki (obtained using data from mediawiki_history) that have VectorSkinVersion preference set to legacy in the user properties table.

Assumptions

  • Since the modern vector version was deployed as default to all of the test wikis in this analysis, it was assumed that any users with a non-default preference recorded as legacy have opt-d out.
  • There are a large number of active editors who do not have data for vector skin preference in the user properties table indicating that they are set to the default 'Modern' skin OR due to being deleted from the user_preference table. In the numbers represented below, all of these users were all added as 'Modern' skin users.

Active Editor Opt-Out Rate Across all Target Wikis

SkinNumber of Active EditorsOpt-Out Rate
legacy713512.9%
modern47991
unknown10

Active Editor Opt-Out Rate For Target Wiki

WikiSkinNumber of Active EditorsOpt-Out Rate
euwikilegacy405.08%
modern748
fawikilegacy187415.09%
modern10546
unknown1
frwikilegacy429912.13%
modern31134
unknown9
frwiktionarylegacy5211.66%
modern394
hewikilegacy86314.49%
modern5095
ptwikiversitylegacy78.64%
modern74

The opt out rates among active editors for each target wiki are still well below 40%. Persian Wikipedia (fawiki) currently has the highest opt out rate (15.1%) among active editors while Basque Wikipedia (euwiki) has the lowest (5.1%). See notebook for further details.
@ovasileva - Let me know if you have any questions!

ovasileva closed this task as Resolved.Dec 7 2020, 5:14 PM

All done here. Thanks @MNeisler! Will create another task to follow-up once T261842: Create schema to track users opting in/out of desktop improvements is completed