Page MenuHomePhabricator

Determine opt-out rate for desktop improvements feature on test wikis pt 2
Open, HighPublic

Description

Background

In T260149: Determine opt-out rate for desktop improvements feature on test wikis we determined the overall opt-out/opt-in rates for desktop improvements so far. This task tracks looking at this data in more detail once the new schema T261842: Create schema to track users opting in/out of desktop improvements is completed.

Acceptance criteria

  • Report on the percentage of logged-in users that have opted out from the new version of vector:
  • per test wiki
  • per user edit bucket
  • Add a chart reflecting this to the desktop improvements dashboard

Event Timeline

Per discussions with @ovasileva, we are aiming to complete this in time to include in the Q3 Tuning Sessions. Data will be calculated using the new desktop opt-in and opt-out schema, which is currently in development T261842.

If this schema is not ready, I will use the same methodology used in T260149 for right now and then update later once the new schema is ready.

Pending availability of data in the new desktop preference schema, I reviewed data in the mediawiki user properties table to determine the percent of logged-in users at all of the pliot wikis (looking at both registered users and active users) that have their vector skin preference changed to legacy as of 13 April 2021. See summary below:

Important Note: Providing these for reference now; however, I'm not confident the data below is accurately capturing only user-initiated changes. The user properties table is set up to record any current user preferences when they differ from their default value but it's unclear how the table handles the VectorSkinVersion property as it behaves a little differently and the default state can vary based on the current config state.

The new schema should provide more accurate data and I'll update these numbers using that schema once available.

Opt Out Rate for Registered Users

wikinum_legacy_usersnum_registered_users% opt out rate
First Group of Wikis...
frwiktionary47423038361.56%
hewiki281828056913.50%
ptwikiversity482309881.56%
frwiki7223140685521.78%
euwiki20971223851.71%
fawiki2159210186532.12%
Second Group of Wikis...
ptwiki14530225692015.66%
kowiki463166761546.85%
trwiki10021312892467.78%
srwiki159872832285.64%
bnwiki4042532663612.38%
dewikivoyage2739425766.43%
vecwiki2052285907.18%

Opt-Out Rate for Active Users (Defined as users that had 5 or more content edits overall in the last year from March 2020 to March 2021)

wikinum_legacy_usersnum_active_users% opt out rate
First Group of Wikis...
frwiktionary6144313.77%
hewiki998631015.82%
ptwikiversity121348.96%
frwiki49303692413.35%
euwiki499355.24%
fawiki20751425814.55%
Second Group of Wikis...
ptwiki81461497754.39%
kowiki2819549651.29%
trwiki3669711251.59%
srwiki725203335.66%
bnwiki1361251254.18%
dewikivoyage7420037.00%
vecwiki133339.39%

For what it's worth, I would recommend not reporting the opt-out rate for registered users at all. Since most of those users are long vanished, it's not really a fair metric. You've already dealt with that by providing a better metric (opt-out rate for active users), but in this case, it's so much better that I think it should just stand on its own.

Also, I suggest filtering out users that have a skin other than Vector in the first place. I know those users are pretty rare overall (about 3% of active users according to this analysis), but it's possible that they're a substantially larger proportion at one of the wikis you're working with. For example, I think I vaguely remember that Timeless was particularly popular at French Wiktionary.

ovasileva raised the priority of this task from Medium to High.Mon, Jun 7, 7:23 AM