Per discussions with @ovasileva, we are aiming to complete this in time to include in the Q3 Tuning Sessions. Data will be calculated using the new desktop opt-in and opt-out schema, which is currently in development T261842.

If this schema is not ready, I will use the same methodology used in T260149 for right now and then update later once the new schema is ready.

Pending availability of data in the new desktop preference schema, I reviewed data in the mediawiki user properties table to determine the percent of logged-in users at all of the pliot wikis (looking at both registered users and active users) that have their vector skin preference changed to legacy as of 13 April 2021. See summary below:

Important Note: Providing these for reference now; however, I'm not confident the data below is accurately capturing only user-initiated changes. The user properties table is set up to record any current user preferences when they differ from their default value but it's unclear how the table handles the VectorSkinVersion property as it behaves a little differently and the default state can vary based on the current config state.

The new schema should provide more accurate data and I'll update these numbers using that schema once available.

Opt Out Rate for Registered Users

Opt-Out Rate for Active Users (Defined as users that had 5 or more content edits overall in the last year from March 2020 to March 2021)

For what it's worth, I would recommend not reporting the opt-out rate for registered users at all. Since most of those users are long vanished, it's not really a fair metric. You've already dealt with that by providing a better metric (opt-out rate for active users), but in this case, it's so much better that I think it should just stand on its own.

Also, I suggest filtering out users that have a skin other than Vector in the first place. I know those users are pretty rare overall (about 3% of active users according to this analysis), but it's possible that they're a substantially larger proportion at one of the wikis you're working with. For example, I think I vaguely remember that Timeless was particularly popular at French Wiktionary.

@ovasileva

Here are the updated numbers and analysis of the desktop improvements opt-out rate as well as superset dashboard that can be used to track the number of opt-outs each day. Full report with additional details coming soon but let me know if you have any questions or issues accessing the dashaboard.

Data comes from the new pref_diff schema along with data contained in the user_properties table.

Opt Out Rate for Registered Users [1, 2]

The average opt-out rate among all registered users on the test wikis is 4.8%.

[1] Please note that registered users likely include a large proportion of users that are not active and may not have visited the site while the desktop improvement features have been available. Finding a way to track site visits would help provide a more accurate opt-out rate as we could restrict to only users that viewed the site and saw the new desktop vector skin.
[2} I excluded users that were not using the vector skin from this analysis.

Opt-Out Rate for Active Editors [2, 4]

The average opt-out rate among active editors in June 2021 on the test wikis was 24%. There is a higher opt-out rate for the second group of test wikis compared to the first; however, the opt-out rates among active editors for each target wiki are still below 40% except for German Wikivoyage which only had 2 active editors in June 2021- with 1 having a user preference set to legacy.

[3] I reviewed the opt-out rate among active editors defined as the number of registered users who made at least 5 content edits across all projects in the given month (See Research:Active_editor). I looked at active editors in June 2021 for this analysis as the new vector skin was available to all the test wikis as default for the whole month and we had a full month of complete data collected in the new schema.
[4] I excluded users that were not using the vector skin from this analysis.

Daily Trends Analysis

Superset Dashboard I've added a chart to the Desktop Improvements Dashboard to track the daily number of users that are opting out of the new vector skin (switching from initial_state = 'vector2') on the test wikis. There is a filter to break down the number of opt-outs by test wiki and selected desktop skin type (legacy, timeless, etc.).

Summary of Trends and Other Insights:
The results below are based on data collected from 18 May 2021 through 15 July 2021 on the test wikis.

• A total of 1514 users across all the early adopter wikis have opted out of the new vector at least once during the reviewed time period.
• About 2% of those users have opted back into the new vector skin after opting out.
• 11% of users that opt'd out of the modern vector, changed their preferences multiple times during the reviewed timeframe.
• Since 17 May 2021, there has been a daily average of about 29.6 distinct users across all test wikis that have opted out of the new vector skin.
• The highest number of daily opt-outs logged so far was 56 distinct users on 24 June 2021, a couple of days following the deployment of the language switcher AB test. The number of users opting out quickly declined after that peak with closer to an average of 20 users opting out per day towards mid-July.
• A little more than half of all users (54.6%) switched to the legacy vector at least once when they opted out. Other users switched to minerva, modern, monobook, or timeless.

Data Caveats:

• Events reflected in chart are user-initiated switches from the new vector skin to another desktop skin type.
• Instrumentation does not provide information on the default skin that was first presented to the user. The new desktop skin is currently available as default on all test wikis; however, we do not have data on user preference changes that occured prior to 17 May 2021.

Chart of daily opt-outs across all test wikis from 18 May 2021 through 15 July 2021

The chart is also available on the Desktop Improvements Superset dashboard and can currently be filtered by target wiki and selected desktop skin state. Note: The chart may take a little bit to load; however, if you are unable to access it, it's likely because you do not currently have access to the analytics-privatedata-access group. You can request access through Phabricator by submitting the following form. You do not need to complete the SSH key requirement. Refer T286746 to as an example.

Hi @ovasileva - Just checking in on this task. Any follow-up needed here or can this task be resolved?