(Let me know if you have any questions on any of the following)

So I've been investigating this and learned two things
Q1) Which subset of users, based on their preferences, were accidentally opted into the Vector 22 skin during the skin splitting
A1)
All users using old Vector were opted into the new Vector on pilot wikis.
The reason for this is that user preferences when not different from the defaults, will change any time default is changed. This is what the code saids, and my tests have verified.

I verified this most recently by setting my skin to Vector legacy in MediaWiki.org prior to making it the default skin on Thursday 10th. I was instantly thrown into the new Vector skin.

For the non-pilot wiki, nobody was impacted as the default skin did not change.

Bonus question 2) Why were so many people seeing it for the first time?
Bonus answer:

It seems that each time we've been adding a new pilot wiki, we've been making the assumption that existing logged in users would automatically have been opted in. However, this doesn't seem to be the case.

We broke the “skin for existing accounts” in a patch that was deployed to French on 4th May 2020 https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/593063. I believe the implication of this change, was that if anybody saved a preference (e.g. gadget or visit to Special:Preference) they got stuck into whatever skin was default at the time. Prior to that, this setting seems to have been working, but the instrumentation around opt ins was not.

On July 20th we thought we were converting a bunch of logged in users from Vector to new Vector (T254227) on French Wikipedia by adding it as a pilot wiki. However we were actually converting users who had not used Special:Preferences between May 4th and July 20th.

TLDR: When enabling pilot wikis, we've only opted in logged-in users, who had not saved their preferences since 4th May 2020. Anyone who had changed any preference (including unrelated) between that period was also making sure they got the legacy Vector skin.

This likely explains why so many people were shocked at seeing new Vector for the first time.

I will investigate this further in T300919.

Bonus question 3) How could we have prevented this?

1. The beta cluster will be updated so that it mirrors pilot wikis closer. This issue would likely have come up during QA, had we done that before.
2. When deploying we first enabled on Basque Wikipedia. It's clear we should have done this on a larger wiki like French or PT as the opt out/in graph impact was minimal for Basque Wikipedia.
3. I've setup alerting on high volume of opt ins/outs however this would likely alert us to the problem too late.
4. A different approach would have relied on a database script. Too late for that now but worth considering in future projects where we need to do something similar. I'll follow up with this in

T299104

5. Clearly our integration tests didn't work here. As part of completing the migration we need to make sure they cover scenarios for enabling pilot wikis. (To be continued in T300919)

Investigation (make a wiki opt in)

If another engineer wants to confirm this, I did the following to confirm this:

Made my wiki opt in only

$wgVectorDefaultSkinVersion = '1';
$wgVectorSkinMigrationMode  = false;
$wgVectorDefaultSkinVersionForExistingAccounts = '1';
$wgVectorDefaultSkinVersionForNewAccounts = '1';
$wgDefaultSkin = "vector";

2. Saved an unrelated preference relating to edit tab. This commits a change to the database and sets my skin to legacy Vector.

3. If I now make the wiki, new Vector, and refresh the page like so:

$wgVectorDefaultSkinVersion = '2';
$wgVectorDefaultSkinVersionForExistingAccounts = '2';

I am still in legacy Vector.

It seems the "default skin version for existing accounts" option only gets taken into account if no preference has been written to the database.
the UserOptionsManager includes/user/UserOptionsManager.

The fallback is seldom utilized, due to includes/user/DefaultOptionsLookup.php which calls UserGetDefaultOptions