As far as I can tell, "not-supported" has always been a statement to reduce cognitive load on developers when developing new features and not something that has much meaning outside of that scope. We deploy it and thousands of ppl use these skins, so it is supported in practice. If it breaks we will hear about it, so there is some LoS there. And most importantly, this wasn't a new feature, it was the modification of an existing, very stable codepath.

Should that hold the train ? Hard to say. Here it was my understanding the website was usable for those affected and the issue did not present to anonymous users, nor the majority of WMF users. In my opinion that is not a train blocker and can easily be fixed with a scap deploy within a day or two. But it might not be a good idea to set such a subjective line as it might require too much interpretation and knowledge from the release train conductor.

"should we should likely add some basic tests to all the non-default skins e.g. UI Regression tests, some basic integration smoke tests." Probably, yes, these days we should.

Having said that... I think "reduce cognitive load on developers when developing new features" is a stupid justification. Developers SHOULD develop for multiple skins and they should use 3G download speeds to do so, using either Firefox or Safari as their primary browser. Because there is just no way you are ever going to write durable code by just using the latest Chrome on FTTH with a single skin. Double when modifying existing code.

The status quo as I understand it is that Vector is the prime skin and the others ones are support on a best effort basis (we would fix things when reported but we don't actively verify we support all skins). If some feature is broken in CologneBlue it would only affect users of that specific skin (which is probably a very tiny user base) and only those hitting the broken feature (another thin slice of the thin user base on that skin). So I don't thin it is worth holding the train for that given the issue will surely be reported and can be hot fixed within a few days. I claim it is good enough.

For a bit of historical perspective: in the old day we had a yellowish skin and potentially had some others at the time I cant remember, CologneBlue probably come from that time. Eventually in 2004 Gabriel Wicke adapted a skin from Plone which got us Monobook which was definitely an improvement and I think some of the old guards editors hated it so we kept the old skin cause some really preferred the yellow background. As the foundation started employing people we had a usability initiative back in 2009 (iirc) which resulted in the Vector skin we are still using nowadays (although lot of things have changed behind the hood, the general aspect is more or less the same).

So essentially we did yellow skin → monobook → vector but kept carrying the old skins over and never made the call to drop those legacy skins. In the ideal world an audit should be made to find out which skins are still actively used (some users might have an old skin set in their preference but no more be actively browsing the site), reach out to our users to explain the challenges faced supporting so many poorly maintained skins and hopefully reach out a consensus backed by our user that it is finally time to remove old skins we don't have the mean to properly support.

Else we fallback to the status quo, legacy skins issues are fixed on a best effort basis and if something break it is not holding the train.

In T300169#7655746, @hashar wrote:

In the ideal world an audit should be made to find out which skins are still actively used (some users might have an old skin set in their preference but no more be actively browsing the site)

There's some relevant stats about that at T180860#6241657 (a year or two old)