I don't know yet how this could happen, but I am thinking about it very hard. Assuming that there is no catastrophic bug in browser's parsing of CSS (this is very simple code… so many things would be broken if this was the case, not just checkboxes), my current theory is that there was a mistake during deployment, and for a while, production Wikimedia wikis were running OOUI's PHP code at version v0.26.3, but OOUI's CSS code at version v0.26.2. I need to check some logs to verify this.

Screenshots I've just taken of a test wiki where I intentionally introduced this bug (PHP code at v0.26.3, CSS code at v0.26.2):

However, I am confident that the Safari issue is a different bug. If this was a v0.26.3/v0.26.2 version mismatch, it would be broken in a different way (see screenshots above). And Safari used to have issues with our checkmarks (T89309). We worked around that a long time ago, but maybe we accidentally triggered this again.

As a side note, the opposite (PHP code at v0.26.2, CSS code at v0.26.3) also breaks things, but only in WikimediaUI theme (the default in Vector) and in a different way, see screenshots on T192754. I think people only saw that issue when forgetting to update their own test wikis, and it never affected production Wikimedia wikis.

I tried installing Opera 36 on a Windows XP virtual machine I have lying around and testing in it, but everything looks okay right now:

I think the old browser / old OS was a red herring, it was just a confusing coincidence that the first MonoBook user to notice the problem also runs those.

(Opera 36 is probably the best option if you're stuck on Windows XP. I don't think anyone else supported Windows XP for as long as they did.)

This looks a bit like a race condition between css files being added to the queue, where rules have otherwise the same specificity or something ?

This can't be a load order issue, because both of the rules which are supposed to show/hide the extra checkmark are loaded from a single URL (e.g. https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=...%7Coojs-ui-core.styles%7C...&only=styles&skin=monobook F17292473). It is extremely unlikely to be a browser bug because those are very simple CSS selectors with the same specificity that depend on the order within the file for correct rendering, .oo-ui-iconWidget.oo-ui-iconElement.oo-ui-iconElement-icon is overridden by .oo-ui-checkboxInputWidget .oo-ui-checkboxInputWidget-checkIcon.oo-ui-iconElement-icon which appears later. Browsers from IE 7 and later should handle this just fine (IE 6 did not support multiple class selectors).

I think what happened here was that there were several instances in the last few days where, for about 5 minutes each time, users would see pages with CSS from a different version of OOUI than the PHP code that generated them.

This happened because:

• When serving special pages, the edit form, etc., we instruct browsers to not cache them: e.g. index.php?action=edit has HTTP headers Cache-Control: private, s-maxage=0, max-age=0, must-revalidate.
• When serving CSS, we instruct browsers to cache them for 5 minutes: e.g. load.php?only=styles has HTTP headers Cache-Control: public, max-age=300, s-maxage=300 (300 seconds = 5 minutes).

This means that for about 5 minutes after the deployment of each new MediaWiki version, users will see pages generated by the new PHP code, but their browsers may still use the old cached CSS code. I think this is simply what happened here, but due to unrelated problems with the new MediaWiki version (T183969) it happened three times this week instead of once (when it was deployed, reverted, then deployed again).

The timeline of events (per wikitech: Server Admin Log):

• before 2018-04-19
  • English Wikipedia is running OOUI v0.26.2 (MediaWiki 1.31.0-wmf.29)
• 2018-04-19 20:32 thcipriani@tin: rebuilt and synchronized wikiversions files: All wikis to 1.31.0-wmf.30
  • English Wikipedia is now running OOUI v0.26.3. But for the next five minutes, users may still see CSS from OOUI v0.26.2. This is the first time the issue appeared.
  • Redrose64's comment: "They first appeared a few days ago, but vanished within minutes."
• 2018-04-19 22:30 thcipriani@tin: rebuilt and synchronized wikiversions files: group1 and group2 wikis back to 1.31.0-wmf.29
  • English Wikipedia is now running OOUI v0.26.2. But for the next five minutes, users may still see CSS from OOUI v0.26.3. This caused a different issue (checkboxes disappearing on Vector): T192754: Regression: OOUI checkboxes collapsing.
• 2018-04-23 20:40 thcipriani@tin: rebuilt and synchronized wikiversions files: All wikis to 1.31.0-wmf.30
  • English Wikipedia is now running OOUI v0.26.3 (again). But for the next five minutes, users may still see CSS from OOUI v0.26.2.
  • Redrose64's comment was written at 20:58, 23 April 2018.

We could have avoided this issue if we had anticipated it (by first deploying CSS code that worked with both old and new PHP code, waiting at least 5 minutes, then deploying new PHP and CSS code). We are pretty good at doing that when necessary (e.g. with article HTML, which is cached for up to 14 days), but when the worst possible issue is a minor visual problem that is going to disappear after 5 minutes, the general consensus seems to be that it's not worth the couple hours of engineering time to prevent it.

I don't think there is anything else left to do here, except maybe keep this in mind for future changes (not just OOUI, this affects all code we run). Nothing terrible happened this time, but we could potentially break the interface in worse ways if we forget about this caching behavior. FYI @Jdforrester-WMF @greg

(Also, big thanks to Redrose64 for including the screenshot. I noticed that it is showing the old design of the checkmark icon, which we changed in v0.26.3. Without that hint towards the mismatched versions I probably would not have thought of the reason for the bug.)

In T192951#4155329, @matmarex wrote:

(Opera 36 is probably the best option if you're stuck on Windows XP. I don't think anyone else supported Windows XP for as long as they did.)

Just as a side note: The current Firefox ESR version still runs on Windows XP, and will receive security updates until June 2018 (https://blog.mozilla.org/futurereleases/2017/10/04/firefox-support-for-windows-xp-and-vista/).

Thanks for the in-depth analysis, @matmarex. Definitely worth thinking about in future.