According to https://grafana.wikimedia.org/dashboard/db/mobile-2g?orgId=1 the fully loaded time for the beta cluster article of Barack Obama rose by about 0.5s on the 1st August.
Around the same time CSS jumped by about 0.7kb
My guess is someone enabled a feature and unintentionally loaded it on all mobile pages.
We should look to get to the bottom of this before the next train.
Adding to sprint as unplanned as we don't want to introduce a negative performance change to production. Let's understand what happened here.
Any notable SWATs that day? Any changes that might bloat our CSS?
Well.. it's on production now...
Lucky for us this time this didn't translate into an increase in fully loaded on stable, however the same CSS spike can now be seen in production.
I'm going to investigate this today to see what caused it while we can still debug.
Only config change that matches up with the dates is https://gerrit.wikimedia.org/r/366867 - to enable OOUI editing forms. My read of the code however is that the css is only loaded in edit mode.
The stylesheet loaded by anons is:
https://en.m.wikipedia.org/w/load.php?debug=false&lang=en&modules=ext.cite.styles%7Cmediawiki.ui.button%2Cicon%7Cskins.minerva.base.reset%2Cstyles%7Cskins.minerva.content.styles%7Cskins.minerva.icons.images%7Cskins.minerva.print.styles%7Cskins.minerva.tablet.styles&only=styles&skin=minerva
Immediately purging the page I see an additional 1kb from ext.math.styles:
https://en.m.wikipedia.org/w/load.php?debug=false&lang=en&modules=ext.cite.styles%7Cext.math.styles%7Cmediawiki.ui.button%2Cicon%7Cskins.minerva.base.reset%2Cstyles%7Cskins.minerva.content.styles%7Cskins.minerva.icons.images%7Cskins.minerva.print.styles%7Cskins.minerva.tablet.styles&only=styles&skin=minerva
Given the lack of changes to mediawiki ui, this suggests the spike is either due to our own changes or Math/Cite extension. I'm not seeing any recent changes there - so I think we can either account for this small spike as being a result of our own changes or a glitch (where Math module is added unexpectedly).
Suggested action: Nothing for time being.
I agree - no action needed for now. On August 2nd the peak CSS size was 9.58 KiB, and now it's 9.49 KiB, although it was 9.41 KiB on August 1. We should look into creating a CSS time frames graph for easy comparison.
Thanks for the investigation and write up, @Jdlrobson and for the additional detail, @bmansurov!
I've added a new graph (of CSS size) to the bottom of the dashboard. Feel free to adjust as you see fit.
I've moved it next to "total bytes" and I added a line for CSS beta cluster (since this tells us when something is about to regress/improve).
I also removed CSS from those graphs since they now have a dedicated graph.
Calling this closed!