@Nirmos Seems like it would be worth monitoring, for sure. Do you know when you're planning for this migration to happen?

What we'd want to set up for this is pretty straightforward. We'll want to use WebPageTest to run a series of performance tests against a few pages on svwiki before the change is made. Then, after the change, we'll re-run the tests and compare the results. We'll want to do those tests pretty close to the time that the actual change is made, in order to lessen the chances that other things may happen that affect the results (such as the page content changing).

If we look at T176082 we can see that Tgr suggested that TemplateStyles should be deployed to svwiki next week, but Deskana has not yet answered. Once TemplateStyles is deployed to svwiki, I would probably start migrating styles the same day or the day after.

Also, I'm not sure if this is obvious to you, but to be super clear: I would first have to add the <templatestyles> tag to the templates and then some time later, only once the styles are loaded twice, will I remove them from the MediaWiki namespace. This is because of cache and it would be unacceptable for viewers to, say, see an infobox that isn't floated to the right. Exactly how long this "some time" is, I cannot say. You will have to ask Krinkle about that. This also means that for a limited time, performance will stay the same or even be degraded. Of course, a few days of degraded performance has to be put into perspective of (hopefully) improved performance for years to come. It's a good example of "you can't make an omelette without breaking some eggs".

@Nirmos Makes sense, thanks. We'll see what we can do about getting a test against svwiki set up before that deploy happens.

In T185853#3934305, @Nirmos wrote:

Also, I'm not sure if this is obvious to you, but to be super clear: I would first have to add the <templatestyles> tag to the templates and then some time later, only once the styles are loaded twice, will I remove them from the MediaWiki namespace. This is because of cache and it would be unacceptable for viewers to, say, see an infobox that isn't floated to the right.

TemplateStyles generates CSS inside the page HTML, there is no caching involved in the HTTP layer.

What I mean is, after editing a template, all articles are not updated immediately.

Let me add a WebPageTest run, is there any specific URL you wanna test?

In T185853#3934479, @Nirmos wrote:

What I mean is, after editing a template, all articles are not updated immediately.

Ah, fair point. Although if the tests are against a specific page you can just purge it by hand. (OTOH I don't know if changes to sitewide styles take effect immediately.)

I just realized that this might coincide with T185945. We don't want to make one measurement before, and then another when the HTML has changed because of that task. That would be comparing apples and oranges. I'm so sorry.

@Nirmos If you can suggest a page that you want us to measure, what we'll do is to set up a test that runs every so often (like once every 60 or 90 minutes), and we'll leave that running for a while. As long as this change doesn't happen at exactly the same moment as T185945, we should be able to distinguish between their effects.

Just need to know what page you want measured :-)

In T185853#3935216, @Tgr wrote:

In T185853#3934479, @Nirmos wrote:

What I mean is, after editing a template, all articles are not updated immediately.

Ah, fair point. Although if the tests are against a specific page you can just purge it by hand. (OTOH I don't know if changes to sitewide styles take effect immediately.)

Purging a specific page does not help here. That would only make sure the styles are loaded twice for that page, but the measurement where styles are loaded twice is unimportant. I cannot remove the styles from the MediaWiki namespace until all pages are loading the styles twice.

Can I only pick one page? To be able to draw any conclusions from this, I think we need to look at at least two different pages. Most styles here are only for articles. In that regard, performance should stay about the same for articles, but for non-articles, there should be a noticeable improvement. Here are some articles:
https://sv.wikipedia.org/wiki/Stockholm (long article)
https://sv.wikipedia.org/wiki/Butanon (short article without any navboxes)
https://sv.wikipedia.org/wiki/M%C3%A1rio_Zagallo (short article, but many navboxes. Useful for determining the impact of T168333)

And here are some non-articles:
https://sv.wikipedia.org/wiki/Special:Senaste_%C3%A4ndringar (RecentChanges)
https://sv.wikipedia.org/wiki/Wikipedia:Bybrunnen (the village pump)
https://sv.wikipedia.org/wiki/Portal:Huvudsida (the main page)

Let me prepare and fix that today. I guess we only need to run it for a couple of days right? I'll make dashboard specific for the pages when we get some metrics.

So it is only desktop that is interesting? I wanna measure in two different browsers so have better data. I'll spin up a new machine for that later today, I guess in Germany or what's the closest location for Sweden in AWS.

Or actually I think it's easier/better to setup another instance to run with WebPageReplay: it will give us more stable metrics and easier to see the difference.

I've got it up and running now, I'll watch the log for a while to verify that everything is ok and then I'll create the dashboard.

It runs as it should now, there was a problem in beginning with the URL names, the lib we use to upload the metrics couldn't use % in the path names. It needs to run a couple of hours so all metric paths is created in Graphite but it should be done by tomorrow and then we can make the switch whenever you want to do that ping @Nirmos

One thing: Our metrics at AWS_S3 is removed after 7 days so let me know when you do the switch so I can collect some of the raw data (videos, trace logs etc).

I guess we only need to run it for a couple of days right?

TemplateStyles is not yet deployed to svwiki. If you turn if off in a few days, we'd need to turn it on again once TemplateStyles is deployed. I'm not sure what is most convenient for you.

It runs as it should now

Thank you.

make the switch whenever you want to do that

TemplateStyles is not yet deployed to svwiki, so there is nothing I can do. This is being tracked in T176082. As I understand it, there are no technical blockers there – it just needs an "ok" from Deskana.

I think we can just keep it running, hope it will go live this week.

We test six URLs with Firefox and Chrome, two different set of latencies (50 ms and 100 ms).

I created a dashboard: https://grafana.wikimedia.org/dashboard/db/webpagereplay-sweden focusing on the metrics we usually pay most attention to. There's a lot of graphs there now and I think it will make it easier for us to spot the change.

Visual Metrics = metrics that we collect from a video recording of the screen. First Visual Change is when the first content is painted on the screen. In almost every case that is the text and the "frame" for us. Speed Index is the average time at which visible parts of the page are displayed. You can read the full definition here.

For Chrome we also collect internal Browser Metrics so we can see how much time each page spends in Loading/Scripting/Layout/Painting.

The metrics looks really stable so hopefully we can measure the difference and then we can also create a comparison video with before/after so it easier for us to actually see the difference.

I'm starting adding templatestyles tags now.

Are they there now? I couldn't see any difference in rendering the last seven days:
https://grafana.wikimedia.org/dashboard/db/webpagereplay-sweden?orgId=1&from=now-7d&to=now

As you can see at https://sv.wikipedia.org/wiki/MediaWiki-diskussion:Gadgets-definition#Kandidater I have added templatestyles tags for all templates except ambox. Once ambox is done too, I will start removing styles from the MediaWiki namespace.

If you haven't seen any difference in performance, that's a good thing! It means that the amount of CSS for templates is so insignificant compared to other things (OOUI etc) that we can say with great certainty that other communities can migrate styles freely without having to worry about performance.

templatestyles tag is now added to ambox as well. I will now start remove CSS from the MediaWiki namespace, so that styles aren't loaded twice (although Peter's comment above suggests this hasn't been noticeable).

@Nirmos Do you have the exact time/date for changes?

When I look at the charts all First Visual Change and Speed Index are really stable. Only thing I could see is Bybrunnen that got an increase 2/3 21.00 UTC. I can see it in Firefox and slow increase in Chrome, but I guess this is other content changes right?

Time spent in JavaScript has increased, for articles I could see an increase in time spent in scripting 24/2 for https://sv.wikipedia.org/wiki/M%C3%A1rio_Zagallo and https://sv.wikipedia.org/wiki/Butanon. For non-articles I could see the same increase (ca 100 ms) 19-20/2 for all the different pages.

Does it match your expectations or should I look into it deeper?

I thought we had established in T185853#3966830 that the amount of styles is so small that it does not affect performance. I've been meaning to ask if this task should be closed, but other things have gotten in the way.

Do you have the exact time/date for changes?

So, this is a bit of a mess. I've had to revert my changes because of T188143, and I'm now in the process of adding back the templatestyles tags.

Bybrunnen that got an increase 2/3 21.00 UTC

This does not seem to coincide with anything that I have done.

Time spent in JavaScript has increased, for articles I could see an increase in time spent in scripting 24/2 for https://sv.wikipedia.org/wiki/M%C3%A1rio_Zagallo and https://sv.wikipedia.org/wiki/Butanon. For non-articles I could see the same increase (ca 100 ms) 19-20/2 for all the different pages.

This is almost definitely because of https://sv.wikipedia.org/wiki/MediaWiki:Gadget-maphack.js. This is a workaround for T187741. We should probably just add lat and long to the mapframes and remove that gadget. I'll make sure that is done. Thanks for picking that up.

With all of that said , though, I think this task can be closed. When you asked on Feb 13, all styles were loaded twice (from the MediaWiki namespace as well as templatestyles), which is the worst case scenario, and you said there weren't any difference in rendering the last seven days.