Change 879659 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/879659

Change 879661 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] [Just in case] Disable thumbnails on English Wikipedia

https://gerrit.wikimedia.org/r/879661

Noting that switch of one row in codfw is broken (T327001) which means degraded capacity across the whole infra specially in swift frontend and thumbor. We hope to get it fixed ASAP but also today is a holiday in US

Change 880531 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Add enwiki to desktop-improvements group

https://gerrit.wikimedia.org/r/880531

Some random numbers

General page view:

• Images: I see around twenty more images being loaded than legacy vector and 40KB heavier. Looking at number of 150M page views per day (from stats.wikimedia.org), this roughly translates to 3 billion more requests a day if people only visit wikipedia once, to correct that assumption, I think a more realistic number would be around 1B more requests (a pessimistic number) or 1.8TB more network. Thankfully the images are part of the interface and heavily cached both server-side and client-side. No impact on appservers.
• HTML I'm seeing around 15KB increase in HTML of the page. There is no client-side caching here so it'll translate to an extra 2TB traffic per day to the text cluster (without increase in number of requests). This will increase the pressure on the network between appservers and outside but not much.
• JS: It is around 40KB heavier which would actually cause a large increase in text cluster traffic without increasing number of reqs. Since this is heavily cached both client-side and server-side. It's a bit hard to pin down. But the traffic hitting text cluster (and fully cached aka CDN) would probably increase by 2TB a day but would slowly subside to a much smaller number when users browsers cache the assets. Gradual deployment would decrease the cache stampede in these cases. I don't think this will have any major impact outside of CDN.

Search:

• Looking at turnilo. It seems we have around 33M search reqs/day in English Wikipedia. The API response size will be bigger (new vs old), 61GB more data in total, that's not much. I checked and the new one is also cached at the CDN level, so no major increased pressure on the appservers (again, gradual deployment to avoid cache stampede)
• The biggest impact of this change: Thumbnails: Search will now have thumbnails. It's hard to pin down how many more requests will happen to fetch thumbnails but I estimate around 300M more requests to the upload cluster. Some of it would inevitably lead to more pressure on swift and thumbor. Going with 3KB size per thumbnail, I estimate around 1TB more to upload which is not that much.
• Adding thumbnails might change user behavior. Either get them to search more or make them avoid hitting Special:Search directly. That would change numbers.

Server-side:

• This is the most complicated one to estimate. I'm getting all sorts of different number of queries made to the database which are products of different levels of caching. I can't say anything conclusive but my gut feeling is towards some level of increase but not much.
• It also means we can't really estimate the page performance. I'm seeing no major difference in WebPageReplay tests but maybe I'm missing something.

Overall, I'm not seeing any major pressure on the appservers but major increase of traffic and requests across all layers of cache in all clusters possibly impacting swift and thumbor. None of them are too concerning (famous last words) assuming T327001 would be fixed by then and deployment is gradual (Emphasize: I'm not authority on traffic and cache side). But I think this will reduce our resiliency against spikes and attacks. There is nothing bad about it. It's an important feature and our infrastructure must adapt. Thankfully I don't see any vertical scalability issue (=we only need to buy more servers) but I suggest working on expanding cache clusters across the fleet.

Change 881020 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] [25%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/881020

Change 881021 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] [50%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/881021

Change 881022 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] [75%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/881022

Noting that T327256 just got filed, which I can reproduce. It seems that using the desktop version of legacy vector on mobile has broken. I'm not sure if it is related to this, but given that its broken all group0 and group1 wikis but not group2 it may be related?

Hi @Asartea - thanks for your comment! Thanks for flagging, we will look at T327256: Desktop mode problem on mobile with the legacy Vector later today. None of the patches on this task are currently on the train (we will be deploying them separately) so this task is not directly related.

Change 879659 merged by jenkins-bot:

[operations/mediawiki-config@master] [10%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/879659

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:17:47Z] <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:19:36Z] <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:29:17Z] <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 11m 30s)

Change 881020 merged by jenkins-bot:

[operations/mediawiki-config@master] [25%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/881020

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:35:10Z] <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:36:58Z] <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:44:16Z] <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 09m 06s)

Change 881021 merged by jenkins-bot:

[operations/mediawiki-config@master] [50%] English Wikipedia uses Vector 2022 skin, adds instrumentation

https://gerrit.wikimedia.org/r/881021

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:49:41Z] <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:51:27Z] <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-18T15:58:34Z] <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]] (duration: 08m 52s)

Change 881022 merged by jenkins-bot:

[operations/mediawiki-config@master] [75%] English Wikipedia uses Vector 2022 skin

https://gerrit.wikimedia.org/r/881022

Mentioned in SAL (#wikimedia-operations) [2023-01-18T16:11:15Z] <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-18T16:13:03Z] <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-18T16:20:39Z] <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 09m 24s)

Change 881451 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] Bump English Wikipedia event logging from 0.5 to 1%

https://gerrit.wikimedia.org/r/881451

Change 881451 merged by jenkins-bot:

[operations/mediawiki-config@master] Bump English Wikipedia event logging from 0.5 to 1%

https://gerrit.wikimedia.org/r/881451

Mentioned in SAL (#wikimedia-operations) [2023-01-18T21:23:25Z] <kindrobot@deploy1002> Started scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-18T21:25:49Z] <kindrobot@deploy1002> kindrobot and jdlrobson: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-18T21:36:27Z] <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] (duration: 13m 01s)

Change 879661 abandoned by Jdlrobson:

[operations/mediawiki-config@master] [Just in case] Disable thumbnails on English Wikipedia

Reason:

Not needed. Nothing exploded (yay)

https://gerrit.wikimedia.org/r/879661

Change 880531 abandoned by Jdrewniak:

[operations/mediawiki-config@master] Add enwiki to desktop-improvements group

Reason:

https://gerrit.wikimedia.org/r/880531