• Monday: Ensure QA of T321498 has been completed.
• Tuesday: configuration change to enable the feature on mediawiki.org and run synthetic tests (most likely @nray)
• Wednesday/Thursday: Enable on all wikis.

@Jdlrobson If possible, I'm interested in also deploying the config change on Wednesday to monitor the synthetic test we have for cawiki as well as the additional navigation timing data we will get. Do you mind if I change this to state the following?

• Monday: Ensure QA of T321498 has been completed.
• Tuesday: configuration change to enable the feature on Group 0 wikis. Monitor synthetic test dashboard and Navigation timing dashboard
• Wednesday: configuration change to also enable the feature on Group 1 wikis. Monitor synthetic test dashboard and Navigation timing dashboard
• Thursday: Enable on all wikis. Monitor synthetic test dashboard and Navigation timing dashboard

@Edtadros we'll need to do some QA on this today on mediawiki.org, @nray will ping you when it's ready to be tested.

Change 885395 had a related patch set uploaded (by Nray; author: Nray):

[operations/mediawiki-config@master] Enable ClientPreferences for group0

https://gerrit.wikimedia.org/r/885395

Change 885395 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable ClientPreferences for group0

https://gerrit.wikimedia.org/r/885395

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:23:24Z] <kindrobot@deploy1002> Started scap: Backport for [[gerrit:885395|Enable ClientPreferences for group0 (T327979)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:25:03Z] <kindrobot@deploy1002> kindrobot and nray: Backport for [[gerrit:885395|Enable ClientPreferences for group0 (T327979)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:33:41Z] <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:885395|Enable ClientPreferences for group0 (T327979)]] (duration: 10m 17s)

This was deployed to all group0 wikis at 2023-01-31 21:33:41 UTC. If all goes well, I'll plan to deploy to group 1 tomorrow

Test Result - Prod

Status: ✅ PASS
Environment: testwiki
OS: macOS Ventura
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

As anonymous user open any page in Vector 2022 skin
Hit the toggle button in the bottom right to disable the limited width
refresh page
✅ AC1: limited width remains disabled

Change 885841 had a related patch set uploaded (by Nray; author: Nray):

[operations/mediawiki-config@master] Enable client preferences for group1

https://gerrit.wikimedia.org/r/885841

This was deployed to all group0 wikis at 2023-01-31 21:33:41 UTC. If all goes well, I'll plan to deploy to group 1 tomorrow

I have a meeting conflict during the backport window so I cannot deploy to group 1 today, but the performance graphs for group 0 look okay to me, so I think this can wait until tomorrow

Change 885841 abandoned by Nray:

[operations/mediawiki-config@master] Enable client preferences for group1

Reason:

https://gerrit.wikimedia.org/r/885841

Change 885841 restored by Nray:

[operations/mediawiki-config@master] Enable client preferences for group1

https://gerrit.wikimedia.org/r/885841

In T327979#8579996, @nray wrote:

This was deployed to all group0 wikis at 2023-01-31 21:33:41 UTC. If all goes well, I'll plan to deploy to group 1 tomorrow

I have a meeting conflict during the backport window so I cannot deploy to group 1 today, but the performance graphs for group 0 look okay to me, so I think this can wait until tomorrow

After more discussion, @Jdlrobson will deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/885841 to group 1 today

Change 885841 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable client preferences for group1

https://gerrit.wikimedia.org/r/885841

Mentioned in SAL (#wikimedia-operations) [2023-02-01T22:20:55Z] <kindrobot@deploy1002> Started scap: Backport for [[gerrit:885841|Enable client preferences for group1 (T327979)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-01T22:22:47Z] <kindrobot@deploy1002> nray and kindrobot: Backport for [[gerrit:885841|Enable client preferences for group1 (T327979)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-01T22:31:33Z] <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:885841|Enable client preferences for group1 (T327979)]] (duration: 10m 37s)

Change 886118 had a related patch set uploaded (by Nray; author: Nray):

[operations/mediawiki-config@master] Enable client preferences everywhere

https://gerrit.wikimedia.org/r/886118

Group 0 Webpagereplay results

• We deployed the client preferences feature to all Group 0 wikis on Jan 31 2:33 pm Mountain Time
• Many of the metrics had a big spike starting Jan 31 9:00 pm Mountain time (~6.5 hours after the deploy), but the tests suggest this was caused by the release of the Wiki Loves banner campaign.
• Apart from the apparent Wiki Loves banner spike, no consistent spikes in any of the metrics we collect
• I met with Peter on Feb 1 and we discussed that, although still helpful in detecting front-end performance regressions, the caveat with the webpagereplay results are that they run on a fairly fast CPU. The performance team has future plans to pin the CPU to a lower speed to have more representative results.

Group 1 Webpagereplay results

• We deployed the client preferences feature to all group 1 wikis on Feb 1 4:17 p.m. Mountain time
• No consistent spikes in any of the metrics we collect.
• Many of the metrics had a big spike starting Jan 31 9:00 p.m. Mountain time, but the tests suggest this was caused by the release of the Wiki Loves banner campaign.

Navigation timing

• In addition to monitoring the synthetic tests, I've also been monitoring many of the navigation timing metrics such as first contentful paint (pictured below) which this deployment might affect
• I haven't noticed any consistent and obvious spikes in these metrics

Based on these results, I feel comfortable deploying to all wikis today and will continue to monitor the metrics after the deployment.

Change 886118 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable client preferences everywhere

https://gerrit.wikimedia.org/r/886118

Mentioned in SAL (#wikimedia-operations) [2023-02-02T21:19:38Z] <brennen@deploy1002> Started scap: Backport for [[gerrit:886118|Enable client preferences everywhere (T327979)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-02T21:21:23Z] <brennen@deploy1002> brennen and nray: Backport for [[gerrit:886118|Enable client preferences everywhere (T327979)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-02T21:30:53Z] <brennen@deploy1002> Finished scap: Backport for [[gerrit:886118|Enable client preferences everywhere (T327979)]] (duration: 11m 14s)

Group 2 Webpagereplay results

• We deployed the client preferences feature to all group 2 wikis on Feb 2 2:30 p.m. Mountain time
• On the Barack Obama page, the "largest image painted" and largest heading painted" metrics had a series of runs that consistently showed 30ms increases to the "largest image painted" and "largest heading painted" metrics. I looked at the sitespeedio trace log for these runs to determine a reason for this, and it looks like this is a result of the popups code running before the relevant paints of the wiki loves banner instead of after (pictured below). Additionally, I did not see this increase on the other webpagereplay pages for enwiki, and the latest run settled back to pre-deploy levels. Therefore, I think this is okay.

Trace log before deployment

Trace log after deployment

To clarify my previous comment, this is the small ambiguous increase I'm referring to:

Hi everyone -
I wanted to weigh in on the sign-off criterion, "Work out next steps for continuing the persistent settings functionality."

Clarifying what this means: we decided for this particular issue to merge the current, client-side cookies-based approach so as to address the time-sensitive need to make this particular preference persistent.

At the same time, when we initially proposed this change, we heard about some potential concerns with how this would be implemented (specifically initially referring to using LocalStorage for this purpose), and consequently, we had a conversation with both Performance and SRE about how to ensure that this fix would scale for other potential future use cases beyond the storage of the the initial width preference.

The specific next steps here:

1. The parties involved in discussing this initial patch will be meeting tomorrow to do a retrospective on this fix and how we approached in
2. Several weeks from now, we will be discussing longer-term technical approaches to storing this and potentially other preferences in the future.

I would say overall that I sign off on this particular aspect of these next steps, and feel they will bring about the right kind of conversations that will lead to us discussing how this solution will scale.

For full sign-off, I would want QTE to weigh in.