Let's do this: Rollout page previews to 100% of anons on English Wikipedia
Closed, ResolvedPublic3 Story Points

Description

Blockers

  • all subtask of epic reviewed and resolved(if not removed as subtasks)
  • Community consultation for behavior for logged-in users started and any follow ups outlined by @CKoerner_WMF have been added here.
  • T189906 has been completed and VirtualPageViews is enabled on all wikis
  • Patches relating to T190188 are on English Wikipedia
  • @CKoerner_WMF has reviewed blockers and ticked this box personally to say they are correct
  • @ovasileva has reviewed blockers and ticked this box personally to say they are correct
  • @Tbayer has reviewed blockers and ticked this box personally to say they are correct
  • @Jdlrobson has reviewed blockers and ticked this box personally to say they are correct
  • @Nirzar has reviewed blockers and ticked this box personally to say they are correct

Acceptance criteria

  • All boxes in the blockers have been ticked. We're ready to go!
  • Patch is written per developer notes deploying to 10% of English Wikipedia
  • Patch is written per developer notes deploying to 25% of English Wikipedia
  • Patch is written per developer notes deploying to 50% of English Wikipedia
  • Patch is written per developer notes deploying to 75% of English Wikipedia
  • Patch is written per developer notes deploying to 100% of English Wikipedia
  • Deploy is scheduled and a meeting is added to the calendar. @ABorbaWMF , a design rep and @ovasileva should be available for testing.

Developer notes

This should be done in three deploys - Bump to 10% (0.1) then 50% (0.5) and then 100% (0).
Note: wgPopupsAnonsExperimentalGroupSize should be set to 0 for 100% ('default'). Counter-intuitively this means Popups is enabled to 100% of anons. Note given the value of wmgUsePopups this will not enable Popups for non-wikipedias

During deploy

  • Verify that Page previews is showing on all wikipedias for anonymous users and for logged in (only if opted in) - I suggest testing Spanish, German, Hebrew and English to verify. Be careful to test anonymous behaviour in an incognito window as you may have previously been opted in via the footer)
  • Verify that the rate of events for VirtualPageViews is still at healthy values (estimated average: ~700-800 events/sec; may be above or below that depending on time of day and day of week). Errors are around normal rate.
  • Check any change in error rate (and where those errors are coming from)
  • Verify that page previews remains disabled on all other projects e.g. Wikidata, Wiktionary
  • Green light from @ABorbaWMF to sync
  • Green light from @Nirzar to sync
  • Green light from @ovasileva to sync

Post deploy

For checking during deployments:

There are a very large number of changes, so older changes are hidden. Show Older Changes

Deploy is scheduled for Monday 9th April

Sorry, is this a typo of some sort? It's not possible to do a full consultation and gather consensus from the community to enable the page previews feature in a single week. (Related: comments on T182317 .)

Also, could someone please clarify exactly what the relationship is between this task and its two parent tasks? Thanks.

@Yair_rand - this deployment only refers to behavior for logged-out users. We will be starting a consultation for the behavior for editors and other logged-in users tomorrow but are not planning on implementing any decisions from the consultation as a part of this task.

ovasileva updated the task description. (Show Details)Apr 2 2018, 6:13 PM

could someone please clarify exactly what the relationship is between this task and its two parent tasks? Thanks.

This is the task to actually make the config change (Wikimedia-Site-requests tag). The other tasks are the goal and epic cards.

Jdlrobson updated the task description. (Show Details)Apr 2 2018, 6:14 PM
This comment was removed by ABorbaWMF.

@ABorbaWMF did you mean to paste that here on the deploy task? If so could you please add some context by what "language tests" means ? Thanks!

ABorbaWMF added a comment.EditedApr 2 2018, 10:32 PM

Premature testing, sorry about that

> Verify that Page previews is showing on all wikipedias - I suggest testing Spanish, German, Hebrew and English to verify

I was testing this one. Just figured I would test the top ten.

Jdlrobson updated the task description. (Show Details)Apr 2 2018, 11:08 PM
Jdlrobson updated the task description. (Show Details)

@Yair_rand - this deployment only refers to behavior for logged-out users.

A week isn't long enough to gather consensus for enabling it for logged-out users either. There isn't a separate shorter process for different classes of users, and limiting it to logged-out users doesn't make it okay to defy consensus.

There's a few ways this could go:

  • The deployment to logged-out users happens against consensus. An admin quickly implements the already-made community decision to not have Page Previews, by adding .mwe-popups { display: none !important; } to Common.css. Either the staff decides not to take action against that, or they do and the board overrides that decision, as many of its members were elected/selected specifically on the platform that they would stop the WMF from doing such things in the aftermath of the superprotect crisis. The CSS solution makes Page Previews no longer available for anyone, even those who want to opt in. The relationship between the community and the WMF is heavily damaged.
  • The deployment happens against consensus and the board does not veto a new superprotect. The community's ability to manage local affairs is damaged from all JS/CSS/gadgets becoming uneditable, the software develops loads of new problems coming out of the inevitable fundamental issues with trying to prevent admins from doing things on Mediawiki. Trust is gone, and the WMF and community get stuck in a conflict as they spend much of their time trying to stop the other from doing any more damage. Board elections come around, and the community either succeeds or fails in fixing both the board and the WMF as a whole.
  • The deployment happens against consensus, the community has an immediate shift of opinion on whether it should be deployed, and a consensus that supports the deployment for not-logged-in users. No action is taken against the deployment, and we have the feature running several weeks earlier than we would have had if we waited for consensus first. Also, trust in the WMF goes down because they defied consensus as it was at the time.
  • Consensus changes before deployment happens, as a result of the discussion on whether to enable for logged-in users. Similar results.

These sound like a terrible set of outcomes. Please reconsider this.

DannyH added a subscriber: DannyH.Apr 3 2018, 8:53 PM

Hi @Yair_rand: Thanks for raising the alam. You're right, we don't want any of those outcomes, and there's no superprotect coming, on this or any other product release.

Olga's going to post tomorrow on the village pump about the work that the team's done to improve the feature, and our plans to continue deploying the feature. Our intention is to continue the conversation with the community. Chris posted a message on VP technical last month:

https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_164#Improvements_to_Page_Previews

This didn't get any response, so we're hoping that tomorrow's post will get more people involved.

Olga's going to include a list of the problems that people reported last time we talked, and how those problems were addressed and fixed. We want to make sure that we've cleared up the things that people were concerned about.

Once that conversation starts, we want to know what people think, especially if there's a problem. I'm glad that you care about this -- we could use some help making sure people see our post. We don't usually start RfCs, but if there is one, then obviously we'll want to participate and be a part of the discussion. Do you have any advice?

Tbayer updated the task description. (Show Details)Apr 4 2018, 5:44 PM
ovasileva set the point value for this task to 3.Apr 5 2018, 5:10 PM
Jdlrobson renamed this task from Let's do this: Deploy page previews to Let's do this: Deploy page previews to English Wikipedia.Apr 11 2018, 5:52 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)Apr 11 2018, 6:05 PM
Jdlrobson renamed this task from Let's do this: Deploy page previews to English Wikipedia to Let's do this: Deploy page previews to 100% of anons on English Wikipedia.Apr 11 2018, 6:13 PM
Jdlrobson updated the task description. (Show Details)
Nuria renamed this task from Let's do this: Deploy page previews to 100% of anons on English Wikipedia to Let's do this: Rollout page previews to 100% of anons on English Wikipedia.Apr 11 2018, 6:14 PM
Jdlrobson updated the task description. (Show Details)Apr 11 2018, 6:23 PM

Change 425588 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable Page Previews for 10% enwiki anon users

https://gerrit.wikimedia.org/r/425588

^ Above patch is just to check we can cope with the added load in EventLogging. Further deploys will be dependent on the community consensus (see blockers)

Ottomata updated the task description. (Show Details)Apr 12 2018, 1:25 PM
ovasileva updated the task description. (Show Details)Apr 16 2018, 12:48 PM
ovasileva updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)Apr 16 2018, 3:55 PM
pmiazga updated the task description. (Show Details)Apr 17 2018, 9:11 AM

Change 427082 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable PagePreviews for 25% anon users on enwiki

https://gerrit.wikimedia.org/r/427082

ovasileva updated the task description. (Show Details)Apr 17 2018, 11:08 AM

Change 427082 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable PagePreviews for 25% anon users on enwiki

https://gerrit.wikimedia.org/r/427082

Mentioned in SAL (#wikimedia-operations) [2018-04-17T12:04:31Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 25% of annons on enwiki - T191101 (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2018-04-17T12:31:44Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 25% of annons on enwiki, take #2 - T191101 (duration: 00m 58s)

Change 427103 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable PagePreviews for 50% anon users on en wiki

https://gerrit.wikimedia.org/r/427103

CKoerner_WMF updated the task description. (Show Details)Apr 17 2018, 1:38 PM
CKoerner_WMF updated the task description. (Show Details)

Communities have been informed prior to deployment. T182317

The community discussion is still open and we’ll continue having the conversation if any new comments come in, but it’s been quiet for the past couple of days. So far, we haven’t heard any direct concerns around the changes proposed above.

ovasileva updated the task description. (Show Details)Apr 17 2018, 2:01 PM

Change 427103 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable PagePreviews for 50% anon users on en wiki

https://gerrit.wikimedia.org/r/427103

Mentioned in SAL (#wikimedia-operations) [2018-04-17T14:30:43Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 50% of anons for enwiki - T191101 (duration: 00m 58s)

Nirzar updated the task description. (Show Details)Apr 17 2018, 2:35 PM
ovasileva updated the task description. (Show Details)Apr 17 2018, 2:48 PM

Change 427137 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable PagePreviews for 75% anon users on en wiki

https://gerrit.wikimedia.org/r/427137

Tbayer updated the task description. (Show Details)Apr 17 2018, 3:54 PM

Did not see any remaining blockers from my perspective. (Notice though that the work on measuring the usage of this new feature is still in progress, see T186728 etc.)

Change 427137 merged by Mobrovac:
[operations/mediawiki-config@master] Enable PagePreviews for 75% anon users on en wiki

https://gerrit.wikimedia.org/r/427137

Mentioned in SAL (#wikimedia-operations) [2018-04-17T16:02:22Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 75% of anons for enwiki - T191101 (duration: 00m 58s)

Change 427178 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable PagePreviews to all anonymous users

https://gerrit.wikimedia.org/r/427178

Change 427178 merged by Mobrovac:
[operations/mediawiki-config@master] Enable PagePreviews to all anonymous users

https://gerrit.wikimedia.org/r/427178

Mentioned in SAL (#wikimedia-operations) [2018-04-17T17:16:49Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 100% of anons for enwiki - T191101 (duration: 00m 59s)

Jhernandez added a subscriber: Jhernandez.EditedApr 17 2018, 5:17 PM

There have been some issues today.

We have been deploying progressively using the A/B testing frontend code, and the wgPopupsAnonsExperimentalGroupSize. As such, we have been mistaken in the amount of users we have deployed the feature to. See in the code, how experimentGroupSize is divided by 2, and that will be the % used for the on group.

Basically, if wgPopupsAnonsExperimentalGroupSize is at 0.5, then 25% of the anon users will get Popups enabled, so we have been deploying to half the users we were expecting through the day.

What we wantedWhat actually happened
25%12.5%
50%25%
75%37.5%

At the "75%" deploy we noticed the numbers didn't check out and did the investigation.

The maximum rollout we could do with this mechanism is 50% on.

Given the window and the train schedule, Nuria, Marco, Piotr, Olga and myself decided to go 100% (disabling the A/B test) and checking the logs and stats before the train, to have the chance to roll back if needed.

There are a few lessons in the story, one of them being: do not use A/B testing code to do progressive rollouts, and if you really do, test and double check your progressive rollouts.

Nuria added a subscriber: Nuria.Apr 17 2018, 5:57 PM

Given the comment above we think the ratio of events (discounting lauch traffic do to "artifical" usage of feature) is goinmg to be about 1200 reqs/sec

Jdlrobson updated the task description. (Show Details)Apr 17 2018, 9:51 PM

Error rate still at 0

Smoke test results thus far on Chrome, Safari, Edge, and Firefox:
https://docs.google.com/spreadsheets/d/1CdHhFDoTOfbLDCuwVViAWSFlduNE88h3mCbSKIRbKcs/edit?usp=sharing

Sheet is called "Smoke Test - Previews - April 17"

pmiazga updated the task description. (Show Details)Apr 17 2018, 10:39 PM
pmiazga updated the task description. (Show Details)
ovasileva updated the task description. (Show Details)Apr 17 2018, 10:57 PM

On the back-end side things are looking good. After the initial increase in number of requests that caused Varnish misses, we are back to normal levels, indicating that the majority of requests are served from cache. In terms of errors things are looking good too: no noticeable changes in error rates in parsing/serving summaries.

Nuria added a comment.Apr 18 2018, 2:49 PM

Request ratio peak is about this time, looks like things are all good.

Jdlrobson updated the task description. (Show Details)Apr 18 2018, 3:21 PM
Nirzar updated the task description. (Show Details)Apr 18 2018, 3:57 PM
ABorbaWMF updated the task description. (Show Details)Apr 18 2018, 4:08 PM

removing " Parent tasks- both goal and epic are resolved." from the acceptance criteria. Let's go ahead an resolve the epic, but keep T191888: Change page previews configuration for opt-in accounts under the goal card just so that everything is in one place.

ovasileva updated the task description. (Show Details)Apr 19 2018, 8:46 AM

@Jdlrobson - do we still need an epic for post-deploy cleanup?

Jdlrobson updated the task description. (Show Details)Apr 20 2018, 1:04 AM

^ Twitter went a bit crazy so we can consider that checkbox done :)

@Jdlrobson - do we still need an epic for post-deploy cleanup?

I've opened up T192622 to continue this journey!

Jdlrobson closed this task as Resolved.Apr 20 2018, 1:06 AM
Jdlrobson updated the task description. (Show Details)