Maniphest T162672

Deploy page previews to 90% of users on all wikis but English and German
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Blockers

All current EventLogging bugs are resolved and data is confirmed by @Tbayer (see T162672#3496940)
- Tracking task for @Tbayer's analysis: T158172: Analyze results of stage 0 page previews test
Deployment must happen after green light from @ovasileva (see T162672#3496940)

Plan (YMMV):

Reset the instrumentation sampling rate to 0.001 for all wikis, i.e. revert rOMWCd5c4bd728014: pagePreviews: Increase instrumentation sampling rate.
Deployment must happen in few stages (3 or more) in order to make rollback easier – if we start overloading the edge caches (and origin servers potentially), say.
- Deploy to a chunk of 50 wikis and assess the impact on Page Previews, RESTBase, and Varnish performance, if any.
  - https://grafana.wikimedia.org/dashboard/db/reading-web-page-previews
  - https://grafana.wikimedia.org/dashboard/db/varnish-caching?var-cluster=text&var-site=All
- If there's no significant impact, then increase the chunk size to 100/150/200.
Ensure that the EventLogging instrumentation is disabled for the chunk.
The Ops and Services teams are notified after each deploy.
We pause for a day (at least) between deploys so that each team has a chance to observe the effect of increased load.

Current status

~~Thursday, 3rd August: The first 50 wikis (in alphabetical order) from the list were deployed to 🎉🎉🎉~~

Monday, 21st August: Page Previews is currently deployed to all wikis but enwiki and dewiki 🎉🎉🎉

Testing criteria

For each rollout (in sets of 50?), select 5 wikis with different scripts:

For each wiki:

Go to the main page of the wiki and hover over a couple of links, ensure previews appear
Open a new page by selecting the preview, ensure page opens
On new page, ensure previews appear
Hover over an article containing disambiguation text, ensure disambiguation text does not appear in preview
If the script is rtl, ensure preview appears rtl and the settings cog is placed accordingly

For a single wiki:

Disable previews using settings cog, ensure previews are disabled
Re-enable previews using link in footer, ensure previews appear again
Log in, enable previews, ensure previews appear
Disable previews, ensure previews do not appear

Also:

Ensure previews are NOT enabled on dewiki and enwiki
Ensure previews are enabled on ruwiki

Details

Subject	Repo	Branch	Lines +/-
Roll page previews out to all wikis except en and de wiki	operations/mediawiki-config	master	+152 -3
pagePreviews: Deploy to next 100 stage 1 wikis	operations/mediawiki-config	master	+100 -2
pagePreviews: Deploy to first 50 of stage 1 wikis	operations/mediawiki-config	master	+90 -6
pagePreviews: Create pp_stage0.dblist	operations/mediawiki-config	master	+14 -23
pagePreviews: Tidy up config	operations/mediawiki-config	master	+26 -41

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dereckson	T68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki)
Resolved	Jdlrobson	T70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature
Resolved	ovasileva	T154635 [EPIC] Deploy page previews to English and German Wikipedia
Resolved	ovasileva	T162672 Deploy page previews to 90% of users on all wikis but English and German
Resolved	• Tbayer	T158172 Analyze results of stage 0 page previews test
Resolved	• Tbayer	T167273 pageLoaded events are logged infrequently, if at all
Resolved	• Tbayer	T164256 pageTitleHover not set correctly in Popups schema
Resolved	• Tbayer	T165461 Various sanity checks for data from the rewritten Popups instrumentation
Resolved	• Tbayer	T166323 perceivedWait value should be logged for all card interaction-ending events
Resolved	phuedx	T167365 Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent
Resolved	Jdlrobson	T169365 There is no possibility to check changes on user options save.
Resolved	• Tbayer	T167391 [Spike] Isolate the source of Popups event duplication
Declined	• Tbayer	T169581 Histogram for the number of duplicated events per browser session
Declined	• Tbayer	T169582 Examine several sessions which contained duplicate events
Resolved	phuedx	T170018 Duplicate events sent in Firefox after back button press
Resolved	• Tbayer	T167236 userSessionToken in RelatedArticles schema does not seem to survive beyond one pageview
Resolved	pmiazga	T159490 PagePreviews events system sends duplicate and incorrect events
Resolved	• Tbayer	T161769 Schema:Popups sends extraneous link interaction events in control condition
Resolved	phuedx	T159539 [2 hours] PagePreviews `totalInteractionTime` for `opened` event is calculated incorrectly
Resolved	pmiazga	T162924 PagePreviews sending "dwelledButAbandoned" after "opened"
Resolved	phuedx	T168449 event.linkInteractionToken can be undefined
Resolved	ovasileva	T149947 [Spike 1hr] Double-check through old hovercards EL related bugs
Resolved	phuedx	T140485 Schema:Popups logs "dwelledButAbandoned" and link open events at the same time
Resolved	phuedx	T173422 Investigate the increase in the number of requests to Swift after the Page Previews deploy

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Can you update this task?

Jdlrobson removed a project: Patch-For-Review.Jun 6 2017, 10:07 PM

ovasileva added a subtask: T158172: Analyze results of stage 0 page previews test.Jun 7 2017, 10:53 AM

Jdlrobson changed the task status from Open to Stalled.Jun 7 2017, 4:57 PM

phuedx updated the task description. (Show Details)Jun 8 2017, 9:35 AM

phuedx updated the task description. (Show Details)

ovasileva changed the task status from Stalled to Open.Jul 11 2017, 1:25 PM

ovasileva updated the task description. (Show Details)

Jdlrobson updated the task description. (Show Details)Jul 11 2017, 6:17 PM

Jdlrobson added a project: Readers-Web-Kanbanana-Board-Old.

Jdlrobson moved this task from Upcoming to 2016-17 Q4 on the Web-Team-Backlog board.

phuedx removed phuedx as the assignee of this task.Jul 12 2017, 5:04 PM

ovasileva moved this task from 2016-17 Q4 to 2017-18 Q1 on the Web-Team-Backlog board.Jul 13 2017, 5:40 PM

• Tbayer changed the status of subtask T158172: Analyze results of stage 0 page previews test from Stalled to Open.Jul 16 2017, 7:33 PM

Change 365696 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Enable page previews for everyone on labs

https://gerrit.wikimedia.org/r/365696

gerritbot added a project: Patch-For-Review.Jul 17 2017, 8:11 PM

We will have a week's worth of data from tomorrow. Olga and Tilman have a meeting on Friday to confirm we are still good for deploy on Monday.

ovasileva updated the task description. (Show Details)Jul 24 2017, 5:26 PM

Liuxinyu970226 subscribed.Jul 25 2017, 8:53 AM

phuedx updated the task description. (Show Details)Jul 25 2017, 8:56 AM

@Jdlrobson: Are you happy with the plan thus far? Is the estimate still applicable?

• mobrovac added a project: Services (watching).Jul 31 2017, 5:22 PM

Why wouldn't I be? Can't wait! :)

It's Wednesday! Roll out time?

Not yet. @Tbayer is still working on the analysis. We're looking at Aug 3 or August 7th currently.

phuedx updated the task description. (Show Details)Aug 2 2017, 2:01 PM

Jdlrobson updated the task description. (Show Details)Aug 2 2017, 2:21 PM

Now unblocked based on initial analysis provided by @Tbayer. Let's do this!

In T162672#3496940, @ovasileva wrote:

Now unblocked based on initial analysis provided by @Tbayer. Let's do this!

If possible, could you provide a link to the analysis for posterity?

phuedx updated the task description. (Show Details)Aug 3 2017, 1:35 PM

We'll be publishing it once we've run through the entire set of queries (will provide link then). The decision here in particular was based on a the small negative effects on pageviews (we were not able to reject the null hypothesis that there is no change between the experimental and control groups) as well as on the rate of previews usage of the test wikis.

phuedx updated the task description. (Show Details)Aug 3 2017, 2:54 PM

Change 369960 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] pagePreviews: Deploy to first 50 of stage 1 wikis

https://gerrit.wikimedia.org/r/369960

gerritbot added a project: Patch-For-Review.Aug 3 2017, 5:05 PM

Change 351287 merged by jenkins-bot:
[operations/mediawiki-config@master] pagePreviews: Tidy up config

https://gerrit.wikimedia.org/r/351287

Jdlrobson moved this task from Needs Design Review to Doing on the Readers-Web-Kanbanana-Board-Old board.Aug 3 2017, 6:08 PM

Change 351286 abandoned by Phuedx:
pagePreviews: Create pp_stage0.dblist

Reason:
Superseded by I505e9100.

https://gerrit.wikimedia.org/r/351286

Change 369960 merged by jenkins-bot:
[operations/mediawiki-config@master] pagePreviews: Deploy to first 50 of stage 1 wikis

https://gerrit.wikimedia.org/r/369960

I have some concerns regarding the time line of the roll-out. As per @Jdlrobson (from IRC):

jdlrobson: all wikis but en and de will be out within next 2 weeks
jdlrobson: next one monday

While the page summary API is cached really well (>95% of responses are served from Varnish) and the load induced upon RESTBase by Page Previews is not significant, I feel like it would be best for all parties involved if we coordinate the complete roll out a bit better. Namely, the problem is that next week both Wikimania and DebConf are happening, which means very reduced resources on both the Ops and Services teams. In addition, Aug 21st - Aug 25th is the Services team offsite. I am proposing to slightly modify the schedule:

no deploys the week of Aug 7th (next week)
all but the big five go out in the week of Aug 14th
gradually roll out the big five starting from in the week of Aug 28th.

Thoughts? Suggestions? Are there any constraints on the Readers team side that warrant the roll out to happen sooner?

@ovasileva what do you think?

It depends on the level of technical concerns we have. If they are low, I think it makes sense to proceed with the rollouts as planned. Perhaps the next chunk can contain 1-2 or the larger wikis so we can gauge performance. If any red flags occur, we can roll back and examine a more relaxed schedule. My main concern is that we've informed all of our communities about the rollouts (twice) so they are expecting them sooner or later. Also, as we approach the English and German communities during the upcoming A/B test and begin discussions about rollout, it's helpful for us to be able to know that the feature is ready and rolled out on other larger wikis without issues.

I am not worried about the performance side of things. As stated, we have enough capacity in Varnish + RB to support switching all of the wikis right now without suffering performance issues. I am more concerned about the actual content being served/displayed and its correctness. Not all wikis are the same, and it is highly probable that some new corner cases will be discovered, which may warrant assistance from us (purging/content regeneration/etc). And with reduced human power next week (and focus on other things) I wonder if that is a good idea.

That said, if these conditions are acceptable to you (delayed response and action from our side), I will not object or stop you :)

phuedx updated the task description. (Show Details)Aug 4 2017, 10:50 AM

IMO it'd be unnecessarily risky to continue with the deployments with limited support from Services even if it costs us 7 days (see T162672#3498422). We're currently rolled out to 57 wikis, including ruwiki, which we can point to when reaching out to enwiki and dewiki. That we've deployed to those wikis relatively pain-free, doesn't mean we won't need support with the 240 others 🙂

@ovasileva to add QA steps if she's unavailable during a deploy.

For visibility, this will remain in Blocked until Monday, 14th August per T162672#3498422.

testing steps added in description

Jdlrobson moved this task from Blocked on Others to Needs Design Review on the Readers-Web-Kanbanana-Board-Old board.Aug 15 2017, 2:09 PM

We plan to do the rest of the deploy tomorrow.
@phuedx will SWAT to 100 wikis at 1pm UTC and I will SWAT to the final wikis at 6pm.

Change 372137 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] pagePreviews: Deploy to next 100 stage 1 wikis

https://gerrit.wikimedia.org/r/372137

gerritbot added a project: Patch-For-Review.Aug 16 2017, 12:33 PM

I've scheduled ^ for deployment during today's European SWAT.

Change 372137 merged by jenkins-bot:
[operations/mediawiki-config@master] pagePreviews: Deploy to next 100 stage 1 wikis

https://gerrit.wikimedia.org/r/372137

Mentioned in SAL (#wikimedia-operations) [2017-08-16T13:35:05Z] <zfilipin@tin> Synchronized dblists/pp_stage1.dblist: SWAT: [[gerrit:372137|pagePreviews: Deploy to next 100 stage 1 wikis (T162672)]] (duration: 00m 50s)

fgiunchedi subscribed.Aug 16 2017, 2:22 PM

phuedx removed phuedx as the assignee of this task.Aug 16 2017, 3:13 PM

phuedx moved this task from Doing to Blocked on Others on the Readers-Web-Kanbanana-Board-Old board.

phuedx added a subtask: T173422: Investigate the increase in the number of requests to Swift after the Page Previews deploy.Aug 16 2017, 3:28 PM

Change 372160 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Roll page previews out to all wikis except en and de wiki

https://gerrit.wikimedia.org/r/372160

This appears to be blocked on T173422

The remaining wikis with the exception of zh are small, so I think it's safe to roll this out to remaining wikis. Updates on T173422 do not seem alarming right now.

Jdlrobson mentioned this in T173422: Investigate the increase in the number of requests to Swift after the Page Previews deploy.Aug 21 2017, 5:30 PM

Jdlrobson moved this task from Needs Design Review to Doing on the Readers-Web-Kanbanana-Board-Old board.Aug 21 2017, 5:46 PM

Jdlrobson moved this task from Doing to Needs Code Review on the Readers-Web-Kanbanana-Board-Old board.

Change 372160 merged by jenkins-bot:
[operations/mediawiki-config@master] Roll page previews out to all wikis except en and de wiki

https://gerrit.wikimedia.org/r/372160

Mentioned in SAL (#wikimedia-operations) [2017-08-21T18:40:02Z] <niharika29@tin> Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)