Allow page previews to be rolled out to 90% of logged-out users
Closed, ResolvedPublic2 Story Points

Description

Allow page previews to be rolled out to 90% of logged-out users.

Acceptance criteria:

  • page previews is rolled out enabled by default to 90% of logged-out users.
  • this change does not affect logged-in users
  • instrumentation must run for all users
  • instrumentation must be updated to reflect which users are in the bucketed group
There are a very large number of changes, so older changes are hidden. Show Older Changes
Jhernandez edited the task description. (Show Details)Feb 9 2017, 5:30 PM
ovasileva edited the task description. (Show Details)Feb 13 2017, 7:47 PM
MBinder_WMF edited the task description. (Show Details)Feb 13 2017, 7:50 PM
ovasileva edited the task description. (Show Details)Feb 13 2017, 7:52 PM
ovasileva edited the task description. (Show Details)
ovasileva edited the task description. (Show Details)
bmansurov edited the task description. (Show Details)Feb 13 2017, 7:54 PM
ovasileva edited the task description. (Show Details)Feb 13 2017, 8:00 PM
ovasileva edited the task description. (Show Details)
ovasileva edited the task description. (Show Details)Feb 13 2017, 8:02 PM
ovasileva set the point value for this task to 2.Feb 15 2017, 6:14 PM
bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-92-๐Ÿœ board.
bmansurov edited the task description. (Show Details)Feb 16 2017, 10:34 PM

@ovasileva I've updated the description to make things clearer. No need to mention roll out wikis here for example. Can you verify if I correctly updated the description?

Can you also explain what you mean by the following?

  • instrumentation must be updated to reflect which users are in the bucketed group
bmansurov changed the title from "Allow page previews to be rolled out to 90% of users in stage 0 " to "Allow page previews to be rolled out to 90% of logged-out users".Feb 16 2017, 10:46 PM

Change 338293 had a related patch set uploaded (by Bmansurov):
WIP: T157700

https://gerrit.wikimedia.org/r/338293

@bmansurov - in the original schema, we had popupEnabled to determine whether the user has the feature on or off - I wasn't sure if @phuedx removed that when he was working on the new instrumentation, so this was a doublecheck that it's still there

Change 339079 had a related patch set uploaded (by Bmansurov):
Hygiene: do not pollute the mw namespace

https://gerrit.wikimedia.org/r/339079

Change 339080 had a related patch set uploaded (by Bmansurov):
Extract a utility function to a separate testable module

https://gerrit.wikimedia.org/r/339080

@bmansurov: Firstly, sorry I didn't comment earlier in the day. Are there any changes against the Page Previews (Popups) repository?

@phuedx I think I messed up. All the changes are against the RelatedArticles repo. I'll get something in soon.

@bmansurov: Don't rush. Shout out if you want to pair on this (or something else) or drop this and focus on another task, e.g. T158957: Notification count appears misaligned in all mobile web ๐Ÿ™‚

bmansurov added a comment.EditedMon, Feb 27, 6:58 PM

@phuedx the task you linked is ready for review. I'll ping you for a review of this task.

@ovasileva say an anonymous user has disabled and enabled the feature via the settings cog icon. Should those users also be sampled, or just the ones who have never touched the settings cog?

@bmansurov - I think if we track up to the point of disabling, from that point forward it's okay to not sample

Change 340161 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
Allow bucketing anons

https://gerrit.wikimedia.org/r/340161

To the reviewer: Please ignore the previous patches. Only https://gerrit.wikimedia.org/r/340161 needs to be reviewed.

Change 338293 abandoned by Bmansurov:
Allow bucketing anons

Reason:
not needed for now

https://gerrit.wikimedia.org/r/338293

Change 339080 abandoned by Bmansurov:
Extract a utility function to a separate testable module

Reason:
Not needed for now.

https://gerrit.wikimedia.org/r/339080

Change 339079 abandoned by Bmansurov:
Hygiene: do not pollute the mw namespace

Reason:
Not needed for now.

https://gerrit.wikimedia.org/r/339079

@ovasileva: Should this behaviour be present when Page Previews is configured to be a Beta Feature?

There are few very minor improvements to be made to rEPOP3b6f04d0bd81: Allow bucketing anons. @ovasileva's answer to T157700#3060914 doesn't change the size of the work either.

@phuedx - no, it's not necessary for the beta feature

Change 340486 had a related patch set uploaded (by Phuedx):
[mediawiki/extensions/Popups] storage: Fix UserSettings#hasIsEnabled

https://gerrit.wikimedia.org/r/340486

phuedx claimed this task.Wed, Mar 1, 11:29 AM
phuedx added a subscriber: bmansurov.

^ I've reviewed and tested rEPOPf54f92402c7d: storage: Fix UserSettings#hasIsEnabled, which LGTM (notes from my testing follow). However, I spotted a pre-existing bug with userSettings#hasIsEnabled โ€“ which blocks completion of this task IMO โ€“ the fix for which is rEPOPf54f92402c7d: storage: Fix UserSettings#hasIsEnabled.

Change 340486 merged by jenkins-bot:
storage: Fix UserSettings#hasIsEnabled

https://gerrit.wikimedia.org/r/340486

Change 340161 merged by jenkins-bot:
Allow bucketing anons

https://gerrit.wikimedia.org/r/340161

The staging server has been updated and reconfigured to enable Page Previews as a user preference. The opt-in rate for anons has been left at its default value, 90%.

phuedx reassigned this task from phuedx to ovasileva.Wed, Mar 1, 12:29 PM
ovasileva closed this task as "Resolved".EditedWed, Mar 1, 12:32 PM

@phuedx - have we made the update to the schema to know which users are in which group?

ovasileva reopened this task as "Open".Wed, Mar 1, 12:32 PM
phuedx edited the task description. (Show Details)Wed, Mar 1, 12:37 PM
phuedx removed ovasileva as the assignee of this task.

@phuedx - have we made the update to the schema to know which users are in which group?

^ Nope.

@ovasileva to clarify, we already log a value for popupEnabled. What else do we need to change?

@bmansurov - that should be fine, so long as popupEnabled is logged per user (as true for 90% and false for 10%)

phuedx assigned this task to ovasileva.Wed, Mar 1, 3:46 PM
phuedx added a subscriber: ABorbaWMF.

@ovasileva: I think this should be assigned to you or should I hand it to @ABorbaWMF so that he can go through the anon behaviours another time?

ovasileva closed this task as "Resolved".Wed, Mar 1, 4:00 PM
phuedx added a comment.Wed, Mar 1, 5:16 PM

... (notes from my testing follow).

== Testing notes
The following testing notes are true only when the fix is applied.

=== Configured as a Beta Feature
Logged out: The Page Previews code isn't delivered to the client.
Logged in: The Page Previews code is delivered and is enabled only when the Beta Feature is enabled.
When configured as a user preference (and with wgPopupsAnonsEnabledSamplingRate = 0.5 added to LocalSettings.php)

=== Configured as a user preference

Logged out: The Page Previews code is delivered to the client and the feature is enabled ~50% of the time across 10 distinct browser sessions. Moreover, the > feature remains disabled if I disable it via the settings dialog.
Logged in: The Page Preview code is delivered to the client and the feature is enabled when the user preference is enabled.

Tbayer added a subscriber: Tbayer.Thu, Mar 2, 8:28 AM

I'm a bit confused about the current state of the instrumentation. Currently, the Popup schema isn't sending any events at all (event rate dropped to almost 0 since February 23). I haven't been following along in detail here (maybe I should have), but understood from @ovasileva that we should be seeing data at least from Beta features users and from the Swedish chapter wiki. Is that correct? I actually promised @ovasileva to take a first look at the new data before the rollout next week ;)

In any case, please update the information at https://meta.wikimedia.org/wiki/Schema_talk:Popups#Sampling .

@Tbayer, @ovasileva: This is a regression was caused by rEPOP98b2dcf631f1: Hygiene: Rename SchemaPopupsSamplingRate. We caught it (relatively) late because it only rolled out to the cluster on Thursday, February 23rd.

rOMWC9022281fc027: Re-enable Page Previews instrumentation both fixes the regression and sets the sampling rate to 0.05%. I can deploy it this afternoon during the European SWAT. Once it's deployed, https://meta.wikimedia.org/wiki/Schema_talk:Popups#Sampling can be updated.

Mentioned in SAL (#wikimedia-operations) [2017-03-02T14:47:02Z] <phuedx@tin> Synchronized wmf-config/InitialiseSettings.php: T157700: Re-enable Page Previews instrumentation (duration: 00m 40s)

Tbayer added a comment.Mon, Mar 6, 3:35 AM

Thanks @phuedx ! For the record, when the above mentioned InitialiseSettings changes (rOMWC9022281fc0277fbdf33ae94c5192787552965ab9}) were deployed at 14:47 UTC on Thursday, the event rate first went up from (around) 0 to about 70-90 per minute. However, about five hours later it fell again, to currently about 5-30 per minute. Do we know why?

phuedx added a comment.Tue, Mar 7, 2:06 PM

@Tbayer: Yes. Between 8-10 PM UTC the group2 wikis were updated to the -wmf.14 branch. Included in that update was rEPOP938a4b85d44c: Hygiene: Remove checkin instrumentation, which removed all traces of the initial version of the ReadingDepth instrumentation ("checkin") from the codebase. Looking at the difference in the rates, I'm glad that we did!