Allow page previews to be rolled out to 90% of logged-out users.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dereckson | T68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki) | |||
| Resolved | Jdlrobson | T70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature | |||
| Resolved | ovasileva | T136602 Graduate the Page Previews beta feature on stage 0 wikis | |||
| Resolved | ovasileva | T157700 Allow page previews to be rolled out to 90% of logged-out users |
@ovasileva I've updated the description to make things clearer. No need to mention roll out wikis here for example. Can you verify if I correctly updated the description?
Can you also explain what you mean by the following?
@bmansurov - in the original schema, we had popupEnabled to determine whether the user has the feature on or off - I wasn't sure if @phuedx removed that when he was working on the new instrumentation, so this was a doublecheck that it's still there
Change 339079 had a related patch set uploaded (by Bmansurov):
Hygiene: do not pollute the mw namespace
Change 339080 had a related patch set uploaded (by Bmansurov):
Extract a utility function to a separate testable module
@bmansurov: Firstly, sorry I didn't comment earlier in the day. Are there any changes against the Page Previews (Popups) repository?
@phuedx I think I messed up. All the changes are against the RelatedArticles repo. I'll get something in soon.
@bmansurov: Don't rush. Shout out if you want to pair on this (or something else) or drop this and focus on another task, e.g. T158957: Notification count appears misaligned in all mobile web 🙂
@phuedx the task you linked is ready for review. I'll ping you for a review of this task.
@ovasileva say an anonymous user has disabled and enabled the feature via the settings cog icon. Should those users also be sampled, or just the ones who have never touched the settings cog?
@bmansurov - I think if we track up to the point of disabling, from that point forward it's okay to not sample
Change 340161 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
Allow bucketing anons
To the reviewer: Please ignore the previous patches. Only https://gerrit.wikimedia.org/r/340161 needs to be reviewed.
Change 338293 abandoned by Bmansurov:
Allow bucketing anons
Reason:
not needed for now
Change 339080 abandoned by Bmansurov:
Extract a utility function to a separate testable module
Reason:
Not needed for now.
Change 339079 abandoned by Bmansurov:
Hygiene: do not pollute the mw namespace
Reason:
Not needed for now.
@ovasileva: Should this behaviour be present when Page Previews is configured to be a Beta Feature?
There are few very minor improvements to be made to rEPOP3b6f04d0bd81: Allow bucketing anons. @ovasileva's answer to T157700#3060914 doesn't change the size of the work either.
Change 340486 had a related patch set uploaded (by Phuedx):
[mediawiki/extensions/Popups] storage: Fix UserSettings#hasIsEnabled
^ I've reviewed and tested rEPOPf54f92402c7d: storage: Fix UserSettings#hasIsEnabled, which LGTM (notes from my testing follow). However, I spotted a pre-existing bug with userSettings#hasIsEnabled – which blocks completion of this task IMO – the fix for which is rEPOPf54f92402c7d: storage: Fix UserSettings#hasIsEnabled.
The staging server has been updated and reconfigured to enable Page Previews as a user preference. The opt-in rate for anons has been left at its default value, 90%.
@phuedx - have we made the update to the schema to know which users are in which group?
@ovasileva to clarify, we already log a value for popupEnabled. What else do we need to change?
@bmansurov - that should be fine, so long as popupEnabled is logged per user (as true for 90% and false for 10%)
@ovasileva: I think this should be assigned to you or should I hand it to @ABorbaWMF so that he can go through the anon behaviours another time?
Testing notes
The following testing notes are true only when the fix is applied.
Configured as a Beta Feature
Logged out: The Page Previews code isn't delivered to the client.
Logged in: The Page Previews code is delivered and is enabled only when the Beta Feature is enabled.
When configured as a user preference (and with wgPopupsAnonsEnabledSamplingRate = 0.5 added to LocalSettings.php)Configured as a user preference
Logged out: The Page Previews code is delivered to the client and the feature is enabled ~50% of the time across 10 distinct browser sessions. Moreover, the > feature remains disabled if I disable it via the settings dialog.
Logged in: The Page Preview code is delivered to the client and the feature is enabled when the user preference is enabled.
I'm a bit confused about the current state of the instrumentation. Currently, the Popup schema isn't sending any events at all (event rate dropped to almost 0 since February 23). I haven't been following along in detail here (maybe I should have), but understood from @ovasileva that we should be seeing data at least from Beta features users and from the Swedish chapter wiki. Is that correct? I actually promised @ovasileva to take a first look at the new data before the rollout next week ;)
In any case, please update the information at https://meta.wikimedia.org/wiki/Schema_talk:Popups#Sampling .
@Tbayer, @ovasileva: This is a regression was caused by rEPOP98b2dcf631f1: Hygiene: Rename SchemaPopupsSamplingRate. We caught it (relatively) late because it only rolled out to the cluster on Thursday, February 23rd.
rOMWC9022281fc027: Re-enable Page Previews instrumentation both fixes the regression and sets the sampling rate to 0.05%. I can deploy it this afternoon during the European SWAT. Once it's deployed, https://meta.wikimedia.org/wiki/Schema_talk:Popups#Sampling can be updated.
I scheduled rOMWC9022281fc027: Re-enable Page Previews instrumentation to be deployed during today's European SWAT.
Mentioned in SAL (#wikimedia-operations) [2017-03-02T14:47:02Z] <phuedx@tin> Synchronized wmf-config/InitialiseSettings.php: T157700: Re-enable Page Previews instrumentation (duration: 00m 40s)
Thanks @phuedx ! For the record, when the above mentioned InitialiseSettings changes (rOMWC9022281fc027}) were deployed at 14:47 UTC on Thursday, the event rate first went up from (around) 0 to about 70-90 per minute. However, about five hours later it fell again, to currently about 5-30 per minute. Do we know why?
@Tbayer: Yes. Between 8-10 PM UTC the group2 wikis were updated to the -wmf.14 branch. Included in that update was rEPOP938a4b85d44c: Hygiene: Remove checkin instrumentation, which removed all traces of the initial version of the ReadingDepth instrumentation ("checkin") from the codebase. Looking at the difference in the rates, I'm glad that we did!