Page MenuHomePhabricator

Evaluate Wikipedia Library notification progress [4hr]
Closed, ResolvedPublicSpike

Description

We would like to implement a notice for editors when they become eligible for The Wikipedia Library. Until now we have relied on word of mouth, blog posts, and other ad-hoc notices. The notification will allow us to reach 100% of our potential user base and notify newly eligible editors as they become eligible. This should substantially increase the number of users of the library and ensure that it's well known throughout the community.

Work on a MediaWiki Notification is tracked at T132084. This started in 2016 and was lightly worked on here and there by the Growth-Team into 2018 as Extension:TheWikipediaLibrary. The notification was deployed to the Beta cluster in 2019. Since then progress has stalled.

More recently we received notice that tests were failing for the notification (T256403), and that some users were being spammed by it (T256297). In response to the latter, the extension was disabled on Beta Wiki for the time being. PreferencesFactory::setUser() was identified (T254988) as the cause of failing tests.

We're going to migrate responsibility for this notice into The-Wikipedia-Library, finishing it off so that it can be deployed after we finish our FY20/21 design work and search integration. Before then, we need to evaluate the current state of development and understand the steps needed to progress.

Questions to answer

  • What outstanding issues need to be resolved before this notification could be deployed?
    • Is T254988 resolved?
    • What is the cause of T256297 and does it need work?
  • How long do we estimate this work would take to finalise?
  • Per T132084#6145961, is it possible to make this a staggered rollout, to avoid notifying up to 25,000 editors in a single day?
  • What support do we need from the previous developers to get up to speed in working on this?

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptDec 10 2020, 12:49 PM
Restricted Application added a subscriber: Sadads. · View Herald Transcript
Samwalton9 updated the task description. (Show Details)
Samwalton9 updated the task description. (Show Details)
Samwalton9 renamed this task from Evaluate Wikipedia Library notification progress to Evaluate Wikipedia Library notification progress [4hr].Dec 10 2020, 1:09 PM

Initial results:

  • What outstanding issues need to be resolved before this notification could be deployed?
    • Is T254988 resolved?
      • yes
    • What is the cause of T256297 and does it need work?
      • I was not able to reproduce the issue locally, so I need to look at the differences between the beta deployment and the dev environment. The extension uses a global pref to flag whether the user has already received a notification. The check for that flag is written in such a way that if it isn't retrievable, the user is treated as if they have not been notified. This would result in a user getting notified with every edit, just as reported in the issue. That guard clause needs to be flipped so that we fail without notification. My guess is that there is something about the global preferences setup on beta that isn't accounted for in the code.
  • Additionally, I discovered that this extension has no tests of its own. We should really remedy that since this will be run on every page load of every user. It will also make debugging much easier.
  • Per T132084#6145961, is it possible to make this a staggered rollout, to avoid notifying up to 25,000 editors in a single day?
    • We could just change the existing setting for the editcount threshold, starting high and dropping it over time by deploying updates to the mediawiki configs. If that would have to be done per-project, it might be better to do it in code. This would be really straightforward to implement. The function that decides whether to send the notification could also check the editcount and the date to sort editors into notification stages. We could do something like:
      • week 1 - editors with > 10k edits
      • week 2 - editors with > 4k edits
      • week 3 - editors with > 1k edits
      • week 4 - editors with > 500 edits
    • We'll need to do some analysis to figure out about how many editors is in each of those buckets to determine the actual thresholds, and of course decide what kind of timeline we want for the rollout.
  • What support do we need from the previous developers to get up to speed in working on this?
    • If there are examples of other extensions that exist solely to send notifications, I'd love to have that pointed out to me so I can take a look at them. Otherwise, I don't think we need much. This is a pretty straightforward extension.
  • How long do we estimate this work would take to finalise?
    • I don't have that answer yet, because I haven't answered all my own questions yet.
jsn.sherman moved this task from In Progress to Review on the The-Wikipedia-Library (Kanban) board.

Moving to review, as I've spent my allotted time on this, and there is still more research to do, specifically for T256297. We need to learn more about global preferences in the beta environment.

Another take on the staggered rollout: We can roll the extension out to smaller wikis and then to big wikis like English, Swedish, and German. We can do this in addition to the editor notification stages to observe how the Wikipedia Library behaves with the new volume of users.

This is a great first pass, thanks Jason.

If there are examples of other extensions that exist solely to send notifications, I'd love to have that pointed out to me so I can take a look at them.

Some examples that might be helpful, mostly more complex than this:

We could just change the existing setting for the editcount threshold, starting high and dropping it over time by deploying updates to the mediawiki configs. If that would have to be done per-project, it might be better to do it in code. This would be really straightforward to implement. The function that decides whether to send the notification could also check the editcount and the date to sort editors into notification stages.

We can roll the extension out to smaller wikis and then to big wikis like English, Swedish, and German. We can do this in addition to the editor notification stages to observe how the Wikipedia Library behaves with the new volume of users.

My gut feeling is that the edit count threshold might be more achievable since this is a global notification counting edits across all Wikimedia projects, but I could be wrong.

We'll need to do some analysis to figure out about how many editors is in each of those buckets to determine the actual thresholds, and of course decide what kind of timeline we want for the rollout.

Per modifications on https://quarry.wmflabs.org/query/42517, here's the breakdown for English Wikipedia (only taking into account edits to that project), which I assume is at least approximately representative of all users:

Edit countNumber of editors% of totalCumulative total
> 10k edits3,99713%13%
> 4k edits6,07819%32%
> 1k edits9,66431%63%
> 500 edits11,44437%100%

Since we don't currently have a clear figure for how many users we're going to notify (only the % is useful in that table) I think this kind of staggered rollout, starting with the most active, makes a lot of sense.

This is back in Ready for Dev with the following outstanding items:

  • Decide on rollout strategy - does the edit count breakdown make sense? Is it achievable?
    • A secondary note here that occurred to me. "dropping it over time by deploying updates to the mediawiki configs"; mediawiki deployments happen weekly so we'd be tied to the weekly deployment train, is that right? Or is there a way to deploy those simple config changes on a more customised timeline?
  • Diagnose the likely cause of T256297 a little further.
  • Was T256403 fixed in T254988?

Was T256403 fixed in T254988?

Yes; sorry my initial response wasn't clear on that.

  • Diagnose the likely cause of T256297 a little further.

I researched other extensions that use GlobalPreferences, and found that they all do varying amounts of setup to check the status of GlobalPreferences and to manage preference caching. Caching seems the most likely difference between beta and local dev. Given the way our guard clause is written, a cached preference value that doesn't show someone already received the notification would result in exactly the reported bug.
I found a nice helper class that we can largely crib for our extension to take care of this for us:
https://github.com/wikimedia/mediawiki-extensions-ContentTranslation/blob/ebe1f3576b5edd3439bc91adb07fe5d7d9c8eaea/includes/PreferenceHelper.php

I think we could update our global pref handling and re-enable the extension on beta to see if things work correctly. If it's still broken, it will not send any notifications.

Decide on rollout strategy - does the edit count breakdown make sense? Is it achievable?

What I like about this breakdown is that the stages start small and gets bigger over time. For me the biggest question is a week enough time between stages? We might want to make the delay between stages longer so that if we're overwhelmed we have time to get a config change deployed to pause further stages.

A secondary note here that occurred to me. "dropping it over time by deploying updates to the mediawiki configs"; mediawiki deployments happen weekly so we'd be tied to the weekly deployment train, is that right? Or is there a way to deploy those simple config changes on a more customised timeline?

The only deploys that don't follow the weekly schedule are backports (not us) and incident response (which we hope to avoid). The simplest way that I can see to customize our timeline is to just do it in code by having the required editcount change based on date and checking the date on pageload. We could either remove all config for this and just hardcode the values, or we could make the dates and related counts config as well. That's probably the right way to do it.

Based on this spike I'm seeing the following work as outstanding:

  • T271917: Update the text and link destinations per my update to T132084
  • Fix T256297 by improving the guard clause in the case that global preferences can't be read.
  • T271919: Write some tests
  • T271921: Implement a gradual rollout process based on global edit counts, probably through config changes made manually by us as we're comfortable
  • T271922: Request a performance review prior to deployment

Another consideration that came up this week is since a large number of users are going to receive this notification at deployment, we want to make sure it's translated into as many languages as possible. Since the code was already merged for Beta, it's been live on TranslateWiki for some time, with a substantial number of translations completed. Since we're rewording the text, however, these will mostly get invalidated. We'll want to wait until we're happy the updated text has been sufficiently translated before beginning our rollout. It would make sense, therefore, for updating the text/links to be the first task.

We also discussed that config changes can be deployed separately to the deployment train, and are therefore more flexible. This means we could more manually control our rollout strategy over time.

EDIT: Tasks filed, linked above next to the bullet points.

Something that came up in a conversation with @Prtksxna today - can we get data on how many users receive/see/click the notification and break that down by things like project?

We think we've addressed the issues raised in this Spike - any additional work is going to be based on new observation.