Page MenuHomePhabricator

Deploy Help Panel to Basque Wikipedia
Closed, ResolvedPublic

Description

We are ready to deploy the help panel to Basque Wikipedia, using the configurations in T223619. It should be deployed to all new users -- not in an A/B test.

  • Configure the deployment on Basque Wikipedia
  • Define a date for the deployment with @Theklan
  • Inform communities about it (@Theklan)
  • Push to production
  • Validate that incoming data is correct shortly after deployment (@nettrom_WMF)
  • Look at leading indicators after about a month of usage (@nettrom_WMF)

Event Timeline

We plan to deploy this on Thursday @Theklan. Please remember that this will be deployed as A/B test, meaning half of newcomers will receive and half will not. You will, however, be able to show newcomers how to turn it on for themselves in their Preferences at an event. Does this sound okay?

It will be quite strange, but yes, scientifically is better to have an A/B test. How long will it take the A/B test?

@Theklan -- good question. Let me give you some more information about how we're planning our tests.

  • The first help panel experiment began in January 2019, and we've recently finished analyzing it. It needed about six months to gather enough data. But we're planning new improvements to the help panel based on those learnings, so we want to continue into another experiment.
  • Meanwhile, the homepage experiment began in May 2019, and we'll be analyzing the results next month (October).
  • The main project our team is working on is newcomer tasks, in which the homepage and help panel will need to work together to help newcomers complete suggested edits. When this experiment starts, we'll unify the homepage and help panel experiment groups -- so half of users will have both, and half will have neither. This will begin in October.
  • Taking all this together, we'll want to renew the experiments for about six more months, starting in October, and then decide whether any of the features are impactful enough to be deployed to all users.

By running the experiments in Basque Wikipedia, data from your wiki will contribute to the pool, allowing us to have results a little faster, and have more confidence in them. But given that (I think) many of the new accounts in your wiki come through programs, you may end up finding the experiments disruptive to your work. If that happens, let us know, and we'll talk about whether it makes sense to deploy to all users on your wiki.

How does that sound?

@Theklan -- our team talked about this some more, and we've changed our minds! We now think we should deploy to all Basque newcomers, and not do an A/B test. The reason is that (1) Basque Wikipedia gets few enough new accounts that they won't make a big difference in our experiment analysis (2) since you plan to run in-person programs, this will make things smoother, since all newcomers will have the feature by default.

I hope this sounds good to you. Let me know if you have any questions!

Change 537804 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[operations/mediawiki-config@master] GrowthExperiments: Enable help panel for euwiki

https://gerrit.wikimedia.org/r/537804

Change 537804 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Enable help panel for euwiki

https://gerrit.wikimedia.org/r/537804

Mentioned in SAL (#wikimedia-operations) [2019-09-19T11:23:16Z] <urbanecm@deploy1001> Synchronized wmf-config/VariantSettings.php: SWAT: eab7c6a: c80f026: GrowthExperiments: GrowthExperiments: Enable Special:Homepage for euwiki, GrowthExperiments: Enable help panel for euwiki (T233066, T233065) (duration: 01m 05s)

Checked Help panel on euwiki wmf.23

nettrom_WMF added a subscriber: Mayakp.wiki.

@Mayakp.wiki has led the QA work on this, thanks for helping out!

Data is flowing in and users are opening the Help Panel, but so far there are few uses and no interactions with the panel. I'm therefore keeping this open and moving it to "Blocked" on the PA board for now, to be revisited later when we have more users.

@Theklan -- would you be willing to turn on the help panel for your own account (or make a new account), and try out all its capabilities? This will help validate that the links are going to the right places, the questions get posted to the help desk, and the search results make sense. It will also cause some data to flow into the database to make sure the data is working right.

As far as I see, everything seems correct.

We've not seen any indication that there are data quality issues after our deployment on Basque Wikipedia, and we do not have the capacity to prioritize this work to dig in further. Closing this task as resolved since the Help Panel has been deployed.