Page MenuHomePhabricator

Deploy Add a link to the second set of wikis
Open, MediumPublic

Description

Add a link have been tested on our pilot wikis (deployment task: T278710), and now, it is ready to go to more wikis.

We will go with a shortlist of wikis, to have an opportunity to monitor these wikis closely, before deploying to all wikis. The wikis we select have to be rather active wikis.

Wikipedias to work with:

Event Timeline

Can you add a link to Add a link (haha) so external watchers can get the context?

@Trizek-WMF -- in thinking this through, I would also like to prioritize Spanish, Portuguese, and Turkish, since those will be important wikis in our pilot work next year. What do you think about removing Romanian and adding those? Or just adding those on top of your list (it's probably fine to be 9 instead of 6).

MMiller_WMF renamed this task from Deploy Add a link to 5 wikis to Deploy Add a link to the second set of wikis.Mon, Jun 7, 7:47 PM
MMiller_WMF added subscribers: kostajh, Tgr, Urbanecm_WMF.

Can you add a link to Add a link (haha) so external watchers can get the context?

Thank you for asking me to add the add a link link. :p
https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_a_link

(But Kosta added the link to add a link first, in the task description.)

@Trizek-WMF -- in thinking this through, I would also like to prioritize Spanish, Portuguese, and Turkish, since those will be important wikis in our pilot work next year. What do you think about removing Romanian and adding those? Or just adding those on top of your list (it's probably fine to be 9 instead of 6).

@MMiller_WMF, your initial request is to find active wikis. I picked wikis that are both active by the number of edits being made, but which are also active in a matter of community engagement. We can have more wikis, or shorten the list. It is up to the kind of engagement you're looking for.

@Trizek-WMF we discussed this, and decided to add Spanish, and leave Romanian on the list.

@kostajh @Tgr -- we've decided which wikis we want in the second set to deploy "add a link". This is Ready for Development in terms of populating the indices, but not yet for deployment. We will decide on whether/when to actually deploy the feature to users based on forthcoming data from the first set of wikis.

Where can I add the "inform communities" phase in the checkboxes defined in the task description? :)

Where can I add the "inform communities" phase in the checkboxes defined in the task description? :)

Good question. Inform them that it is coming or that we are deploying on a specific date? It would be nice if we didn't have to specify the exact day / time it is happening, but we can do that if you'd like.

@MMiller_WMF per T277356: Add a link: experiment, should this expansion follow the same pattern where we are opting 50% into the structured task for add link and 50% get the old links task, or are we switching everyone (pilot wikis plus this batch) into the structured task for add link?

How is it deployed? If it is a simple configuration change, like at other wikis, I guess we can do it anytime?

It would be nice to have a way for experienced users to try the features. It would be a way to check if we have new cases of links that should be rejected.

How is it deployed? If it is a simple configuration change, like at other wikis, I guess we can do it anytime?

It would be nice to have a way for experienced users to try the features. It would be a way to check if we have new cases of links that should be rejected.

It could be enabled like we did for the pilot wikis, where 0% of new accounts were opted into the link-recommendation group, but users could use a browser console snippet to opt-in to the link-recommendation group.

It could be enabled like we did for the pilot wikis, where 0% of new accounts were opted into the link-recommendation group, but users could use a browser console snippet to opt-in to the link-recommendation group.

Great!

Can we think of an URL parameter instead of a b console snippet?

  • Trained the models for each wiki on stat1008 (data should be available in the staging-db)
  • Backtesting evaluation does not raise any flags about performance of the model in these wikis; similar to other wikis, i.e. for default threshold 0.5 we get precision ~80% with recall ~40% (or better). I added the detailed results for this round of training/evaluation to the project-page on meta.

After looking at the data in T277355: Add a link: leading indicators, we have decided that this feature is strong enough to go to these additional wikis. The revert rate is low, and user enthusiasm is high. We should continue to proceed with preparing this deployment.

kostajh updated the task description. (Show Details)

Where can I add the "inform communities" phase in the checkboxes defined in the task description? :)

Good question. Inform them that it is coming or that we are deploying on a specific date? It would be nice if we didn't have to specify the exact day / time it is happening, but we can do that if you'd like.

@MMiller_WMF per T277356: Add a link: experiment, should this expansion follow the same pattern where we are opting 50% into the structured task for add link and 50% get the old links task, or are we switching everyone (pilot wikis plus this batch) into the structured task for add link?

@MMiller_WMF I believe you said that you'd like to roll this out like we did for the pilot wikis: 50% of Growth feature newcomers get link-recommendation structured task, 50% get unstructured links task; all older newcomer users are opted into link-recommendation structured task. Is that correct?

@kostajh -- yes, I think we should deploy with the A/B test like we did in our pilot wikis, where half of the users get the structured task and the other half get the unstructured task. @nettrom_WMF, do you agree with that? Would having that additional data from these larger wikis help us see the differences that "add a link" makes?

Also @kostajh -- when is the earliest we would be ready to turn this on? Not because we're in a huge rush, but just so we know what dates we have to choose from?

@Trizek-WMF -- if you think that we should hold off deploying to Spanish because of the Fundraising test, that's fine. Is it because that will be too many changes to communicate to the community at once (I know you're also communicating about mentors)? If you want to hold off, how soon after the Fundraising test would you want to deploy it?

Also,@Trizek-WMF -- for the other wikis on the list (Russian, French, etc.) how much lead time do you want for communication before we deploy?

@kostajh -- yes, I think we should deploy with the A/B test like we did in our pilot wikis, where half of the users get the structured task and the other half get the unstructured task. @nettrom_WMF, do you agree with that? Would having that additional data from these larger wikis help us see the differences that "add a link" makes?

@MMiller_WMF : Yes, I agree. The way I read our measurement plan is that we'll extend the A/B test to the additional wikis so they'll have the same 50/50 split between Add a Link and the unstructured task as the four initial wikis currently do. That extension is what enables us to run a shorter experiment to get enough data to assess the difference between the two types of tasks.

kostajh changed the task status from Stalled to Open.Thu, Jun 24, 9:39 AM
kostajh claimed this task.
kostajh updated the task description. (Show Details)

Change 701351 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation feature

https://gerrit.wikimedia.org/r/701351

Change 701351 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation feature for more wikis

https://gerrit.wikimedia.org/r/701351

Mentioned in SAL (#wikimedia-operations) [2021-06-24T11:10:59Z] <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:701351|GrowthExperiments: Enable link recommendation feature for more wikis (T284481)]] (duration: 01m 07s)

@Trizek-WMF -- if you think that we should hold off deploying to Spanish because of the Fundraising test, that's fine. Is it because that will be too many changes to communicate to the community at once (I know you're also communicating about mentors)? If you want to hold off, how soon after the Fundraising test would you want to deploy it?

I think we should not deploy two things at the same time. So I would differ the deployment at Spanish Wikipedia to after the Fundraising test.

Also,@Trizek-WMF -- for the other wikis on the list (Russian, French, etc.) how much lead time do you want for communication before we deploy?

I'd prefer to have two full weeks after the announcement, these two weeks would be used to let communities to test the features. So, regarding the A/B testing, we need to have a way to bypass it for experienced users who want to try the feature.

And we also need to T285467: Create an help page for Add a link, distinct from the project page. This page would show how the step by step feature will work, so that users who are not sure about using it, or experienced users, can have an overview of the feature.

@Trizek-WMF we enabled the feature in hidden mode for the wikis listed in this task today. No new users will see the features yet. Current users can use these browser console snippets to opt into the experiment group to see the feature and to opt back out of it again:

ge.utils.setUserVariant('linkrecommendation') // enable link recommendation task on Special:Homepage
ge.utils.setUserVariant('control') // opt out of the experiment

I would wait at least until tomorrow before letting these communities know, because filling up the task pools for these wikis will take some time. The charts at the bottom of https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?orgId=1 will provide data when we begin to have some link recommendation tasks available for testing in those wikis.

By the way, @Trizek-WMF & @MMiller_WMF, we have already generated datasets and models for a few other communities during earlier phases of development of Add-Link, and we are in the process of updating those datasets in the production service now. Those wikis are: elwiki, dewiki, simplewiki, and ptwiki. Ptwiki is probably part of the donor project as well, but please let us know if you'd like the link recommendation task to be enabled for any of those other three wikis as part of this second round of deployments.