@Urbanecm_WMF -- this is ready for you to think about, with a couple questions in the task description. Let's discuss!

Quoting from description

• Should we only apply this to mentorship? Or while we're at it, should we also make it possible for suggested edits or the impact module or other parts of the Growth features?

I think mentorship is the only remaining main feature group that cannot be customized to meet per wiki needs. It's possible (at least with some caveats) for all other features you mention: for example, it's possible to disable impact module per wiki (we did that for bnwiki I think, where CivilServant or someone conducted their own research). While we can't set a percentage for the impact module, I can't figure out a situation where that would make sense.

Fortunately, those kind of changes aren't very difficult, and they can be easily worked on as the need occurs. I think we should do mentorship only for now, and use the same solution for other modules, should the need arise.

What would make sense is invest in consolidating the logic used for variants. Right now, we have a set of possibilites to use when A/B testing:

Homepage variants

Each user needs to be in one of the homepage variants. Right now, the system recognizes two of them: control, and linkrecommendation. Previously, other variants were recognized (namely variants A, B, C and D). This solution has some advantages, mainly because it's a semi-standard place to store variants, however, it has many issues that would need to be fixed if it should be used as the only solution for A/B testing.

The main issue with homepage variants is that each user needs to be in exactly one homepage variant -- it wouldn't be possible to put an user both into "nomentorship" and "linkrecommendation", for instance.

Ad hoc user preferences

The other solution we use is ad hoc user preferences that are pre-populated for the newcomers. For instance, there's a preference that controls help panel enrollment, or a preference that enrolls an user to the homepage.

This solution is very flexible, but it has other issues: mainly, it's not centralized -- most of the code works with the preferences directly, and there's no abstraction. With more and more ad hoc preferences, the code becomes more and more messy and harder to maintain. It's also not easy to say which variants play a role where.

Ideal solution

The ideal solution would be to create abstraction for A/B testing. It would allow us to "flag" users to ensure certain behavior, allowing more flags at once. If an user had flags "nomentorship" and "linkrecommendation", they won't be shown mentorship module, but would be shown the structured add a link task instead of the old one.

@Tgr mentioned this a couple of times before, and I think he has his own thoughts about this as well.

• I think for analysis purposes, we'll want to keep track of which users ended up with which mix of features. Will we be able to do that using the same paths that @nettrom_WMF usually uses to tell who is in which variants and treatments?

That depends on the implementation route we decide to use. We can create a new homepage variant for this, which would then show in the same place as it shows now (I believe it shows as event.user_variant in the homepagemodule event). The issue would be that user is only in one variant at a time -- turning add a link on at enwiki would be next to impossible with this change.

I think creating an ad hoc user preference would be better (until we have a centralized system, as mentioned above). That would naturally show in user_properties database table – for ease of access, we can add it to WikimediaEvents allowlist, so it shows in prefupdate events (this would also allow us to see history of the property, if needed).

Not sure which needs @nettrom_WMF has -- if there are any specific requirements, please add them here!

Additional open questions

I think there's one more open question that was not yet mentioned or answered -- what to do once mentorship scaling issues were resolved? As far as I understand this, we want to take away mentorship from some of the users to be able to deploy suggested edits (which scale well) easily, while thinking about how to scale mentorship. This implies we will want to give mentorship to everyone at some point -- including the past users.

Is that correct understanding? If so, how would we do that? By manually deleting the preference disabling mentorship from database? Sounds a bit scary, but as it's not something that will happen every day, maybe acceptable?

In T287903#7253608, @Urbanecm_WMF wrote:

@Tgr mentioned this a couple of times before, and I think he has his own thoughts about this as well.

I wrote it up in T288022: Support multidimensional user variants in GrowthExperiments.

Change 710667 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Decouple mentorship from the rest of the features

https://gerrit.wikimedia.org/r/710667

Change 710667 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Decouple mentorship from the rest of the features

https://gerrit.wikimedia.org/r/710667

@nettrom_WMF FYI, once this will be deployed to production, users who will not have mentorship module will have a preference called growthexperiments-homepage-mentorship-enabled with 0 as the value.

Maybe we should also add an array field to homepagevisit, that will tell you which modules the user can see/use? That would be probably easier to access for you than user_properties DB table, and it will be also tied with the particular visit events.

Let me know if you wish any instrumentation changes to be implemented.

@Urbanecm_WMF -- thanks for writing up your thoughts in this task. I think I agree that we should not use homepage variants for this, because it is quite possible that we will want to independently control both whether someone has "add a link" and whether they have mentorship.

As far as I understand this, we want to take away mentorship from some of the users to be able to deploy suggested edits (which scale well) easily, while thinking about how to scale mentorship. This implies we will want to give mentorship to everyone at some point -- including the past users.

For the first step, we will never need to take away mentorship from users -- all the users who have it from before we change the decoupling will get to keep it. And then when we want to scale up, it could be good to turn mentorship on for the old users. But I think that is more about being organized than about benefits to the users. The reason is because if a user was not receiving mentorship, and then we want to turn it on for them, say, six months later, they are probably in one of these groups:

1. 95% of users: no longer on the wiki, and therefore doesn't need mentorship.
2. 5% of users: still on the wiki, but now has six months of experience, and probably does not need the mentorship module as much.

Therefore, I think it would probably be okay to skip retroactively turning on mentorship for this group, but we should discuss it again when it's time.

I'm going to start talking now to the English community about doing the decoupling, and by the time we're ready to do it there, I think this code will also be ready, right?

In T287903#7282603, @MMiller_WMF wrote:

@Urbanecm_WMF -- thanks for writing up your thoughts in this task. I think I agree that we should not use homepage variants for this, because it is quite possible that we will want to independently control both whether someone has "add a link" and whether they have mentorship.

As far as I understand this, we want to take away mentorship from some of the users to be able to deploy suggested edits (which scale well) easily, while thinking about how to scale mentorship. This implies we will want to give mentorship to everyone at some point -- including the past users.

For the first step, we will never need to take away mentorship from users -- all the users who have it from before we change the decoupling will get to keep it. And then when we want to scale up, it could be good to turn mentorship on for the old users. But I think that is more about being organized than about benefits to the users. The reason is because if a user was not receiving mentorship, and then we want to turn it on for them, say, six months later, they are probably in one of these groups:

1. 95% of users: no longer on the wiki, and therefore doesn't need mentorship.
2. 5% of users: still on the wiki, but now has six months of experience, and probably does not need the mentorship module as much.

Therefore, I think it would probably be okay to skip retroactively turning on mentorship for this group, but we should discuss it again when it's time.

Okay, that makes sense. We should be able to enable it if we want to.

I'm going to start talking now to the English community about doing the decoupling, and by the time we're ready to do it there, I think this code will also be ready, right?

Yes. The code is deployed to the wikis now, and we can do the config change whenever you want.

@Etonkovidova This cannot be directly QA'ed (yet), because we didn't set the percentage to something else than 100% anywhere so far. However, in theory, it could hide mentorship module if the patch was wrong -- maybe that should be tested.

@Urabnecm - may be it's worth to set the percentage for growthexperiments-homepage-mentorship-enabled on beta cswiki?

In T287903#7296289, @Etonkovidova wrote:

@Urabnecm - may be it's worth to set the percentage for growthexperiments-homepage-mentorship-enabled on beta cswiki?

If you want, I can set the percentage for beta enwiki. That makes more sense to me, as that's the wiki it will be used for in production. What do you think?

In T287903#7296450, @Urbanecm_WMF wrote:

In T287903#7296289, @Etonkovidova wrote:

@Urabnecm - may be it's worth to set the percentage for growthexperiments-homepage-mentorship-enabled on beta cswiki?

If you want, I can set the percentage for beta enwiki. That makes more sense to me, as that's the wiki it will be used for in production. What do you think?

Yes, it'd be helpful. And I agree - doing it for enwiki betalabs actually makes more sense.

Change 714144 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] [labs] enwiki: Enable mentorship for 10% of users only

https://gerrit.wikimedia.org/r/714144

Change 714144 merged by jenkins-bot:

[operations/mediawiki-config@master] [labs] enwiki: Enable mentorship for 10% of users only

https://gerrit.wikimedia.org/r/714144

In T287903#7298269, @Etonkovidova wrote:

In T287903#7296450, @Urbanecm_WMF wrote:

In T287903#7296289, @Etonkovidova wrote:

@Urabnecm - may be it's worth to set the percentage for growthexperiments-homepage-mentorship-enabled on beta cswiki?

If you want, I can set the percentage for beta enwiki. That makes more sense to me, as that's the wiki it will be used for in production. What do you think?

Yes, it'd be helpful. And I agree - doing it for enwiki betalabs actually makes more sense.

Here you go! My apologies for the confusing commit message, it's actually 8%.

urbanecm@deployment-deploy01:~$ mwscript shell.php enwiki
Psy Shell v0.10.5 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+icu63 — cli) by Justin Hileman
>>> $wgGEHomepageNewAccountEnablePercentage
=> 25
>>> $wgGEMentorshipNewAccountEnablePercentage
=> 8
>>> ^D
Exit:  Ctrl+D
urbanecm@deployment-deploy01:~$ mwscript shell.php cswiki
Psy Shell v0.10.5 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+icu63 — cli) by Justin Hileman
>>> $wgGEHomepageNewAccountEnablePercentage
=> 80
>>> $wgGEMentorshipNewAccountEnablePercentage
=> 100
>>> ^D
Exit:  Ctrl+D
urbanecm@deployment-deploy01:~$

Note that wgGEMentorshipNewAccountEnablePercentage is only considered after wgGEHomepageNewAccountEnablePercentage decides homepage itself is enabled. In another words. for beta enwiki, 25% of new accounts should get homepage, and 8% of those accounts should also get mentorship (so 2% accounts in total). For beta cswiki, 80% of new accounts should get homepage, and 100% of them also mentorship (so 80% of accounts in total).

Hope this helps!

Thanks, @Urbanecm_WMF! Checked on enwiki betalabs - all is working as expected.

In short, what is the publicly visible impact of this? I'm asking for the newsletter and the documentation.

In T287903#7352562, @Trizek-WMF wrote:

In short, what is the publicly visible impact of this? I'm asking for the newsletter and the documentation.

Nothing really "publicly visible". This lets us (Growth team) to control more aspects of Growth features deployment. Precisely speaking, this allows us to deploy the mentorship module only to a certain subgroup of homepage users. Once T290927 gets deployed, English Wikipedia will have the following settings:

• 25% of newly registered accounts get the homepage, and:
• 20% out of those accounts also get the mentorship module

(out of 100 accounts, 20 accounts will get homepage with no mentor module, and 5 accounts will get homepage with mentor module enabled)

Does that make sense?

Makes total sense, thank you @Urbanecm_WMF. I wanted to check if we needed to document it more broadly. But since it is an internal config, then I'll stop here! :)

In T287903#7352708, @Trizek-WMF wrote:

Makes total sense, thank you @Urbanecm_WMF. I wanted to check if we needed to document it more broadly. But since it is an internal config, then I'll stop here! :)

No problem! Yes, it should affect enwiki only (unless I have a bug in the code :)). Default is still "100% of Growth users have access to the mentor module".