Does group mean MediaWiki user group, or GrowthExperiments user variant?

In T329276#8626657, @Tgr wrote:

Does group mean MediaWiki user group, or GrowthExperiments user variant?

User variant; I updated the title and description.

Related: T332348: Clean up Add Link / Add Image A/B test logic in GrowthExperiments

Change 907859 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Section images: Add feature flag and user variant

https://gerrit.wikimedia.org/r/907859

Change 913965 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] [noop] Disable section image recommendations in production

https://gerrit.wikimedia.org/r/913965

Change 913965 merged by Gergő Tisza:

[operations/mediawiki-config@master] [noop] Disable section image recommendations in production

https://gerrit.wikimedia.org/r/913965

Mentioned in SAL (#wikimedia-operations) [2023-05-02T07:30:39Z] <tgr@deploy1002> Started scap: Backport for [[gerrit:913965|[noop] Disable section image recommendations in production (T329276)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-02T07:32:08Z] <tgr@deploy1002> tgr: Backport for [[gerrit:913965|[noop] Disable section image recommendations in production (T329276)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-05-02T07:38:08Z] <tgr@deploy1002> Finished scap: Backport for [[gerrit:913965|[noop] Disable section image recommendations in production (T329276)]] (duration: 07m 29s)

@KStoller-WMF should users in the experiment variant have only the section level image task type set by default?

Change 907859 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Section images: Add feature flag and user variant

https://gerrit.wikimedia.org/r/907859

Let's chat more about this tomorrow with the full team, but I think we should consider simply looking at leading indicators closely - T332344: Section-Level Images: leading indicators, but not conduct an A/B experiment. This opinion is based on the following:

• This task is a variation of the article-level "add an image" task. I think it's more important to wrap up that analysis (T311531) rather than start another experiment.
• We don't want this to be the first task we suggest to newcomers, but if it's a task newcomers "level up" to, then it will be difficult to gather enough data to come to any significant conclusions in an A/B test.
• Based on annual plan priorities, it's unlikely we will have much capacity next fiscal year to iterate and react to experiment results.

Per Slack discussion, we should revert these:

• Hidden preference exists for users to opt-in / opt-out to the feature
• New experiment variant (sectionlevelimages) is defined as a valid variant

and only keep the configuration variable.

Change 924597 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Section images: Remove user variant

https://gerrit.wikimedia.org/r/924597

In T329276#8883553, @Tgr wrote:

Per Slack discussion, we should revert these:

• Hidden preference exists for users to opt-in / opt-out to the feature
• New experiment variant (sectionlevelimages) is defined as a valid variant

and only keep the configuration variable.

@kostajh pointed out that they are useful for testing before releasing so better keep them for now. We can just remove them when we roll the feature out to all users on the pilot wikis.

From today's meeting: we need to discuss the code changes we'll make when we're ready to roll this feature out to all users.

We can't mass-set the new variant without interfering with the NewImpact A/B test. We could backport 924597 as a way of deployment - not elegant but I'm not sure we have a better way. (We could introduce another config flag but that would be a lot of effort for throwaway code.)

In T329276#8910932, @Tgr wrote:

We can't mass-set the new variant without interfering with the NewImpact A/B test. We could backport 924597 as a way of deployment - not elegant but I'm not sure we have a better way. (We could introduce another config flag but that would be a lot of effort for throwaway code.)

Yeah, I think backporting 924597 could work, if a bit cumbersome.

Alternatively, we could just say that ambassadors should do any pre-launch testing on beta wikis. IMO, this is reasonable as:

1. this isn't going to be a default task type; I suspect relatively few people would discover and interact with it before ambassadors have time to do their own reviews
2. if ambassadors find issues, they can switch off the task type via community configuration
3. beta isn't a great approximation of production, but in this case it should be close enough to inspire confidence for a release

cc @KStoller-WMF

In T329276#8913794, @kostajh wrote:

In T329276#8910932, @Tgr wrote:

We can't mass-set the new variant without interfering with the NewImpact A/B test. We could backport 924597 as a way of deployment - not elegant but I'm not sure we have a better way. (We could introduce another config flag but that would be a lot of effort for throwaway code.)

Yeah, I think backporting 924597 could work, if a bit cumbersome.

Alternatively, we could just say that ambassadors should do any pre-launch testing on beta wikis. IMO, this is reasonable as:

1. this isn't going to be a default task type; I suspect relatively few people would discover and interact with it before ambassadors have time to do their own reviews
2. if ambassadors find issues, they can switch off the task type via community configuration
3. beta isn't a great approximation of production, but in this case it should be close enough to inspire confidence for a release

cc @KStoller-WMF

I'm OK with this approach if it helps reduce complexity and backport frustrations.

From my perspective I'm more worried about the possibility that this release causes some sort of regression / bug for the article-level "add an image" task (or Suggested edits as a whole). The reality is that few editors will be exposed to the section-level "add an image" task initially, so I agree it's fairly low-risk.

In T329276#8913794, @kostajh wrote:

Alternatively, we could just say that ambassadors should do any pre-launch testing on beta wikis.

Right, so merge the patch now, and use $wgGENewcomerTasksSectionImageRecommendationsEnabled for deployment?

In T329276#8916550, @Tgr wrote:

In T329276#8913794, @kostajh wrote:

Alternatively, we could just say that ambassadors should do any pre-launch testing on beta wikis.

Right, so merge the patch now, and use $wgGENewcomerTasksSectionImageRecommendationsEnabled for deployment?

Yes, I think so, and sounds like @KStoller-WMF is OK with it from the previous comment (T329276#8915382).

Change 924597 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Section images: Remove user variant

https://gerrit.wikimedia.org/r/924597