As mentioned in T337320, I wonder if the large number of outside mainspace activations are in User Talk and Wikipedia/Wikipedia Talk tracks with newcomers using the Help panel/Mentor feature? I also wonder if we are saying that this is proportion of non-mainspace activation is higher than the old Impact module design or overall? If not, it generally seems that the non-mainspace activation levels would be hard to shift via a change in the Impact module design.

In T338640#8946862, @RHo wrote:

As mentioned in T337320, I wonder if the large number of outside mainspace activations are in User Talk and Wikipedia/Wikipedia Talk tracks with newcomers using the Help panel/Mentor feature?

Good question, I'm sure that some of the "outside mainspace activations" are from newcomers asking Mentee questions. But my understanding is that this doesn't differ significantly across the experiment and control group:
"Mentor module interactions: We find no significant difference in the proportion of newcomers who interact with the Mentor module. The proportion for newcomers who get the old module is 2.4%, for those who get the new module it's 2.2%. A Chi-square test finds this difference not significant". https://www.mediawiki.org/wiki/Growth/Positive_reinforcement#Impact_4

I also wonder if we are saying that this is proportion of non-mainspace activation is higher than the old Impact module design or overall? If not, it generally seems that the non-mainspace activation levels would be hard to shift via a change in the Impact module design.

I don't think the new impact module has really shifted this, based on @nettrom_WMF 's comment here: https://phabricator.wikimedia.org/T337320#8919203 "There's not a significant difference between the experiment groups."

All that being said, I am considering if we want to adjust the logic to simply match the previous behavior:
Given a new editor hasn't edited in mainspace,
When they visit the newcomer homepage,
Then they see the Impact module's empty state (which includes a call to action to try Suggested edits)

That way we are removing a potential confounding variable from the experiment. Any concerns with that approach? @nettrom_WMF / @RHo / @JFernandez-WMF

In T338640#8953804, @KStoller-WMF wrote:

In T338640#8946862, @RHo wrote:

As mentioned in T337320, I wonder if the large number of outside mainspace activations are in User Talk and Wikipedia/Wikipedia Talk tracks with newcomers using the Help panel/Mentor feature?

Good question, I'm sure that some of the "outside mainspace activations" are from newcomers asking Mentee questions. But my understanding is that this doesn't differ significantly across the experiment and control group:
"Mentor module interactions: We find no significant difference in the proportion of newcomers who interact with the Mentor module. The proportion for newcomers who get the old module is 2.4%, for those who get the new module it's 2.2%. A Chi-square test finds this difference not significant". https://www.mediawiki.org/wiki/Growth/Positive_reinforcement#Impact_4

I also wonder if we are saying that this is proportion of non-mainspace activation is higher than the old Impact module design or overall? If not, it generally seems that the non-mainspace activation levels would be hard to shift via a change in the Impact module design.

I don't think the new impact module has really shifted this, based on @nettrom_WMF 's comment here: https://phabricator.wikimedia.org/T337320#8919203 "There's not a significant difference between the experiment groups."

All that being said, I am considering if we want to adjust the logic to simply match the previous behavior:
Given a new editor hasn't edited in mainspace,
When they visit the newcomer homepage,
Then they see the Impact module's empty state (which includes a call to action to try Suggested edits)

That way we are removing a potential confounding variable from the experiment. Any concerns with that approach? @nettrom_WMF / @RHo / @JFernandez-WMF

SGTM, thanks!

In T338640#8953804, @KStoller-WMF wrote:

That way we are removing a potential confounding variable from the experiment. Any concerns with that approach? @nettrom_WMF / @RHo / @JFernandez-WMF

This sounds good to me too! Happy to reconfigure some data aggregation to start fresh once this is deployed as well.

Change 934368 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] User impact: set module unactive until user edits in mainspace

https://gerrit.wikimedia.org/r/934368

Change 934368 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] User impact: set module unactive until user edits in mainspace

https://gerrit.wikimedia.org/r/934368

Checked in testwiki wmf.17. The edits in non-main namespace (User page and another User talk page are not present in the new Impact module.

The screenshots are for a user who did two non-main namespace edits.

Desktop	Mobile

Note: Moving to Design review to make sure that the following cases are ok.

• a user with non-main namespace edits sees the notification
  
• a user with non-main namespace edits may receive Thanks. The info popup on the new Impact module informs a user "You may be thanked by another contributor for an edit you’ve made [...]". So a user sees thanks count, and is informed that those thanks are for their edits, and yet, the Impact module edit count is still zero.

FTR: The patch had an implementation error, which was reported and fixed under T341994: New version of Special:Impact returns "0 edits so far" at some wikis even when edits have been made.

@Etonkovidova, thanks for documenting these cases where the "0 edits so far" text seems incongruous with the "You've made your first edit" notification or Thanks count.

@JFernandez-WMF - This is in design review for you to decide if this incongruity is OK, and if you consider this task resolved / complete.

My opinion is that I don't think this is something we need to fix as part of this task. I think we can consider this task resolved and just create a follow-up task to consider an improvement in the future... perhaps the empty state should say "0 article edits so far" or something?