@KStoller-WMF I don't see need for improvements here! I think what you've suggested is very clear and understandable.

Change 966908 had a related patch set uploaded (by Cyndywikime; author: Cyndywikime):

[mediawiki/extensions/GrowthExperiments@master] Add Community Configurations for Leveling Up Notifications

https://gerrit.wikimedia.org/r/966908

@KStoller-WMF , does the new section Notifications: Growth features have a specific location on the Special:EditGrowthConfig ? i.e maybe closer to related sections or at the bottom of the editgrowthconfig page?

In T343288#9264456, @Cyndymediawiksim wrote:

@KStoller-WMF , does the new section Notifications: Growth features have a specific location on the Special:EditGrowthConfig ? i.e maybe closer to related sections or at the bottom of the editgrowthconfig page?

Sorry I didn't specify in the task! Let's just add this section to the very bottom of the form for now. With the CC2.0 work we'll be reconsidering the layout / order of the config options.

Reviewed, moving back to Doing.

Change 972871 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] SpecialEditGrowthConfig: Set min to zero for levelling up config

https://gerrit.wikimedia.org/r/972871

Change 966908 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Add Community Configurations for Leveling Up Notifications

https://gerrit.wikimedia.org/r/966908

Change 972871 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] SpecialEditGrowthConfig: Set min to zero for levelling up config

https://gerrit.wikimedia.org/r/972871

Checked in testwiki wmf.5 Special:EditGrowthConfig has added sections for notifications:

A new user with a suggested edit(s) gets "Keep going notification":

Another of my new test users made 1 edit (non-suggested) and did not receive "Try Suggested edits notification" notification. User contributions for QTETest2 nov17 shows one non-suggested edit; Special:Notificaitons for that user do not show "Try Suggested edits notification" (Preferences-Notificaitons do have Growth features notifications enabled by default):

Notes: minor issues

• Notifications: Growth Features is not placed at the bottom of Special:EditGrowthConfig as specified in https://phabricator.wikimedia.org/T343288#9274164
• "Keep going notification" notification layout/content is different from specs in the task description, i.e. the text doesn't include the count of suggested edits that a user made.

In T343288#9351239, @Etonkovidova wrote:

Checked in testwiki wmf.5
[...]

Thanks for testing @Etonkovidova.

Another of my new test users made 1 edit (non-suggested) and did not receive "Try Suggested edits notification" notification. User contributions for QTETest2 nov17 shows one non-suggested edit; Special:Notificaitons for that user do not show "Try Suggested edits notification" (Preferences-Notificaitons do have Growth features notifications enabled by default):

I'll re-test to see if it's an actual issue. (will provide an update in this comment)

I reviewed this in test.wikipedia production as well, and the Try suggested edits notification not appearing is most definitely a bug. Moving the task back to Doing. @Cyndymediawiksim, can you please check why this is happening and fix this bug that Elena found?

Notes: minor issues

• Notifications: Growth Features is not placed at the bottom of Special:EditGrowthConfig as specified in https://phabricator.wikimedia.org/T343288#9274164

Thanks for noticing this. Personally, I think this is ignoreable, since (as Kirsten noted), we'd be redesigning the config page with CC2.0 anyway. But, if we want to, it should be trivial to change.

• "Keep going notification" notification layout/content is different from specs in the task description, i.e. the text doesn't include the count of suggested edits that a user made.

I think that wasn't meant to be in scope of this task originally. Can we split that to a new task?

Another of my new test users made 1 edit (non-suggested) and did not receive "Try Suggested edits notification" notification. User contributions for QTETest2 nov17 shows one non-suggested edit; Special:Notificaitons for that user do not show "Try Suggested edits notification" (Preferences-Notificaitons do have Growth features notifications enabled by default):

I'll re-test to see if it's an actual issue. (will provide an update in this comment)

I reviewed this in test.wikipedia production as well, and the Try suggested edits notification not appearing is most definitely a bug. Moving the task back to Doing. @Cyndymediawiksim, can you please check why this is happening and fix this bug that Elena found?

@Urbanecm_WMF , tested why this case was not showing and i think maybe because of the property used as you had highlighted before:GELevelingUpGetStartedMaxTotalEdits -> "description": "Do not send a 'get started' notification to users with more than this edits.".
Using the GELevelingUpManagerInvitationThresholds property, a notification drawer is seen at the bottom after set number of edits according to the config settings. I have a question though, if we already have the notification drawer happening after a certain number of edits anyway, do we really need the notification?IMO, this may not be very intuitive to the user.e.g The Get started notification is set to 1 and the suggested edit is also set to 1. Thoughts? cc:@Sgs

In T343288#9356032, @Urbanecm_WMF wrote:

I reviewed this in test.wikipedia production as well, and the Try suggested edits notification not appearing is most definitely a bug. Moving the task back to Doing. @Cyndymediawiksim, can you please check why this is happening and fix this bug that Elena found?

I've tested the issue in testwiki and could not reproduce it. Note that for a Get started/Try suggested edits notification to be shown in testwiki the following criteria need to be met:

• Leveling up features are enabled (GELevelingUpFeaturesEnabled is true in both beta and test)
• User is in "control" group (changing the user variant after the account is created won't work since the variant is checked on the local user created hook)
• Maximum number of edits the user in test has is (strictly) lesser than setting in EditGrowthConfig (10 in beta, 1 in testwiki)
• Wait 300 seconds in beta (source) or 48 hours in test (source, there's no specific setting for test/prod wikis)

My successful test was done in testwiki with user TestNotification123 (0 edits), waited 48hours, the notification was displayed. For some reason I haven't yet found the notification is not showing in Beta. Is the job queue setup in beta equal to testwiki?

@Etonkovidova @Urbanecm_WMF did you wait the 48hours on testwiki? Any clues why we're not seeing it in Beta? Is there a job queue log for beta we can look at? Thank you!

In T343288#9455261, @Cyndymediawiksim wrote:

@Urbanecm_WMF , tested why this case was not showing and i think maybe because of the property used as you had highlighted before:GELevelingUpGetStartedMaxTotalEdits -> "description": "Do not send a 'get started' notification to users with more than this edits.".

What steps you performed to try to reproduce this? Can you elaborate how the property is being used?

Using the GELevelingUpManagerInvitationThresholds property, a notification drawer is seen at the bottom after set number of edits according to the config settings. I have a question though, if we already have the notification drawer happening after a certain number of edits anyway, do we really need the notification? IMO, this may not be very intuitive to the user.e.g The Get started notification is set to 1 and the suggested edit is also set to 1. Thoughts? cc:@Sgs

GELevelingUpManagerInvitationThresholds controls whether the user will be invited to try suggested edits from the article namespace, so when they have already edited some articles (non-suggested edits). GELevelingUpGetStartedMaxTotalEdits controls whether the user will be invited to try suggested edits, regardeless of the namespace the edits are made (non-suggested edits). The outreach of the notification is bigger than drawer prompts because there's no minimal number of edits set for it.

In T343288#9459551, @Sgs wrote:

In T343288#9356032, @Urbanecm_WMF wrote:

I reviewed this in test.wikipedia production as well, and the Try suggested edits notification not appearing is most definitely a bug. Moving the task back to Doing. @Cyndymediawiksim, can you please check why this is happening and fix this bug that Elena found?

I've tested the issue in testwiki and could not reproduce it. Note that for a Get started/Try suggested edits notification to be shown in testwiki the following criteria need to be met:

• Leveling up features are enabled (GELevelingUpFeaturesEnabled is true in both beta and test)
• User is in "control" group (changing the user variant after the account is created won't work since the variant is checked on the local user created hook)
• Maximum number of edits the user in test has is (strictly) lesser than setting in EditGrowthConfig (10 in beta, 1 in testwiki)
• Wait 300 seconds in beta (source) or 48 hours in test (source, there's no specific setting for test/prod wikis)

My successful test was done in testwiki with user TestNotification123 (0 edits), waited 48hours, the notification was displayed. For some reason I haven't yet found the notification is not showing in Beta. Is the job queue setup in beta equal to testwiki?

@Etonkovidova @Urbanecm_WMF did you wait the 48hours on testwiki? Any clues why we're not seeing it in Beta? Is there a job queue log for beta we can look at? Thank you!

Thx, @Sgs! Yes, I waited for 48 hours on testwiki. I did not take into account that a user should be in the control group.

Maximum number of edits the user in test has is (strictly) lesser than setting in EditGrowthConfig

Logically it makes sense. So, if the max number of edits is set to 1, that means new users making one edit won't receive the "keep going" notification since they exceeded the max. Then, the issue that I mentioned in my comment, actually, isn't an issue - it works as expected.
Anyway, I've started a new test on testwiki wmf.14 - two new accounts - one made 0 edits, another - 1 edit. On beta, I will test with the default settings for max edits.

Test results on testwiki wmf.14 has shown the expected notification behavior:

testwiki Special:EditGrowthConfig (Jan 21/2024) settings:

In T343288#9356032, @Urbanecm_WMF wrote:

Notes: minor issues

• Notifications: Growth Features is not placed at the bottom of Special:EditGrowthConfig as specified in https://phabricator.wikimedia.org/T343288#9274164

Thanks for noticing this. Personally, I think this is ignoreable, since (as Kirsten noted), we'd be redesigning the config page with CC2.0 anyway. But, if we want to, it should be trivial to change.

Agree.

• "Keep going notification" notification layout/content is different from specs in the task description, i.e. the text doesn't include the count of suggested edits that a user made.

I think that wasn't meant to be in scope of this task originally. Can we split that to a new task?

The most recent "Keep going notification" notification does include the count. Any changes/adjustments are needed, @JFernandez-WMF?

"Keep going notification" from the task description	testwiki wmf.15
Example "keep going" notification:

In T343288#9475250, @Etonkovidova wrote:

Test results on testwiki wmf.14 has shown the expected notification behavior:

testwiki Special:EditGrowthConfig (Jan 21/2024) settings:

Try Suggested edits notification	1
Keep going notification	1

new user	Made suggested edits?	Made non-suggested edits	Notification received
user A	no	no	received "Try Suggested edits notification" ✅ Expected behavior
user B	no	1 edit	no notifications the user might not be in the control group?
user C	1 edit	no	received "Keep going to learn more..." ✅ Expected behavior

Thanks @Etonkovidova for testing this. @Sgs I had a quick question - how do I know in Beta which control group a user is assigned?

In T343288#9513000, @Cyndymediawiksim wrote:

Thanks @Etonkovidova for testing this. @Sgs I had a quick question - how do I know in Beta which control group a user is assigned?

We have a client helper to check for this: ge.utils.getUserVariant(). Although in this case you need to make user the user is assigned to control group on creation since the check is run in the onLocalUserCreated.

I've tried to debug this through looking at Logs from the Beta Cluster are sent to Logstash. I found errors about a worker dying and restarting. But I couldn't directly link it to why the notification was not being sent for new users. I'm moving this to blocked until I can maybe be able to ssh to deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud and find out why. cc: @Sgs

In T343288#9532399, @Cyndymediawiksim wrote:

I've tried to debug this through looking at Logs from the Beta Cluster are sent to Logstash. I found errors about a worker dying and restarting. But I couldn't directly link it to why the notification was not being sent for new users. I'm moving this to blocked until I can maybe be able to ssh to deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud and find out why. cc: @Sgs

The job queue has probably been down for some time, see T357476: beta.wmflabs.org: Job infrastructure is down. I think we can perform a regular user test now that's been fixed. If something fails in the test we hopefully will see some error in https://beta-logs.wmcloud.org.

Thanks @Sgs and @Urbanecm for unblocking this. A Try Suggested Edit notification is shown for a new user with 0 edits. See below :

I have tested in Beta Cluster with Try Suggested Edits set to 1, and I found that a new user does not receive a notification when 1 non-suggested edit is made. This is equivalent to setting the value to zero. I suggest that we modify the strict comparison in the code:

public function shouldSendGetStartedNotification( UserIdentity $userIdentity ): bool {
		$maxEdits = (int)$this->growthConfig->get( 'GELevelingUpGetStartedMaxTotalEdits' );

		return $this->getSuggestedEditsCount( $userIdentity ) === 0 &&
			 $this->userEditTracker->getUserEditCount( $userIdentity ) < $maxEdits;
	}

Thoughts, @Sgs? @Urbanecm_WMF

In T343288#9545883, @Cyndymediawiksim wrote:

I have tested in Beta Cluster with Try Suggested Edits set to 1, and I found that a new user does not receive a notification when 1 non-suggested edit is made. This is equivalent to setting the value to zero. I suggest that we modify the strict comparison in the code:

public function shouldSendGetStartedNotification( UserIdentity $userIdentity ): bool {
		$maxEdits = (int)$this->growthConfig->get( 'GELevelingUpGetStartedMaxTotalEdits' );

		return $this->getSuggestedEditsCount( $userIdentity ) === 0 &&
			 $this->userEditTracker->getUserEditCount( $userIdentity ) < $maxEdits;
	}

Thoughts, @Sgs? @Urbanecm_WMF

Per the review comments in 943622 it seems the strict comparison was intentional. But the wording for users in the help text is indeed confusing to me: Maximum number of edits a newcomer can have to qualify for receiving this notification. If set to 0, no newcomers will receive this notification. I think we should change the strict comparison to allow equality or we should rephrase the help text accordingly: Newcomers who have made less edits than this number will get a notification.. And remove the If set to 0.... @KStoller-WMF @Urbanecm ?

My understanding is that the "set to 0" option is basically meant to disable these notifications. I wonder if we should just include a disable toggle rather than including that language? That being said, if you think that's a good idea I'm happy to write up a new task, as that's definitely not included in the original acceptance criteria of this task.

I agree the copy is pretty confusing right now. As long as when set to zero no notifications are sent, then I think that's OK for now. @JFernandez-WMF is working to improve Community Configuration copy now: T356606: Community Configuration: finalize copy and update designs, so I think it's OK to wait on copy improvements until that task is complete. Unless @JFernandez-WMF has any immediate recommendations.

Moving this to QA. cc: @Etonkovidova.