@KStoller-WMF @RHo : I noticed one thing about this notification when I was working on investigating baseline for this and checked the Figma designs: if we define this as written then the longest possible interval a newcomer can go between making their first suggested edit and the notification being sent is 9 x 120 hours, which is 1080 hours or 45 days. Or say a newcomer activates by making a suggested edit on the first day, then they can wait 4 days between each of the 8 subsequent edits and get the notification 5 days after that for like 37 days total.

We won't be able to run an experiment lasting one month with that. I'm uncertain whether we'd be able to effectively experiment with it at all.

An alternative way to look at it is to start the clock when they make their first suggested edit, which most newcomers do on the first day (just like activation). That would change the first part of the definition to: Sent to newcomers with Growth features who have made less than $(10) suggested edits within $(120) hours since their first suggested edit.

I'm currently investigating what number of edits and amount of time would be reasonable for that second definition.

I dug into this as part of T322433 and found that the main limiting factor here is the threshold, and not time. The figure below exemplifies this, where the X-axis is time since first edit tagged "newcomer task" in hours, the Y-axis is the proportion of editors making a threshold, and each line is a different threshold (this figure is also on Commons as: Newcomer_task_milestones_by_hour_Feb_2023.png) It doesn't matter if we change the scale of the X-axis to minutes, hours, or days, the graph looks basically the same.

My interpretation of this is that it won't matter if we measure from the first or last suggested edit. We'll have to decide how many editors we wish to send notifications to (e.g. if we set the threshold to 2 edits we message roughly 55% of them, 3 means 75%, and 5 means >85%), and how quickly we want to reach out to them. Maybe there's something in the background research for this project that suggests certain time frames are good ones? One thing this analysis clearly showed is the 24-hour cycles of user behaviour, there are bumps around 24 and 48 hours in the graph above.

Based on needing to keep this data contained within a measurable period and the above data, I'm suggesting we change acceptance criteria to:

Sent to newcomers $(48) hours after account creation who have made between $(1) - $(4) suggested edits. (This notification should not be sent to newcomers who have not completed any suggested edits.)

@RHo does that seem reasonable? Do you think an additional time element is still needed?

Eventually we should make this community configurable - T328386: Leveling Up: community configuration - but I'm hoping this initial baseline results in enough notifications that we can see if this notification has an impact.

In T328288#8615831, @KStoller-WMF wrote:

Based on needing to keep this data contained within a measurable period and the above data, I'm suggesting we change acceptance criteria to:

Sent to newcomers $(48) hours after account creation who have made between $(1) - $(4) suggested edits. (This notification should not be sent to newcomers who have not completed any suggested edits.)

@RHo does that seem reasonable? Do you think an additional time element is still needed?

Eventually we should make this community configurable - T328386: Leveling Up: community configuration - but I'm hoping this initial baseline results in enough notifications that we can see if this notification has an impact.

Sounds good to me, thanks!

We might not need a maintenance script to trigger these. We could enqueue a delayed job on account creation (we do this for newcomer task caching on visits to Special:Homepage), and when the job evaluates, it can check if the user has performed any suggested edits, and if so, send the notification.

Change 894043 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] levelingup: Define "Keep Going" notification presentation model

https://gerrit.wikimedia.org/r/894043

Change 894048 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

https://gerrit.wikimedia.org/r/894048

Change 894048 abandoned by Kosta Harlan:

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

Reason:

squashed into I72c1840414fd67dd7c92b05c29eac9c0a0577e9b

https://gerrit.wikimedia.org/r/894048

Change 894651 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/Echo@master] icons: Support OOUI icons from editing-core bundle

https://gerrit.wikimedia.org/r/894651

@RHo @KStoller-WMF @JFernandez-WMF a couple of notes:

Providing the same link + label for primary link and secondary link looks a little awkward in email, is that acceptable for initial release, or do we want to fix this?

It looks less awkward in web, since the primary link label is not shown:

I don't think it is easily possible to set "Make a suggested edit" to bold per the mockup without making changes in Echo; is it OK to leave non-bolded?

In T328288#8668569, @kostajh wrote:

@RHo @KStoller-WMF @JFernandez-WMF a couple of notes:

Providing the same link + label for primary link and secondary link looks a little awkward in email, is that acceptable for initial release, or do we want to fix this?

Oof, that looks rough. Could we amend the secondary link to be about going to the homepage instead as a quick fix?

It looks less awkward in web, since the primary link label is not shown:

I don't think it is easily possible to set "Make a suggested edit" to bold per the mockup without making changes in Echo; is it OK to leave non-bolded?

Yes, sorry that is my bad as only page titles are meant to be bolded. I will update the mock.

In T328288#8668676, @RHo wrote:

In T328288#8668569, @kostajh wrote:

@RHo @KStoller-WMF @JFernandez-WMF a couple of notes:

Providing the same link + label for primary link and secondary link looks a little awkward in email, is that acceptable for initial release, or do we want to fix this?

Oof, that looks rough. Could we amend the secondary link to be about going to the homepage instead as a quick fix?

Sorry, I belatedly realized that we can alter the output as we wish for email notifications, so I have just excluded secondary links for email distribution type.

In T328288#8668676, @RHo wrote:

I don't think it is easily possible to set "Make a suggested edit" to bold per the mockup without making changes in Echo; is it OK to leave non-bolded?

Yes, sorry that is my bad as only page titles are meant to be bolded. I will update the mock.

I see that in the updated mock (F36894445), "suggested edit" is no longer bolded. So I'll update the patch. But the problem is with the "Make a suggested edit" string

In T328288#8669021, @kostajh wrote:

In T328288#8668676, @RHo wrote:

I don't think it is easily possible to set "Make a suggested edit" to bold per the mockup without making changes in Echo; is it OK to leave non-bolded?

Yes, sorry that is my bad as only page titles are meant to be bolded. I will update the mock.

I see that in the updated mock (F36894445), "suggested edit" is no longer bolded. So I'll update the patch. But the problem is with the "Make a suggested edit" string

AFAICT there is not an easy way to make that bold.

Oh derp, you're right it is also per the Echo styleguide to have non-bolded. Updated as well.

Change 894699 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[schemas/event/secondary@master] homepagevisit: Add new referer routes

https://gerrit.wikimedia.org/r/894699

Change 894699 merged by jenkins-bot:

[schemas/event/secondary@master] homepagevisit: Add new referer routes

https://gerrit.wikimedia.org/r/894699

Change 894651 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] icons: Support OOUI icons from editing-core bundle

https://gerrit.wikimedia.org/r/894651

Change 894043 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

https://gerrit.wikimedia.org/r/894043

Change 896011 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] config: Make GELevelingUpKeepGoingNotificationSendAfterSeconds an int

https://gerrit.wikimedia.org/r/896011

In T328288#8671424, @gerritbot wrote:

Change 894043 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

https://gerrit.wikimedia.org/r/894043

SRE, the above job uses jobReleaseTimestamp, does that need special handling anywhere in e.g. the operations/deployment-charts/changeprop service, or should it "just work"?

The job is enqueued on account registration with a jobReleaseTimestamp of 48 hours later.

In T328288#8679125, @kostajh wrote:

In T328288#8671424, @gerritbot wrote:

Change 894043 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

https://gerrit.wikimedia.org/r/894043

SRE, the above job uses jobReleaseTimestamp, does that need special handling anywhere in e.g. the operations/deployment-charts/changeprop service, or should it "just work"?

The job is enqueued on account registration with a jobReleaseTimestamp of 48 hours later.

jobReleaseTimestamp is supported by changeprop, via the delay_until parameter we add to the jobs via the eventbus extension:
https://gerrit.wikimedia.org/g/mediawiki/extensions/EventBus/+/4bc310d352129be192c10c23b7a64b06aea71bf3/includes/EventFactory.php#1105

so it works out of the box, but we will need to add a stanza to changeprop-jobqueue's deployment similar to what you already did in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/636078 so that:

1. This job gets a dedicated worker and doesn't block execution of other jobs
2. You can add a reenqueue_delay configuration larger than the delay of the job, so that we avoid having changeprop reenqueueing jobs that are just delayed.

In T328288#8679320, @Joe wrote:

In T328288#8679125, @kostajh wrote:

In T328288#8671424, @gerritbot wrote:

Change 894043 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] levelingup: Keep going notification

https://gerrit.wikimedia.org/r/894043

SRE, the above job uses jobReleaseTimestamp, does that need special handling anywhere in e.g. the operations/deployment-charts/changeprop service, or should it "just work"?

The job is enqueued on account registration with a jobReleaseTimestamp of 48 hours later.

jobReleaseTimestamp is supported by changeprop, via the delay_until parameter we add to the jobs via the eventbus extension:
https://gerrit.wikimedia.org/g/mediawiki/extensions/EventBus/+/4bc310d352129be192c10c23b7a64b06aea71bf3/includes/EventFactory.php#1105

so it works out of the box, but we will need to add a stanza to changeprop-jobqueue's deployment similar to what you already did in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/636078 so that:

1. This job gets a dedicated worker and doesn't block execution of other jobs
2. You can add a reenqueue_delay configuration larger than the delay of the job, so that we avoid having changeprop reenqueueing jobs that are just delayed.

Thanks! Moved over to T331616: changeprop configuration for notificationGetStartedJob and notificationKeepGoingJob for a patch, code review, and deployment.

Change 896011 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] config: Make GELevelingUpKeepGoingNotificationSendAfterSeconds an int

https://gerrit.wikimedia.org/r/896011

@kostajh
Checking on betalabs - the secondary header (i.e. "You've made [number of suggested edits]. Keep going...] is not wrapped. A user will see the full text only on Special:Notifications page.

	Special:Notifications page

In T328288#8712474, @Etonkovidova wrote:

@kostajh
Checking on betalabs - the secondary header (i.e. "You've made [number of suggested edits]. Keep going...] is not wrapped. A user will see the full text only on Special:Notifications page.

	Special:Notifications page

That sounds like a limitation in Echo; I'm not sure why it is limited to one line of text in the overlay but not on Special:Notifications. If we want to make it consistent with Special:Notifications and allow multiple (2? 3? unlimited?) lines of text in the overlay, then we should make a separate task for that in the Notifications (Echo) project. @KStoller-WMF @RHo, what do you think?

That sounds like a limitation in Echo

That's my understanding too. As I understand it, that the bold text can wrap, but the lighter text never wraps.

We definitely don't want to allow unlimited lines of text, since for talk page Notices, that text could be very long. I also imagine that many users prefer to keep that summary short so they can quickly scan notices. But perhaps two lines is reasonable?

Or should we simply consider shortening the copy? (Although acknowledge this still will be an issue in some translations).

Or perhaps it's a non-issue. It is possible that more people tap on the notice and explore Suggested Edits because the whole message is cut off and the leads to curiosity.

Either way, I don't think this is a release blocker, and if we make an improvement it should be a separate task.

In T328288#8716486, @KStoller-WMF wrote:

That sounds like a limitation in Echo

That's my understanding too. As I understand it, that the bold text can wrap, but the lighter text never wraps.

We definitely don't want to allow unlimited lines of text, since for talk page Notices, that text could be very long. I also imagine that many users prefer to keep that summary short so they can quickly scan notices. But perhaps two lines is reasonable?

Or should we simply consider shortening the copy? (Although acknowledge this still will be an issue in some translations).

Or perhaps it's a non-issue. It is possible that more people tap on the notice and explore Suggested Edits because the whole message is cut off and the leads to curiosity.

Either way, I don't think this is a release blocker, and if we make an improvement it should be a separate task.

Thanks, @KStoller-WMF and @kostajh!
I looked at Notifications spreadsheet. It seems that all wrapped text should be in a header. Even in a long notification, e.g. user-right changes notification:
Your user rights were {{GENDER:$2|changed}} by $1. You are now a member of {{PLURAL:$4|this group|these groups}}: $3; You are no longer a member of {{PLURAL:$6|this group|these groups}}: $5. - the whole text is a header.

I filed T332744: Leveling up notifications shouldn't be truncated from Notices panel to explore the options for displaying the whole text.