Page MenuHomePhabricator

Personalized praise: mentor dashboard settings
Closed, ResolvedPublic

Description

User Story:

As a mentor,
I want to offer feedback and encouragement to newcomers and I want to customize those settings,
Because then newcomers are more likely to continue contributing.

Background:
Designs

Figma Designs: Personalized Praise Settings

Module settings.png (592×367 px, 35 KB)

Community Configuration:

Mentors can set their own preferences, but we need reasonable defaults set. And what is reasonable will vary considerably by wiki (depending on the ratio of newcomers to mentors and other factors), so we should allow wikis to set their own default via community configuration.

Setting defaults should be configurable by wiki via Special:EditGrowthConfig, at the bottom of the Mentorship section.

Personalized praise:
Should the Mentor dashboard include a module to surface mentees who are editing successfully?
[x] Enabled

How often should Mentors receive notifications to remind them to send encouragement to mentees who are editing successfully?

  • Never
  • Monthly
  • Weekly
  • Daily
  • Immediately

To qualify a newcomer for personal encouragement from their mentor, they should have:

  • At least _____ edits within _____.

Based on data gathered in this task, we should set these as the initial baselines:

  • Weekly
  • At least 8 edits within 1 week
Questions:
  • How can we ensure we don't suggest newcomers whose edits are being reverted? Should we only include users with <2 reverts within that same time period or some other "quality gate" here? Or should we only consider unreverted edits in the edit count? Or is it worth bringing these active but struggling newcomers to the attention of their Mentor?
  • On wikis that don't have a large enough Mentor group to cover all newcomers, how can we ensure that we aren't suggesting Mentees who don't know they have a Mentor?
  • Should we have wikis customize the default message?
Acceptance Criteria:

Given I'm a Mentor and I visit my Mentor Dashboard,
When I have praiseworthy mentees,
Then I can adjust the following settings:

  • frequency of notifications: Immediately, Daily, Weekly, Monthly
  • default message subject
  • default message
  • metric that triggers adding mentee to "praiseworthy" list

Given I'm an Admin or privileged user,
When I want to adjust community configuration
Then I can adjust the following:

  • frequency of notifications: Immediately, Daily, Weekly, Monthly
  • metric that triggers adding mentee to "praiseworthy" list

Event Timeline

KStoller-WMF assigned this task to Urbanecm_WMF.
KStoller-WMF lowered the priority of this task from High to Medium.
KStoller-WMF updated the task description. (Show Details)
KStoller-WMF updated the task description. (Show Details)
KStoller-WMF updated the task description. (Show Details)
KStoller-WMF added a subscriber: BKots-WMF.

Change 894588 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Add backend for settings

https://gerrit.wikimedia.org/r/894588

Change 894588 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Add backend for settings

https://gerrit.wikimedia.org/r/894588

Change 889642 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] WIP: Frontend for Personalized praise module

https://gerrit.wikimedia.org/r/889642

Change 901120 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Personalized praise: Make defaults configurable via community config

https://gerrit.wikimedia.org/r/901120

Hi @KStoller-WMF, I just got to the community configuration bit of this task, and I have a couple of questions/suggestions, which you can find below (inlined in the specification quoted from the description).

Personalized praise:
Should the Mentor dashboard include a module to surface mentees who are editing successfully?

  • Enabled

I don't think this needs to be (community) configurable. It feels very unlikely a community would want to prohibit mentors to make talk page posts via this feature. Theoretically, a community might consider Personalized praise notifications as annoying and therefore want to disable the notifications (at least by default), which is possible below, but I find it hard to imagine a case when someone would want the whole module gone. To keep the config page of a reasonable length, I suggest removing this configuration option.

How often should Mentors receive notifications to remind them to send encouragement to mentees who are editing successfully?

  • Never
  • Monthly
  • Weekly
  • Daily
  • Immediately

No objections with including this. However, note that in practice, this will only affect new mentors (whenever a mentor presses "Save" in the settings dialog, all settings are set in their personal preferences; since that moment, only the mentor's preferences decide about the notification frequency, and community configuration would be ignored).

To qualify a newcomer for personal encouragement from their mentor, they should have:

  • At least _____ edits within _____.

With the way how community configuration form is currently set up, it's non-trivial to have two fields displayed next to each other. We need to resort to something like this:

image.png (491×963 px, 43 KB)

Unfortunately, this also means we need two copies. I suggested something in the screenshot above, but those are probably not ideal. I also think we should highlight mentors can change those values. I added a help text for that -- see screenshot above.

I also think that the maximum number of edits a mentee can have to be considered for Personalized praise should be configurable. See my scrrenshot above for how that can look like.

Hi @KStoller-WMF, I just got to the community configuration bit of this task, and I have a couple of questions/suggestions, which you can find below (inlined in the specification quoted from the description).

Personalized praise:
Should the Mentor dashboard include a module to surface mentees who are editing successfully?

  • Enabled

I don't think this needs to be (community) configurable. It feels very unlikely a community would want to prohibit mentors to make talk page posts via this feature. Theoretically, a community might consider Personalized praise notifications as annoying and therefore want to disable the notifications (at least by default), which is possible below, but I find it hard to imagine a case when someone would want the whole module gone. To keep the config page of a reasonable length, I suggest removing this configuration option.

OK, I'm fine with removing that configuration option.

How often should Mentors receive notifications to remind them to send encouragement to mentees who are editing successfully?

  • Never
  • Monthly
  • Weekly
  • Daily
  • Immediately

No objections with including this. However, note that in practice, this will only affect new mentors (whenever a mentor presses "Save" in the settings dialog, all settings are set in their personal preferences; since that moment, only the mentor's preferences decide about the notification frequency, and community configuration would be ignored).

Perfect, that was my assumption as to how this could be handled. Sorry, I could have been clearer in detailing assumption.

To qualify a newcomer for personal encouragement from their mentor, they should have:

  • At least _____ edits within _____.

With the way how community configuration form is currently set up, it's non-trivial to have two fields displayed next to each other. We need to resort to something like this:

image.png (491×963 px, 43 KB)

Unfortunately, this also means we need two copies. I suggested something in the screenshot above, but those are probably not ideal. I also think we should highlight mentors can change those values. I added a help text for that -- see screenshot above.

I also think that the maximum number of edits a mentee can have to be considered for Personalized praise should be configurable. See my scrrenshot above for how that can look like.

🏆 Great thinking on that last addition (Maximum number of days to consider a user new). I fully agree that we needed to add in a criteria to ensure we weren't suggesting established newcomers again and again.

I'll review all Personalized praise copy this week and finalize it. But if you want to make a few tweaks in the meantime:

Typo in "evulting" (should be evaluating).

For the Personalized praise description, I wonder if we should focus more on the underlying purpose of the feature? I'll chat with @JFernandez-WMF and @Trizek-WMF as well, but I'm thinking something like:

Receiving encouraging messages from established editors can improve new editor retention. The Personalized praise module surfaces praiseworthy new editors to their Mentor, to help productive new editors feel more engaged and involved in the community.

I'm really hoping we can update some of the copy if the experiment actually proves that this module helps newcomer retention or increases newcomer activity.

Change 901556 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] WIP: Personalized praise settings: Add message content

https://gerrit.wikimedia.org/r/901556

Change 901584 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise settings: Use single case style

https://gerrit.wikimedia.org/r/901584

I also think that the maximum number of edits a mentee can have to be considered for Personalized praise should be configurable. See my scrrenshot above for how that can look like.

🏆 Great thinking on that last addition (Maximum number of days to consider a user new). I fully agree that we needed to add in a criteria to ensure we weren't suggesting established newcomers again and again.

I was thinking about reusing the number of edits filter defined on the table of mentees, but it might be counterintuitive.

For the Personalized praise description, I wonder if we should focus more on the underlying purpose of the feature? I'll chat with @JFernandez-WMF and @Trizek-WMF as well, but I'm thinking something like:

Receiving encouraging messages from established editors can improve new editor retention. The Personalized praise module surfaces praiseworthy new editors to their Mentor, to help productive new editors feel more engaged and involved in the community.

It sounds fine with me.

For the Personalized praise description, I wonder if we should focus more on the underlying purpose of the feature? I'll chat with @JFernandez-WMF and @Trizek-WMF as well, but I'm thinking something like:

Receiving encouraging messages from established editors can improve new editor retention. The Personalized praise module surfaces praiseworthy new editors to their Mentor, to help productive new editors feel more engaged and involved in the community.

It sounds fine with me.

Sounds good to me too! Added to the patch for now (although with lowercase M; afaik we don't write "Mentors" with capital M elsewhere) -- we can of course always change the copy afterwards as-needed.

Test wiki created on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/2abf1605fe/w

Change 901120 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Make Personalized praise configurable via Special:EditGrowthConfig

https://gerrit.wikimedia.org/r/901120

Change 901584 abandoned by Urbanecm:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise settings: Use single case style

Reason:

squashed

https://gerrit.wikimedia.org/r/901584

Test wiki created on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/6411b44c82/w

Change 901556 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise settings: Add message content

https://gerrit.wikimedia.org/r/901556

Change 889642 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Frontend for Personalized praise module

https://gerrit.wikimedia.org/r/889642

Change 905256 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Remove hardcoded strings in seconds

https://gerrit.wikimedia.org/r/905256

Change 905256 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Remove hardcoded strings in settings

https://gerrit.wikimedia.org/r/905256

Change 906765 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Frontend improvements

https://gerrit.wikimedia.org/r/906765

Hi @JFernandez-WMF and @KStoller-WMF!

I uploaded a patch with several improvements of the Settings dialog for Personalized praise. Here is how it currently looks like in my development environment:

image.png (524×507 px, 18 KB)

I ended treating the Figma design (F35845335) as an inspirational one, as I find it difficult to put the submit button where it is with Codex's Dialog component (maybe I'm missing something [CC @Sgs], but I don't see a direct equivalent of OOUI's Process dialog component, which is what the Figma design seems to work with). I'd like to check if this deviation is okay (at least for the initial version), or if I should find a way to change the layout as specified.

I'm also unsure about certain other aspects of the settings module:

  • The subject/text fields don't have any label as of now. The Figma mockup contains a placeholder, but in practice, neither of those fields is going to be empty (initially, it is prefilled with Growth-provided default subject/message; after a mentor changes it, the fields will have mentor's message). I feel like it's understandable even without a label, but I'm asking anyway.
  • The in-form headlines look differently than specified. That's because how they look like in the mentor's settings module (displayed below Personalized praise). While we certainly can use two style of headings within one featureset if we want to, I feel like consistency would be better. If you agree, which of the two header styles should be used?

Test wiki on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/6411b44c82/w/

Test wiki on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/2abf1605fe/w/

Change 906765 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Frontend improvements

https://gerrit.wikimedia.org/r/906765

Change 908355 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise settings: Include a help text

https://gerrit.wikimedia.org/r/908355

Ad T322446#8766285: The Settings module was okay-ed in the changed version in today's mentorship meeting (except a helper explaining what $1 stands for is missing; added by a patch above).

Change 908355 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise settings: Include a help text

https://gerrit.wikimedia.org/r/908355

Change 909661 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Change type= to weight=

https://gerrit.wikimedia.org/r/909661

Change 909661 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Change type= to weight=

https://gerrit.wikimedia.org/r/909661

Etonkovidova subscribed.

Checked in betalabs (the screenshots below are from enwiki betalabs); the testwiki wmf.7 doesn't have notification options.
The functionality was checked - works as expected.

Settings with options for time frame for eidting
Screen Shot 2023-05-04 at 3.14.24 PM.png (1×1 px, 157 KB)
Settings with mentor's notification frequency
Screen Shot 2023-05-04 at 3.14.59 PM.png (1×1 px, 185 KB)

Special:EditGrowthConfig - Personalized praise (maybe the first field "By default, how often should mentors receive notifications ..." should clarify that notifications should be send "every 168 hours (weekly)"?

Screen Shot 2023-05-04 at 3.16.10 PM.png (1×2 px, 225 KB)

Urbanecm_WMF changed the task status from In Progress to Open.May 15 2023, 10:13 PM

Checked testwiki wmf.11 - all specs are in place.