Page MenuHomePhabricator

Mentorship: let mentors choose volume of mentees
Closed, ResolvedPublic

Description

"As a mentor, I want to choose how many mentees are assigned to me so that I can make sure I'm only receiving as many questions as I can handle."

On some large wikis, mentors receive more questions than they have time to answer. While increasing the size of the mentor pool is one solution, another is to distribute mentees more intelligently. Some mentors want to spend a lot of mentoring and some only want to spend a small amount of time. We could allow them to indicate their desired volume, and then assign more or fewer mentees to them.

Open questions
  • With what granularity should we allow mentors to sign up? It could be like "high, medium, low".
  • Where should the indication happen? Should it be where they list their username and greeting? Or should it be part of T280307: Mentor dashboard: M2 mentor tools/settings?
  • Maybe an even more sophisticated version of this would allow mentors to indicate exactly how many mentees they want to receive per week, and then if more newcomers show up than the mentors have signed up for, those extra don't receive mentors (because of insufficient capacity).
  • Should we tell wikis what's their mentorship capacity vs. number of newcomers?
Scoping
Using weights

As @Tgr already correctly noted, the easiest thing to do here is to use high/low/medium as weights. So, if UserA is enrolled with factor of "high", UserB is enrolled with factor of "medium" and userC with a factor of "low", the assigning system could see the pool of mentors as:

  • UserA
  • UserA
  • UserA
  • UserA
  • UserB
  • UserB
  • UserC

UserC would have the lowest chance to be assigned (1/7), because they have the lowest factor. UserA, on the other hand, would have the highest chance: 4/7 (over 57%). However, this will only be true if most people sign up with factor of "medium". For example, consider following list of mentors: UserA (low), UserB (low), UserC (low). With this list and factors, the pool of mentors will be (using same weights as above):

  • UserA
  • UserB
  • UserC

Each of the users will have an equal chance of being picked up as a mentor.

Different weights

An alternative solution here would be to still use weights, but interpret them differently. It could be a percentage of the ordinary mentor traffic. For instance, if a mentor would set their percentage to 80%, the system will ignore 20% of mentees that would be otherwise assigned to the mentor (and instead won't offer any mentorship to those users). For mentors wishing to increase their percentage, we can simply duplicate their name in the pool.

Number of questions asked

Since questions go through our interface, we can in theory measure number of questions asked in the last week, and if this number exceeds a given threshold, stop asssigning the mentor temporarily. The threshold can be configurable by the mentor themselves. The good thing is that this system will be perfectly understandable: mentors will likely be able to estimate what the workload for answering 2 questions vs 5 questions vs 10 questions is. The bad thing is that it is considerably more difficult to implement than the weighted solutions.

As before, we can "boost" the number of newcomers assigned to the task by duplicating their name in the pool.

Event Timeline

I'm moving this to @RHo's column while we discuss it. @Urbanecm_WMF @Trizek-WMF, could you please add any open questions or thoughts you can think of?

Let's say that we have a wiki with a potential of 150 questions per week. And we 10 mentors, and they only want to get 10 questions per week (or they have a granularity system that does the same).

What do we do with the (50) remaining questions? Should we remove the mentorship module for these newcomers? Is it fair? Should we revive our old "Ask on help desk" module as a backup plan?

How do we tell the overall community that some newcomers have some potential questions that will never be answered because they don't have a mentor? Can we transform active users into mentors to cover the needs? (Possibly T271317: Notification to invite people who reach certain conditions to become mentors)

If we go with a granularity system, how do we define "high, medium, low"? Based on the number of new accounts? The number of new active accounts? on the feeling of mentors? If one selects "low" and gets too many questions to their taste, what would be their opinion regarding mentoring?

Could this perception of "too many questions" being directly connected with the perception of the quality of the questions received? Having too many nonconstructive questions (even if nicely written) could influence the perception of having too many questions. (Covered in T287915: Mentorship: allow newcomers to opt-in and opt-out of mentorship)

If we go with a granularity system, how do we define "high, medium, low"? Based on the number of new accounts? The number of new active accounts? on the feeling of mentors?If one selects "low" and gets too many questions to their taste, what would be their opinion regarding mentoring?

The easy thing would be to just use them as weights when randomly picking mentors.

Could this perception of "too many questions" being directly connected with the perception of the quality of the questions received? Having too many nonconstructive questions (even if nicely written) could influence the perception of having too many questions. (Covered in T287915: Mentorship: allow newcomers to opt-in and opt-out of mentorship)

On one hand, getting silly questions and having to spend time answering them can be annoying and demotivating. On the other hand, they are easy to answer, so still much less effort than answering real questions that are e.g. outside one's area of experience/expertise, or just nuanced. (Aside: I think the effectiveness of mentoring depends a lot on the extent to which help material which can be linked for common questions is available, and to the extent that help material is actually helpful and user-friendly, given wikipedians' fascination with long, dense and super-detailed help pages.)

Let's say that we have a wiki with a potential of 150 questions per week. And we 10 mentors, and they only want to get 10 questions per week (or they have a granularity system that does the same).
What do we do with the (50) remaining questions? Should we remove the mentorship module for these newcomers? Is it fair? Should we revive our old "Ask on help desk" module as a backup plan?
How do we tell the overall community that some newcomers have some potential questions that will never be answered because they don't have a mentor? Can we transform active users into mentors to cover the needs? (Possibly T271317: Notification to invite people who reach certain conditions to become mentors)

! In T287917#7257627, @Tgr wrote:

! In T287917#7255408, @Trizek-WMF wrote:

If we go with a granularity system, how do we define "high, medium, low"? Based on the number of new accounts? The number of new active accounts? on the feeling of mentors?If one selects "low" and gets too many questions to their taste, what would be their opinion regarding mentoring?

The easy thing would be to just use them as weights when randomly picking mentors.

I think it is beneficial to considering this from the perspective what the stated user job story is – to accommodate mentors who want to be able to control the volume of questions they get each week. My understanding from the description and discussion is that pinpointing an exact number is not so easy, esp. as @Trizek_WMF notes above this would potentially result in 'remaining' questions, so a general weighting sounds reasonable and easy. However, as a mentor seeing this setting, I would expect:

  1. There is a rough estimated number of questions to expect each week after each weighting. For example High (30+ q/week), Medium (15-30q/week), Low (less than 15q/week).
  2. An ability to change the weighting in my mentor settings at different times (e.g., maybe I less time to spend in December). Is this intended to be an initial setting though, or something that mentors can (in theory) constantly change?

Could this perception of "too many questions" being directly connected with the perception of the quality of the questions received? Having too many nonconstructive questions (even if nicely written) could influence the perception of having too many questions. (Covered in T287915: Mentorship: allow newcomers to opt-in and opt-out of mentorship)

On one hand, getting silly questions and having to spend time answering them can be annoying and demotivating. On the other hand, they are easy to answer, so still much less effort than answering real questions that are e.g. outside one's area of experience/expertise, or just nuanced. (Aside: I think the effectiveness of mentoring depends a lot on the extent to which help material which can be linked for common questions is available, and to the extent that help material is actually helpful and user-friendly, given wikipedians' fascination with long, dense and super-detailed help pages.)

The perception of question quality is another reason why I think it's important to indicate a rough number of questions whatever the weighting decision may be. One person may consider 10 questions a week to be Low and another very high, depending also on how they like to respond to questions.

Something I tend to forget is that we are discussing about the number of questions, which is actually very difficult to predict.

As a mentor, while I have a constant number of newcomers who get my username (in theory). I can have weeks with zero questions, and weeks with 10+ questions. Why? Just because I'm randomly assigned motivated people. So we can't provide a way for mentors to select a volume of questions, while the only way to have it varying would be to work on the number of assigned mentees.

An example: on week 1, 5 questions were posted on my talk page - I was on "medium" and the system assigned me all my mentees, let's say 100. On week 2, as I'm busy, I decide to lower the number of "questions" I'd receive. I select "low" and the system now assign my name to only 50% of mentees, meaning 50. What if this week I still get 5 questions? The system will be perceived as broken (I have the same number fo questions), while it perfectly worked (you get a lower number of mentees).

We need to be careful on what we promise.

The more I think about this, the more I'm in favor of a percentage or a proportion of questions instead of a fixed number. This would cover the case of a community where 100 mentees sign up monthly, but all of the 5 mentors have selected 15 mentees/week. It would mean 25 mentees left alone. But if a mentor selects to be assigned to 1/3 of mentees, based on the overall number of registrations and the other thresholds selected by other mentors, the number would vary week after week. This would also encourage mentors to recruit other mentors.

Speaking of, we should inform the community about the impact of selecting a volume of questions, in live. Something should be done to show the number of mentors needed to reply to the "medium" volume, for instance. It would also be a nice way to encourage the community to recruit more mentors. Better, we could use volume variations to automatically launch recruitment campaigns. (see T271317: Notification to invite people who reach certain conditions to become mentors)

Thanks @Trizek-WMF and @RHo for your comments, they are helpful. Replying to a couple of important points below (summary is at the very bottom):

[...]

If we go with a granularity system, how do we define "high, medium, low"? Based on the number of new accounts? The number of new active accounts? on the feeling of mentors?If one selects "low" and gets too many questions to their taste, what would be their opinion regarding mentoring?

The easy thing would be to just use them as weights when randomly picking mentors.

I think it is beneficial to considering this from the perspective what the stated user job story is – to accommodate mentors who want to be able to control the volume of questions they get each week. My understanding from the description and discussion is that pinpointing an exact number is not so easy, esp. as @Trizek_WMF notes above this would potentially result in 'remaining' questions, so a general weighting sounds reasonable and easy. However, as a mentor seeing this setting, I would expect:

  1. There is a rough estimated number of questions to expect each week after each weighting. For example High (30+ q/week), Medium (15-30q/week), Low (less than 15q/week).

As a mentor myself, I would appreciate knowing that number. However, I'm afraid the best we can provide is guesstimates: as you can see at https://w.wiki/4TuH, it's easy for arwiki to have 28 questions in one day, but 10 in the other. Similar variety will be present in questions shown to individual mentors, too. In another words, I don't think we can promise they will not receive more than ie. 15 questions per week.

  1. An ability to change the weighting in my mentor settings at different times (e.g., maybe I less time to spend in December). Is this intended to be an initial setting though, or something that mentors can (in theory) constantly change?

This is definitely a must-have, thanks for pointing it out. Personally, I would put this into mentor tools module at the mentor dashboard: do you think that'd work?

Something I tend to forget is that we are discussing about the number of questions, which is actually very difficult to predict.

As a mentor, while I have a constant number of newcomers who get my username (in theory). I can have weeks with zero questions, and weeks with 10+ questions. Why? Just because I'm randomly assigned motivated people. So we can't provide a way for mentors to select a volume of questions, while the only way to have it varying would be to work on the number of assigned mentees.

At the theoretical level, we can make sure a mentor isn't asked a question more than N times in a week, but we'd need to accept the relationship will not be permanent anymore (as we'd have to rely on backup mentors). However, that solution would be quite complex (but possible :)).

An example: on week 1, 5 questions were posted on my talk page - I was on "medium" and the system assigned me all my mentees, let's say 100. On week 2, as I'm busy, I decide to lower the number of "questions" I'd receive. I select "low" and the system now assign my name to only 50% of mentees, meaning 50. What if this week I still get 5 questions? The system will be perceived as broken (I have the same number fo questions), while it perfectly worked (you get a lower number of mentees).

We need to be careful on what we promise.

Big +1 to this sentence :).

The more I think about this, the more I'm in favor of a percentage or a proportion of questions instead of a fixed number. This would cover the case of a community where 100 mentees sign up monthly, but all of the 5 mentors have selected 15 mentees/week. It would mean 25 mentees left alone. But if a mentor selects to be assigned to 1/3 of mentees, based on the overall number of registrations and the other thresholds selected by other mentors, the number would vary week after week. This would also encourage mentors to recruit other mentors.

If I understand your idea right, it's letting mentors to select how many mentees they receive via a weigh factor (for instance, I could have a weigh of 2/3 and you just 1/3, which would mean you'll have less mentees than me), so something similar to what's explained in the description as "weighted solution". I agree that would be simple, but I don't understand how it would lead to "percentage of questions". Wouldn't it still be "of mentees"?

Speaking of, we should inform the community about the impact of selecting a volume of questions, in live. Something should be done to show the number of mentors needed to reply to the "medium" volume, for instance. It would also be a nice way to encourage the community to recruit more mentors. Better, we could use volume variations to automatically launch recruitment campaigns. (see T271317: Notification to invite people who reach certain conditions to become mentors)

I agree with this, but I think this is out of scope for this task.


Taking into account everything that was said here (as well as discussion with Dennis and @Tgr), I think we should go forward with a pure weighted solution and see if that helps. The configuration option should IMO be a part of the mentor dashboard (part of mentor tools module). If desired, we can deploy the change first on the pilot wikis (+maybe some wikis volunteering to try this new feature), and check if that is enough, or if we need to do something more.

@Trizek-WMF @RHo I'll start working on the weights, but do let me know if you have any remaining concerns.

Change 742269 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] MentorManager: Support weights for mentors

https://gerrit.wikimedia.org/r/742269

Change 742936 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/WikimediaEvents@master] PrefUpdate: Track growthexperiments-mentorship-weight for Growth

https://gerrit.wikimedia.org/r/742936

Change 742938 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Mentor tools: Add frontend for changing mentor load

https://gerrit.wikimedia.org/r/742938

Hello! So far, this is what I came up with:

image.png (549×690 px, 26 KB)

You can see the dropdown in action at https://ctrlv.tv/EjxZ, too.

I'm not sure if labelling the feature as "Mentor load" makes sense, or if there's a better label we can use. I'm also not sure where to put the message saying "two times normal load", "two times less normal load", etc. Would changing the popup that appears if you use the (i) icon be sufficient? Or should we somehow put that to the dropdown itself? Or maybe using a dropdown is not a wise choice at all, and we should use an overlay? @RHo, I would appreciate your thoughts about this.

Hello! So far, this is what I came up with:

image.png (549×690 px, 26 KB)

You can see the dropdown in action at https://ctrlv.tv/EjxZ, too.

I'm not sure if labelling the feature as "Mentor load" makes sense, or if there's a better label we can use. I'm also not sure where to put the message saying "two times normal load", "two times less normal load", etc. Would changing the popup that appears if you use the (i) icon be sufficient? Or should we somehow put that to the dropdown itself? Or maybe using a dropdown is not a wise choice at all, and we should use an overlay? @RHo, I would appreciate your thoughts about this.

Hi @Urbanecm_WMF - I agree with the alternative label suggestion to Mentor load that @Trizek-WMF gave off-phab:
Number of mentees assigned to me

Besides that, I think the dropdown is fine, but with different option labels and a tooltip that can better explain the intention of the setting as per below:

image.png (662×1 px, 89 KB)

*Draft* Tooltip text in the mock:

Changing to more or less mentees will help you manage the number of questions you may expect to answer.
It’s recommended to start on the “Average” weighting to gauge your support capacity before changing to the other options.

Change 742269 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] MentorManager: Support weights for mentors

https://gerrit.wikimedia.org/r/742269

*Draft* Tooltip text in the mock:

Changing to more or less mentees will help you manage the number of questions you may expect to answer.
It’s recommended to start on the “Average” weighting to gauge your support capacity before changing to the other options.

May I suggest:

Changing to more or less mentees will help you manage the number of questions you may expect to answer. However, the number of questions only depends on the motivation of the mentees assigned to you.
It’s recommended to start on the “Average” weighting to gauge your support capacity before changing to the other options.

*Draft* Tooltip text in the mock:

Changing to more or less mentees will help you manage the number of questions you may expect to answer.
It’s recommended to start on the “Average” weighting to gauge your support capacity before changing to the other options.

May I suggest:

Changing to more or less mentees will help you manage the number of questions you may expect to answer. However, the number of questions only depends on the motivation of the mentees assigned to you.
It’s recommended to start on the “Average” weighting to gauge your support capacity before changing to the other options.

+1 The extra sentence to remind people that it is somewhat down to the question-asking tendencies of the specific mentees assigned to each mentor makes sense.

Test wiki created on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/d9c47f82ca/w/

Wording updated in the patch per T287917#7567880.

@RHo @Trizek-WMF You can review the current interface via https://patchdemo.wmflabs.org/wikis/d9c47f82ca/wiki/Special:MentorDashboard (login as "Patch Demo", password "patchdemo1"). Let me know what you think :).

Wording updated in the patch per T287917#7567880.

@RHo @Trizek-WMF You can review the current interface via https://patchdemo.wmflabs.org/wikis/d9c47f82ca/wiki/Special:MentorDashboard (login as "Patch Demo", password "patchdemo1"). Let me know what you think :).

Thanks @Urbanecm_WMF - the wording and dropdown looks fine to me. However I suggest adding a margin-bottom:1embelow the weighting field, as currently the "Message to your mentees" label is too close to it, and looks like it belongs with the weighting dropdown selection.

image.png (1×1 px, 303 KB)

Change 742936 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] PrefUpdate: Track growthexperiments-mentorship-weight for Growth

https://gerrit.wikimedia.org/r/742936

Change 742938 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Mentor tools: Add frontend for changing mentor load

https://gerrit.wikimedia.org/r/742938

@Trizek-WMF You can review the current interface via https://patchdemo.wmflabs.org/wikis/d9c47f82ca/wiki/Special:MentorDashboard (login as "Patch Demo", password "patchdemo1"). Let me know what you think :).

Looks good to me, thank you Martin!

Change 748837 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] MentorTools: Add margin below weighting field

https://gerrit.wikimedia.org/r/748837

[...]
Thanks @Urbanecm_WMF - the wording and dropdown looks fine to me. However I suggest adding a margin-bottom:1embelow the weighting field, as currently the "Message to your mentees" label is too close to it, and looks like it belongs with the weighting dropdown selection.

image.png (1×1 px, 303 KB)

Good point, thanks for suggesting the change. Uploaded a patch to do that.

I don't think this is in the right column. Once https://gerrit.wikimedia.org/r/748837 is merged, I think we can call this resolved.

For deployment, I think we can deploy this as part of T280307: Mentor dashboard: M2 mentor tools/settings deployment (@Trizek-WMF, do you think that's a good idea?)

Change 748837 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] MentorTools: Add margin below weighting field

https://gerrit.wikimedia.org/r/748837

Checked on testwiki wmf.19.

Screen Shot 2022-01-26 at 4.00.16 PM.png (530×517 px, 52 KB)
Screen Shot 2022-01-26 at 4.02.00 PM.png (521×603 px, 83 KB)

Moving for Design Review to review the following

Questions:
(1) The usage of tilde in front of words - usually the tilde (as a sub for "approximation") is used in front of numbers. And, probably, the tilde is not necessary to use here at all.

(2) Can we re-check the wording of the tooltip? "Changing to more or less mentees " looks incomplete, and ""Average" weighting" refers to the technical implementation detail that is not relevant in this context.

Changing to more or less mentees will help you manage the number of questions you may expect to answer. However, the >number of questions only depends on the motivation of the mentees assigned to you.

It’s recommended to start on the "Average" weighting to gauge your support capacity before changing to the other options.

Some suggestions:
"Changing to more or less mentees" -> changing the number of mentees assigned to you

It’s recommended to start on the "Average" weighting - > Start with "Average" (default)

Thanks @Etonkovidova – checking beta now I can see some updated wording on the status dropdown and tooltip that @MMiller_WMF provided along the same lines as your recommendations.

Module
image.png (892×1 px, 107 KB)
Status dropdown
image.png (368×548 px, 38 KB)
Tooltip
image.png (502×718 px, 80 KB)

I have noticed an unrelated issue where the "i" info icons on beta are showing the smaller icon instead of the 20x20px one in use on the newcomer homepage, but else this particular task is good to close.

Test wiki on Patch demo by Martin Urbanec (WMF) using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/d9c47f82ca/w/