Page MenuHomePhabricator

Scale: pilot help panel with mentorship
Open, Needs TriagePublic

Description

As we think about how we'll scale the help panel to larger wikis (see T249375), one idea is to convert the help panel to ask mentor questions instead of help desk questions. This would simplify our process to deploy to more wikis, because we wouldn't need to worry about help desk capacity, how help desks are archived, etc. Therefore, we want to decide on this before deploying to too many more wikis.

There are several reasons that it might be a better idea for the help panel's "question asking" feature to ask questions to the newcomer's mentor instead of to a central help desk:

  • When the original newcomer homepage was deployed, users had equal opportunity to ask a question to their mentor and ask a question to the help desk. Almost all of the question askers chose their mentor. Perhaps asking a mentor is much more appealing than asking a faceless help desk.
  • It is easier to scale mentorship capacity up and down than to scale help desk capacity. To scale mentorship capacity, we can recruit more mentors to sign up on the list. They'll know they are getting asked questions because they'll get an alert every time they get one. But for the help desk, it relies entirely on sufficient number of people watching and engaging the help desk.
  • We eventually intend to tie mentors in more closely to the suggested edit experience, like by allowing newcomers to notify their mentor when they complete an edit. This would be a step in that direction.

The main downside is:

  • Newcomers may get answered more quickly on the help desk than through their mentor. But we don't have evidence that speed is more important than the personal touch. Both the help panel and homepage experiments did not show increases in activation or retention.

Before we convert all help panel over to mentorship outright, it may be best to pilot that on one wiki. The reason is that we're not sure if it will lead to a dramatic increase in mentorship questions, and so we want to have a sense of how the volume changes. Our pilot will be on Czech Wikipedia. We will want to try it out there for about two weeks, and see how much volume comes through before deciding whether to keep the change for other wikis.

Below is what we want to do on Czech Wikipedia. If some of these specifications will take more time than others, we can exclude some of them for the pilot.

  • Convert over the help panel to direct questions to the user's mentor's talk page, instead of to the central help desk.
    • The large button to ask a question would be changed to have much of the same information and icon as the mentorship module on the homepage, like this:
    • After clicking that button, the next screen should contain the same content as the mentorship module (shown below), but should still contain the option to include the article title. The header should also change from "Ask a question" to "Ask your mentor".
  • The header of the new section that ends up on the mentor's talk page should contain the article title (if the user has that option selected. Whereas mentorship questions from the homepage have headers like "Question from Cloud atlas (22:20, 14 April 2020)", these should have headers like "Question from Cloud atlas on Ada Lovelace (22:20, 14 April 2020)".
  • These questions should show up listed on the homepage in the mentorship module along with the questions the user asked from the homepage (not required for pilot).
  • No changes to instrumentation are needed. We will be able to understand behavior changes attributable to this pilot based on timestamps in EventLogging.
  • Instead of having the existing "Help panel question" or "Mentorship module question" edit tags, these should have a new edit tag: "Mentorship panel question".

Details

Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@RHo -- could you read this over and see if I'm missing anything? And see if you think this is a good plan?

@Catrope @kostajh @Tgr -- I'm interested in your first impressions of this, whether this idea is easier said than done.

This seems like it would be relatively simple, at most a little awkward in places (exposing awkwardness in existing code). We'd just have to change the icon, preview card and text for the ask-help panel in the help panel depending on a config setting, and post to the mentor's talk page instead of the help page somehow. The latter could be done by lying to the API module and sending source=homepage-mentorship instead of source=helppanel, or by having the API module route source=helppanel to MentorshipModuleQuestionPoster instead of HelpPanelQuestionPoster if the setting is enabled, or by having HelpPanelQuestionPoster post to the mentor's talk page if the setting is enabled; they're all a bit awkward in different ways and it's not immediately clear to me which would be better.

RHo added a comment.Apr 15 2020, 2:25 PM

hi @MMiller_WMF - this SGTM in principle, but I had a question. In the task description it says the change in volume during the pilot will determine whether to roll out more widely, but might we also want to look to see if the 'personal appeal' leads to changes in activation and retention rates? If so, then (a) would this affect instrumentation needed, and (b) require extending the pilot to be longer than two weeks?

@RHo -- the way I'm thinking about is that as long as the change in volume is not overwhelming or troubling in some other way, then we will want to convert all wikis over to mentor questions. In other words, the bar is lower than evaluating activation and retention -- but that could be an experiment to run in the future, once we have more volume of newcomers: measuring whether help panel w/ mentorship has an impact compared to help panel w/ help desk.

RHo added a comment.Apr 15 2020, 8:35 PM

@RHo -- the way I'm thinking about is that as long as the change in volume is not overwhelming or troubling in some other way, then we will want to convert all wikis over to mentor questions. In other words, the bar is lower than evaluating activation and retention -- but that could be an experiment to run in the future, once we have more volume of newcomers: measuring whether help panel w/ mentorship has an impact compared to help panel w/ help desk.

Makes sense, thanks for clarifying. Plan sounds good and can't see anything amiss. 👍🏻

Trizek-WMF added a comment.EditedApr 17 2020, 12:43 PM

As suggested elsewhere by Rita, we should tell newcomers that asking their mentor is not a live chat, and that they may have some delays concerning the reply.

MMiller_WMF updated the task description. (Show Details)Apr 27 2020, 8:45 PM
Tgr added a comment.Jun 1 2020, 6:08 PM

Bonus objective: get rid of the weird relationship between HelpPanelProcessDialog and QuestionPoster. The help panel already has the ability to be locked into a sub-panel, and we'll need to make the mentor sub-panel available everywhere for this task, so we can probably replace QuestionPoster with the combination of those two things. Logging works differently on the homepage, so that will need some thought.

Bonus objective: get rid of the weird relationship between HelpPanelProcessDialog and QuestionPoster. The help panel already has the ability to be locked into a sub-panel, and we'll need to make the mentor sub-panel available everywhere for this task, so we can probably replace QuestionPoster with the combination of those two things. Logging works differently on the homepage, so that will need some thought.

Proposed solution for this in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/604704

Change 604704 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Help panel: Refactor mode tracking & remove question poster dialog

https://gerrit.wikimedia.org/r/604704

Change 604659 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] WIP: Help panel guidance behavior rule updates

https://gerrit.wikimedia.org/r/604659

Change 604704 abandoned by Kosta Harlan:
Help panel: Refactor mode tracking & remove question poster dialog

Reason:
Moved this into the parent patch

https://gerrit.wikimedia.org/r/604704

Change 604659 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Help panel: Remove locked mode and refactor code

https://gerrit.wikimedia.org/r/604659

@Urbanecm -- we will be working on this soon. Could you please follow along so that you can announce this change a week before it is deployed?

Tgr claimed this task.Jul 3 2020, 12:27 AM

@Urbanecm -- we will be working on this soon. Could you please follow along so that you can announce this change a week before it is deployed?

Sure!

Change 613714 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Rename help panel question-related functionality

https://gerrit.wikimedia.org/r/613714

Change 613715 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] QuestionPoster: Add new helppanel->mentor question type

https://gerrit.wikimedia.org/r/613715

Note: we would like this to be deployed during the week of August 3, and not sooner. Since it's behind a config flag, we have options for when we can deploy.

@MMiller_WMF Setting a reminder for myself for Jul 28. I'll check with you in advance prior to sending anything.

Change 613714 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Rename help panel question-related functionality

https://gerrit.wikimedia.org/r/613714

Change 613715 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] QuestionPoster: Add new helppanel->mentor question type

https://gerrit.wikimedia.org/r/613715

Change 615524 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Help panel: feature flag for asking mentors instead of the help desk

https://gerrit.wikimedia.org/r/615524

Change 615524 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Help panel: feature flag for asking mentors instead of the help desk

https://gerrit.wikimedia.org/r/615524

Change 615571 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments (beta): Send help panel to mentors on cswiki in labs

https://gerrit.wikimedia.org/r/615571

Change 615571 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments (beta): Send help panel to mentors on cswiki in labs

https://gerrit.wikimedia.org/r/615571

For testability, I've enabled the feature flag for this feature on cswiki in beta labs. It will stay disabled in production until we decide to enable it (which per T250235#6320257 we won't do before August 3rd)

I missed that some of the post-submission text must be different as well, I'll make a follow-up patch.

Also T258621: [regression] frwiki - Mentorship module is missing for old user accounts made me realize some users with the help panel might have no homepage, and thus no mentor. Not sure how to handle that... we could either fall back to the helpdesk, or try to just automatically assign a mentor. (Which we'd have to attempt on basically any pageview - is there any disadvantage to that? If there's a race situation, the user might see a different mentor from whom they actually end up assigned to - that would be confusing, but rare.)

Tgr added a comment.Jul 25 2020, 6:09 PM

Also, it seems the title parameter is set when asking from the homepage, which shouldn't happen.

Change 616339 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Introduce MentorManager and virtually assign mentors to all users

https://gerrit.wikimedia.org/r/616339

Change 616450 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Do not send the relevant title when asking a mentor from the homepage

https://gerrit.wikimedia.org/r/616450

Change 616450 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Do not send the relevant title when asking a mentor from the homepage

https://gerrit.wikimedia.org/r/616450

@Urbanecm -- we will be working on this soon. Could you please follow along so that you can announce this change a week before it is deployed?

Note: we would like this to be deployed during the week of August 3, and not sooner. Since it's behind a config flag, we have options for when we can deploy.

@MMiller_WMF Setting a reminder for myself for Jul 28. I'll check with you in advance prior to sending anything.

Any update on timeframe?

Any update on timeframe?

I've reviewed the pending patch and left comments. I think we'll likely be able to merge this in time for wmf.3.

Change 617261 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Fix help panel ask screen wording

https://gerrit.wikimedia.org/r/617261

Change 616339 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Introduce MentorManager and virtually assign mentors to all users

https://gerrit.wikimedia.org/r/616339

Change 617261 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Fix help panel ask screen wording

https://gerrit.wikimedia.org/r/617261

@Tgr Testing in betalabs cswiki - - there is weird issue with the Mentorship dialog showing the checkbox every other load (like a load balancer serving two types of different code), see the animated gif below:

It happens on Special:Homepage, SE articles and regular articles (in Edit mode, of course). And posting a question through the dialog without the checkbox and through the dialog with the checkbox would result in totally different postings on a mentor's Talk page:

@Tgr Testing in betalabs cswiki - - there is weird issue with the Mentorship dialog showing the checkbox every other load (like a load balancer serving two types of different code), see the animated gif below:

Ohoho, I'm curious about this one :).

It happens on Special:Homepage, SE articles and regular articles (in Edit mode, of course). And posting a question through the dialog without the checkbox and through the dialog with the checkbox would result in totally different postings on a mentor's Talk page:

Those are not totally different. First one is Czech for "Question from ET13", this functionality added another message, which I did not translate to Czech yet. The only difference is the software doesn't see the checked checkbox.

It happens on Special:Homepage, SE articles and regular articles (in Edit mode, of course). And posting a question through the dialog without the checkbox and through the dialog with the checkbox would result in totally different postings on a mentor's Talk page:

Those are not totally different. First one is Czech for "Question from ET13", this functionality added another message, which I did not translate to Czech yet. The only difference is the software doesn't see the checked checkbox.

Thanks, @Urbanecm - for some reason I thought that the messages were re-used and, as such, they all have been translated.

Change 618057 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Fix "include page title" checkbox

https://gerrit.wikimedia.org/r/618057

Change 618057 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Fix "include page title" checkbox

https://gerrit.wikimedia.org/r/618057

Re-checked, moving to Design Review.
To summarize

  • "Ask the help desk" dialog is is completely replaced by "Ask your mentor"
  • The "include" check-box is present on
    • SE articles "Ask your mentor" dialog

    • a page in Edit mode (previously Help panel)
    • Special:Homepage "Ask your mentor" won't have the check box
  • the questions still are posted with different titles due to @Urbanecm's comment - the message was not yet translated. In the screenshot below "Question from ET13 on John O'Sullivan " is not translated since the check box for including the article title was checked:

Etonkovidova updated the task description. (Show Details)Aug 4 2020, 1:32 AM

Thanks @Etonkovidova - agree that functionally this seems all good. However, I noticed one minor design issue with the change to the "Ask a mentor" button in the help panel:

  • The font-styles should be swapped for the Mentor name and additional info <#> edits · <Last active time>

(this applies to mobile and desktop)

DesktopActual
Expected
MobileActual
Expected

Re-checked, moving to Design Review.
To summarize

  • "Ask the help desk" dialog is is completely replaced by "Ask your mentor"
  • The "include" check-box is present on
    • SE articles "Ask your mentor" dialog

    • a page in Edit mode (previously Help panel)
    • Special:Homepage "Ask your mentor" won't have the check box
  • the questions still are posted with different titles due to @Urbanecm's comment - the message was not yet translated. In the screenshot below "Question from ET13 on John O'Sullivan " is not translated since the check box for including the article title was checked:

Change 618327 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Fix "Ask mentor" help panel button styling

https://gerrit.wikimedia.org/r/618327

Change 618327 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Fix "Ask mentor" help panel button styling

https://gerrit.wikimedia.org/r/618327

Fixed - checked on desktop and mobile:

Perfecto, thanks!

Fixed - checked on desktop and mobile:

This looks good to me! We'll be ready to backport the style change and turn this all on in cswiki on Monday. Is that okay with you, @Urbanecm?

Change 618583 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.3] Fix "Ask mentor" help panel button styling

https://gerrit.wikimedia.org/r/618583

And to be specific, @Catrope will be flipping it on on Monday.

Change 618786 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Direct GrowthExperiments help panel questions to mentors on cswiki

https://gerrit.wikimedia.org/r/618786

Change 618583 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.3] Fix "Ask mentor" help panel button styling

https://gerrit.wikimedia.org/r/618583

Mentioned in SAL (#wikimedia-operations) [2020-08-06T18:20:19Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/modules/: fb4a80830d7d915479e097cc82c681c5fb03d51b: Fix "Ask mentor" help panel button styling (T250235) (duration: 01m 07s)

Change 618786 merged by jenkins-bot:
[operations/mediawiki-config@master] Direct GrowthExperiments help panel questions to mentors on cswiki

https://gerrit.wikimedia.org/r/618786

Mentioned in SAL (#wikimedia-operations) [2020-08-11T18:05:37Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Direct GrowthExperiments help panel questions to mentors on cswiki (T250235) (duration: 01m 03s)

This is working well in cswiki. I'm leaving this in my column so that we review the results after about 6 weeks.

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 2:08 PM

Change 634945 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Use UserOptionsUpdateJob for mentor assignment

https://gerrit.wikimedia.org/r/634945

Change 634945 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Use UserOptionsUpdateJob for mentor assignment

https://gerrit.wikimedia.org/r/634945

Tgr added a comment.Thu, Nov 5, 7:20 AM

Change 634945 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Use UserOptionsUpdateJob for mentor assignment

https://gerrit.wikimedia.org/r/634945

This is not working (example): setOption makes a master read even when it does not save anything. I suppose we'll have to upstream the whole "lazy set" concept into UserOptionsManager.