Page MenuHomePhabricator

Scale: projections for scaling the help panel
Closed, ResolvedPublic

Description

As we plan to scale to more wikis, one consideration is whether help desks on large wikis will be overwhelmed by questions from many newcomers. To estimate what large wikis will experience, we want to extrapolate from our target wikis. For each of our four target wikis, we want to have quick and basic estimates of these things from @Dyolf77_WMF @PPham @Urbanecm @PPham:

  • How many questions come from the help panel to your help desk per day/week? This means counting new headers that get posted from the help panel.
  • How many different experienced users were active on your help desk during the month of March? In other words, this is simply counting how many different usernames of question answerers edited the page. This counts people answering any kind of question on the help desk, not just help panel questions.

And from @Trizek-WMF, we want to come up with these numbers for French, Portuguese, and Italian Wikipedias (as examples of big wikis):

  • How many questions come to the help desk per day/week?
  • How many different experienced users were active on the help desk during the month of March?

Event Timeline

@Trizek-WMF @PPham @Urbanecm @revi @Dyolf77_WMF -- there are questions for each of you on here. Could you please post what you find in a comment by Wednesday of next week (April 8)? Thank you!

Dyolf77_WMF moved this task from Backlog to In progress on the User-Dyolf77 board.
MMiller_WMF renamed this task from Scaling: basic help panel stats to Scale: basic help panel stats.Apr 3 2020, 10:17 PM

According to https://quarry.wmflabs.org/query/43552, we had 30 questions in March 2020 at cswiki. That gives us about 1 question per day. Surprisingly, according to https://quarry.wmflabs.org/query/43551, we had 68 questions in 2020, which gives us 0,7 question per day.

According to https://quarry.wmflabs.org/query/43553, in 2020, 50 users answered a question at the helpdesk. According to https://quarry.wmflabs.org/query/43554, they saved 206 edits (not necessarily same as number of question answered). Those numbers include all questions and not just helppanel ones.

In March 2020, 28 different users answered a helpdesk question (https://quarry.wmflabs.org/query/43555), and they saved 71 edits (https://quarry.wmflabs.org/query/43556).

Let me know if you have any other questions.

PPham added a comment.EditedApr 4 2020, 3:54 AM

How many questions come from the help panel to your help desk per day/week?
Yesterday viwiki had 3 questions, and this week we had 12 questions from the help panel. While this whole week we only had 1 question not from the help panel.
In March 2020 we had 42 questions.

How many different experienced users were active on your help desk during the month of March?
3, including me. But even I didn't answer many questions, as the majority of them are non-related.

For ar.wiki:

  • 10 new questions every day, 70 per week.
  • 9 different users were active last month to handle the help desk.
revi added a comment.Apr 8 2020, 8:41 AM

For march (Revision 26164172):

  • Total number of topics: 252
  • Originating from Help Desk: 32
    • Week 1: 5
    • Week 2: 7
    • Week 3: 9
    • Week 4: 10
    • Week 5: 1
  • Number of editors answering: more than 15 or so, but this includes the "children" asking and answering themselves. The children's answer quality is generally poor. Excluding them bring the number down to 2 or 3.
Trizek-WMF triaged this task as Medium priority.Apr 8 2020, 8:23 PM
Trizek-WMF moved this task from Incoming to In Progress on the Growth-Team (Current Sprint) board.
FrenchItalianPortuguese
How many questions come to the help desk per month?18110545
How many different experienced users were active on the help desk during the month of March?20188

I went by month for the number of new questions.
Concerning Italian Wikipedia, the place is not dedicated to newcomer's questions and some questions there are asked by already familiar with the wiki users as well.

MMiller_WMF renamed this task from Scale: basic help panel stats to Scale: scaling the help panel.Apr 13 2020, 11:37 PM

Thank you all for these numbers. I've used these to try to project what we can expect when turning on the help panel and mentorship features in large wikis. I have two follow-up questions:

  • @Dyolf77_WMF: arwiki has many more help panel questions per answerer (31 per answerer per month) and many more mentorship questions per mentor (35 per mentor per month) than the other wikis. Is there a general sense that this is too many, and help desk answerers and mentors are overloaded? I'm trying to figure out what the right ratio is.
  • @Urbanecm: you said that in March, cswiki had 30 help panel questions and 28 help desk answerers. Are there really that many answerers? That sort of means that each question is answered by a different person. Or is it that multiple answerers answer the same question?

For the projects, I used the average numbers of questions and mentors across the four target wikis, along with @Trizek-WMF's numbers for the big wikis. Here's what it looks like to me (and here's my spreadsheet with the arithmetic):

  • If we leave the features the way they are now, here's how many help desk answerers and mentors the big wikis would need.
    • French: 53 help desk answerers (up from 20) and 46 mentors (up from 25)
    • Italian: 38 help desk answerers (up from 18) and 19 mentors (but they already have 32)
    • Portuguese: 36 help desk answerers (up from 8) and 25 mentors (unclear how many they have now)
  • If we convert the help panel over to be about mentor questions instead of going to the help desk, these may be the projections:
    • French: 69 mentors (up from 25)
    • Italian: 28 mentors (but they already have 32)
    • Portuguese: 37 mentors (unclear how many they have now)

So the question is back to @Trizek-WMF -- do you think it is reasonable for those wikis to get mentor lists that big? I'm worried that otherwise the existing people will be overwhelmed.

Hi @MMiller_WMF

  • @Dyolf77_WMF: arwiki has many more help panel questions per answerer (31 per answerer per month) and many more mentorship questions per mentor (35 per mentor per month) than the other wikis. Is there a general sense that this is too many, and help desk answerers and mentors are overloaded? I'm trying to figure out what the right ratio is.

Absolutely this is too many for the 16 volunteers registered in the Help project. In the past 2 days, an ansewerer handled about 19 questions and was the only active to answer. He created an account 4 months ago and IMO he will burn out quickly and will not be so efficient like now. Maybe this is a factor of the ar.wiki overload.

If we look closer to the questions we can find many conversations look like live chats. When newcomers ask questions they think they will get an instantaneous reply. Not all helper can be patient,
I will discuss with volunteers about this note this week and will come back here for updates.

MMiller_WMF renamed this task from Scale: scaling the help panel to Scale: analysis for scaling the help panel.Apr 14 2020, 9:51 PM
MMiller_WMF renamed this task from Scale: analysis for scaling the help panel to Scale: projections for scaling the help panel.

We aren't sure if the number of needed mentors will be equally proportional as on other wikis. We need to confirm the projections.
Also, big wikis have enough active users to theoretically easily increase their mentors workforce.

Maybe we should try with one big wiki first, and adapt our communication when we target more wikis.

RHo added a comment.Apr 15 2020, 2:41 PM

If we look closer to the questions we can find many conversations look like live chats. When newcomers ask questions they think they will get an instantaneous reply. Not all helper can be patient,

Perhaps we can post a more explicit "Note: This is not live chat. Please note the average response time is $value." message above the text area field.

MMiller_WMF closed this task as Resolved.May 4 2020, 3:42 AM

These projections are complete. We'll be working on T250235: Scale: pilot help panel with mentorship to take action on them, and then re-evaluate.