Page MenuHomePhabricator

Scale: GrowthExperiments wiki monitoring dashboard
Open, Needs TriagePublic


As we scale to more wikis, it may be helpful for us to monitor several health indicators for each wiki. Because Growth features have a lot of project-specific dependencies, like mentors and templates, we want to notice when a specific project is having a problem.

This could take the form of a dashboard that could be publicly facing. It could be a wiki page or a spreadsheet or something else.
Here are some of the things we would want to know for each wiki:

  • Number of mentors signed up
  • Number of mentors who are answering questions at a lower rate or slower pace than some threshold (@Urbanecm has a script that parses talk pages to count responses. Spreadsheet output here).
  • Number of tasks available for each task type, for each topic
  • Number of suggested edits completed per newcomer
  • Revert rate of suggested edits
  • Number of mentor questions per newcomer
  • Which features are enabled or disabled
  • Number of errors, or whether parts of the feature are broken
  • The page could expose the current configuration, so we can visually check if it seems wrong

Event Timeline

Tgr created this task.Apr 11 2020, 9:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 11 2020, 9:14 AM
Tgr added a comment.Apr 11 2020, 9:26 AM

There are two (not necessarily exclusive) ways that come to mind, a per-wiki special page and a Toolforge tool showing data for all wikis in a single view. It seems nice to be able to have an overview of all wikis so maybe Toolforge is the better option.

I don't think there's a nice way to find out which wikis have suggested edits enabled, but we could fetch the growthexperiments dblist from some git mirror, and see if the API endpoint exists for each.
After that, it's just 6 API queries per site, or a few hundred if we want to get the full (task type X topic) matrix. That's a lot so some form of caching would have to be involved.

MMiller_WMF renamed this task from GrowthExperiments task count dashboard to Scaling: GrowthExperiments wiki monitoring dashboard.Apr 21 2020, 3:57 PM
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF added subscribers: Catrope, kostajh, marcella and 5 others.
Restricted Application added a project: Growth-Team. · View Herald TranscriptApr 21 2020, 3:57 PM
MMiller_WMF renamed this task from Scaling: GrowthExperiments wiki monitoring dashboard to Scale: GrowthExperiments wiki monitoring dashboard.Apr 21 2020, 3:57 PM
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF added a subscriber: Urbanecm.
MMiller_WMF updated the task description. (Show Details)May 18 2020, 5:53 PM

Potentially we could leverage some initial exploration done for a similar project, the Community Health Metrics Kit. For this project, @alexhollender also created a prototype of how information could be imported into a dashboard in Google spreadsheet as an option.

Dashboard sheet eg
Compare projects eg
Compare metrics eg
Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 1:44 PM
MMiller_WMF added a comment.EditedOct 5 2020, 5:30 PM

Idea from retro today (@kostajh @Catrope): perhaps the first version of a dashboard could allow users to input templates and see how many tasks will be available per topic with those templates. Then they could export that as a start to their on-wiki configuration for newcomer tasks. Naturally, we would want this for wikis that do not have our features enabled yet.