Page MenuHomePhabricator

Survey moderation product users on their satisfaction with the product
Closed, DeclinedPublic

Description

As part of our planning for next year (FY24/25) we have drafted the following KR for WE1.3:

Increase user satisfaction in 4 moderation products by X%.

If we are to understand whether we increased satisfaction in certain products, we need to know what the baseline numbers are. If we take a broad approach this may also help us prioritise which products to focus on - which tools are moderators currently least satisfied with?

Possible products/interfaces in which to survey

  • AbuseFilter
  • Watchlist
  • RecentChanges
  • Spam blacklist
  • FlaggedRevisions
  • PageTriage
  • Page protection
  • Page deletion
  • Revision deletion (Special:RevisionDelete)
  • Mobile web interfaces
  • ...

Additionally, although out of scope for 'content moderation', Trust & Safety Product would find it valuable to collect the same data for the following products:

  • Special:Block
  • Special:CheckUser
  • Special:Investigate
  • ...

Open questions

  • How can we best survey users of these tools - QuickSurveys?
    • If QuickSurveys, what decisions do we need to make. There are a wide range of sampling options.
    • Note that QuickSurveys currently only works on article pages (T360996)
    • We also get less targeting with QuickSurveys - we might be better defining our list of patrollers/admins and contacting them directly.
  • What should we be asking?
  • Which tools should we survey? A partial list has been started above.

Event Timeline

In our team meeting yesterday we discussed this a little and noted that for some tools, or in some communities, QuickSurveys may be overkill. If a wiki only has a few administrators it would be frustrating for them to receive multiple surveys about different tools rather than just getting one which asks about all of them in one place. Additionally, even on a big wiki there might only be a handful of users who use, for example, AbuseFilter, so we could easily get the list of active users and message them directly.

After further discussions about this draft KR, we determined that this would be best achieved by creating a standardized methodology for collecting information about product satisfaction, and user-level/community-level product adoption. This would provide an advantage in reducing the amount of repeated preparatory work (e.g. reusable versus bespoke privacy statements, transferrable localization) and allow us to better cross-compare adoption and satisfaction across different products. The current intent is to collect fairly high-level satisfaction/adoption metrics, and fine-tune these on a per-project basis.

This does mean that QuickSurvey is unlikely to be the main tool for collecting survey responses, due to its current limitations.

After I meet with Claudia today, we'll wait until the request comes from the RDS process.
After that, Management will decide how to proceed.