Page MenuHomePhabricator

Moderator tools baseline satisfaction surveys
Closed, ResolvedPublic

Description

The research team will provide work under KR WE 1.3 to measure baseline satisfaction among users of certain specific moderation tools. These surveys will then be followed by subsequent surveys of the same users to measure the impact of tool improvements introduced during FY 2024-2025.

Audience: We anticipate that this work will primarily serve Wikimedia Foundation Product teams, but may also assist Wikimedia affiliates' in their efforts to support moderation activities on the projects.

Impact: This work will support a technology need (improvements to moderation tools) on multiple Wikimedia projects and will also advance the understanding of the Wikimedia projects (measuring the satisfaction of users of existing and improved moderation tools).

Nuke tool satisfaction survey:

  • Tracked here: T376759
  • Timeline: baseline survey fielded in Q2, follow-up survey March 2025
  • Outcome measures: general nuke tool satisfaction, tool attributes satisfaction

Note: as of 9 May 2025, we have deprioritized the Recent Changes and Watchlist surveys so this task will now focus solely on the Nuke Survey.

Event Timeline

Weekly update:

  • Draft survey shared with stakeholders and programmed into Limesurvey project
  • Feedback received to allow for open-text responses on nuke tool attribute battery and to add items asking whether tool fulfills a stated purpose as well as an open text asking whether users use the tool for another purpose (if so, what).
  • Survey to be updated and reshared next week

Other moderator tools covered by KR WE1.3 still TBD

The two most likely tools here are Recent Changes and Watchlist. These are both broad, core MediaWiki, tools which we will have a much less specific userbase to survey.

Weekly update:

  • nuke tool survey has been updated to reflect requested changes
  • @Samwalton9-WMF I wanted to flag this update to T377324 - unsure how being unable to construct a metric for moderators / moderator actions may affect plans for surveys to support tool updates for Recent Changes and Watchlist

We're looking to start working on Recent Changes relatively soon - could we spin up a draft survey based on the Nuke one for that too? Perhaps it would make sense to combine this with Watchlist since we expect to work on that later in the year, and the outreach strategy is likely to be the same since they're both broadly used tools.

@Samwalton9-WMF: yes, absolutely can share this early next week!

Samwalton9-WMF updated the task description. (Show Details)

@Samwalton9-WMF: yes, absolutely can share this early next week!

Were you able to get started on this? :)

Miriam triaged this task as High priority.Nov 20 2024, 1:50 PM

Hi @Samwalton9-WMF ! Sorry this got lost in the rush before the RDS offsite!

Please see here for a draft survey script!

@Samwalton9-WMF: please see an updated draft survey script for Recent Changes/Watchlist here!

Update:

  • nuke tool survey has collected 61 complete responses (~33% complete:contact ratio)
  • recent changes/watchlist survey draft discussed with @Samwalton9-WMF with revision to be shared early next week. We now anticipate fielding this survey in early Q3.

Update:

  • nuke tool survey has been closed and is no longer accessible to respondents

Update:

  • Met with Sam and Olga to discuss:
    1. next steps and timeline for Nuke survey analysis and follow-up survey
    2. updates / timeline for watchlist / recent changes survey
  • Watchlist / Recent Changes english language version to be finalized by EOD Jan 13 and translations requested for German, French, Indonesian, Arabic, Swedish, Hungarian, Finnish, Urdu

Update:

  • We are still awaiting a privacy statement for this survey - it will field as soon as this is available

Update:

  • Privacy statement has been posted and Quicksurvey deployment is anticipated soon (tracked here: T389401)

Update:

  • Quicksurvey deployment has been delayed due to core experiences offsite prep. Expect deployment next week.
  • Still blocked on QuickSurvey deployment: now scheduled for April 24

Prioritization update:

  • The RecentChanges/Watchlist survey has been deprioritized as it's unlikely that the technical interventions will generate a change in the satisfaction metrics due to a focus on smaller wikis. Given that the RecentChanges/Watchlist survey has been deployed, we are going to leave it open for the standard window. We won't be prioritizing the analysis right now but believe that the data can still be valuable outside of measuring satisfaction changes (e.g., % of highly-active editors who say they do patrolling/moderation tasks). We will pick those basic analyses up perhaps in Q1 if we have more time. We won't run any follow-up surveys though.
  • This does not affect the Nuke surveys, which will still proceed as expected.

RecentChanges/Watchlist remain unprioritized though there was some confusion around that (T396250). Closing this task out as T396039: Nuke tool follow-up satisfaction survey captures the remaining work in this space.