Page MenuHomePhabricator

[Request] Support needed to understand baseline satisfaction levels for moderator tools
Open, HighPublic

Description

Please provide all the following information:

  • Context. We would like to improve moderator satisfaction in the moderation experience so that we improve moderator retention and make it more appealing to become a moderator. This serves KR WE1.3 "Increase user satisfaction of 4 moderation products by 5pp each" as well as PES1.2, because it will help us measure the impact of our wishlist implementations. In order to do that we'd like to establish a baseline understanding of moderator satisfaction today, so that we can track improvements of our work by the end of the fiscal year.
  • Description. Rather than measuring moderator satisfaction overall, which might be hard to impact by minor investments here and there by 1-2 teams, we hope to use QuickSurveys or similar to poll users of specific products we’re investing in to understand user satisfaction in that product specifically, tracking how it changes in response to our work.

Note from KC: "The overall satisfaction should not be a direct measure, where we survey all users of all the tools we plan to improve together. Instead, as Mikhail suggested in another thread, there can be micro-surveys to measure pre/post satisfaction of users for specific tools, which can then be used to calculate the overall satisfaction towards the end of the year." slack thread

We will use the Wishlist to determine the tools that would be most impactful to improve, which will be a contionuous process throughout the quarter, so we don't have a clear picture yet of which tools will be included in our top 4. One tool we could start with is AbuseFilter (see notes in this sheet).

  • Expected Deliverable. A survey of sorts that can allow us to understand baseline satisfaction of a given moderator tool so that we can track how much we improve satisfaction by over the course of the year. Ideally we would be able to draw from the first survey to inform questions we might ask in subsequent surveys.
  • Estimated Effort. If it's not possible to have the same survey for all tools, this will mean that we'll require continuous support to create new surveys as we identify the most impactful tools to improve.
  • Priority Without support for this work, we will not be able to measure success for WE1.3 because we currently don't have a clear understanding of moderator satisfaction levels.

I need this task resolved in:

	[x ] Q1 (knowing that we have KC pre-approved to support the Moderator Tools team throughout Q1.
	[ ] 6 months. 
	[ ] Whenever you get to it :-)
  • Other. There might be a dependency here with the task to define the term "Moderator". Happy to discuss further!

For use by WMF Research team; please leave everything below as it is:

  1. Does the request serve one of the existing Research team's audiences? If yes, choose the primary audience. (1 of 4)
  2. What is the type of work requested?
  3. What is the impact of responding to this request?
    • Support a technology or policy need of one or more WM projects
    • Advance the understanding of the WM projects.
    • Something else. If you choose this option, please explain briefly the impact below.

Details

Due Date
Fri, Dec 20, 12:00 AM

Event Timeline

@SonjaPerry I acknowledge that we have received your request. My understanding is that you and Miriam will connect in the coming week. She will ask some questions from you to help inform sizing and potential prioritization on our end.

Given that the quarter has already started and folks have already have picked up work for the quarter, I can't guarantee that we can prioritize this in Q1 on the Research end. I guarantee that we try to look at options.

Thank you, @leila! Just a quick note that this is also in support of PES1.2. Tagging @JWheeler-WMF here as well, since this will impact how we can measure the impact of wishlist implementations. Updated the description as well to include this KR.

@SonjaPerry thanks for the chat on Tuesday!

As discussed, there are three stages involved in calculating a survey-based baseline for moderator satisfaction. Survey design; survey deployment; and survey result analysis.

  • @YLiou_WMF will support the survey design, sampling and questions. We are happy to help you with this task in Q1.
  • As agreed in our meeting, the Moderator Tools team can work on the implementation and deployment of the survey.
  • While we initially thought that Product Analytics could work on the analysis of the survey responses, @mpopov noted in a separate conversation that they don't have capacity for this project. I am wondering whether there is an opportunity for the Moderator Tools team to take this step onboard, as this (or a very similar) survey will be running every quarter. We are happy to provide some guidelines and support for the first iteration of course. This would remove the dependency to other teams for a quarterly survey analysis. Please let us know your thoughts, and happy to sync again next week on this.

Assigned to @YLiou_WMF on our end. The scope and needs were clarified in a call yesterday. @YLiou_WMF please use this task for any updates.

Miriam triaged this task as High priority.Aug 15 2024, 5:54 PM
Miriam moved this task from Backlog to In Progress on the Research board.
Miriam set Due Date to Fri, Dec 20, 12:00 AM.Aug 16 2024, 9:10 AM

Weekly update:

  • Reminder that I will be OoO until September 3
  • Met with PES 1.2 stakeholders to discuss possible paths forward for survey research. Summary of discussion/ action items for survey:
    1. Followup meeting with @Samwalton9-WMF set for September 4
    2. Raised possibility of combination of this survey (given overlap in population of interest, substantive research questions around satisfaction/burnout, timing, and likely survey method) with SDS 1.2.2 survey research (see T371135) with Eli (SDS 1.2.2 hypothesis owner) who will engage in some follow-up conversations to determine how and if this could work.
      • likely alternative would be to run a short, lightweight survey for PES 1.2 to avoid possibility of survey exhaustion
    3. Determined that survey will likely have to be sampled via mass message to talk pages of relevant moderators (since QS cannot currently sample specific users).

Weekly update:

  • Met with @Samwalton9-WMF to provide update on potential integration with SDS 1.2.2 survey work tracked in T371135: unlikely to be feasible, but we will continue to coordinate these projects
  • Sharing last iteration of the Developer Satisfaction Survey as starting point for thinking through how to measure current satisfaction levels with moderator tools

Weekly update:

  • Met with Sam Walton to share initial survey draft and receive feedback and basic survey parameters
  • Nuke has been confirmed as tool of interest