Page MenuHomePhabricator

[Request] Understanding how volunteers "program policy" and their perception of Edit Check
Closed, ResolvedPublic

Description

Please provide all the following information:

  • Context. Provide a short paragraph with some background context for your request, please include links to relevant material.

Edit Check is a new feature designed to guide contributors during the editing process by flagging policy-relevant issues and suggesting fixes. While its current implementation focuses on specific pillars (e.g. references, copyright, non-neutral tone), the broader ambition is to create a flexible framework that reflects community values and norms. However, Wikimedia policies are not always formalized or uniformly interpreted – they’re often “programmed” by volunteers through templates, bots, edit notices, and community consensus. We want to understand how contributors currently translate policy into user-facing interventions, and how they perceive Edit Check as a new potential layer of that process.

  • Description.What is your request about?

We’d like to explore how experienced contributors currently think about encoding policy into tools (e.g. templates, bots, edit filters), and gather reactions to Edit Check as a new layer in that ecosystem. What do they value in current approaches? What limitations do they see? How do they feel about machine-assisted moderation that intervenes in real time?

This research should also investigate the broader cognitive models contributors use when enforcing policy: do they think in terms of rules, social negotiation, precedent, etc.? How might Edit Check fit into that mental framework?

  • Expected Deliverable. What is the ideal outcome or result of your request?

a/ A synthesis of current methods volunteers use to encode or enforce policy. Including, but not limited to, how they decide what policies/guidelines to program, what tools they use to develop and test these scripts, and how they evaluate their impact/effectiveness
b/ Set of recommendations for the capabilities/requirements experienced volunteers would need to see in order to be motivated to author new Checks and Suggestions
c/ Summary of what motivates people to do this kind of work and how they arrived to it.
d/ Perceptions of Edit Check as a tool for shaping behavior

  • Estimated Effort. Please provide an estimate of the amount of work needed to complete this task, if known.

This project will involve defining research goals, recruiting experienced contributors, conducting a number of interviews, synthesizing insights, and sharing findings in a report.

  • Priority Please indicate a priority of your task and a small description of what it would unlock for you. We ask you to leave this task as “needs triage” since your request will go through a Backlog refinement process where our team will prioritize the work.

I need this task resolved in:

  • 1 month.
  • 3 months.
  • 6 months.
  • Whenever you get to it :-)
  • Other. Do you have any other questions or comments ?

This research would unlock the next phase of Edit Check by providing a structural foundation for future checks and informing how we open up authoring capabilities to community members. It ensures that design and technical decisions align with contributor mental models and existing policy-encoding practices.


For use by WMF Research team; please leave everything below as it is:

  1. Does the request serve one of the existing Research team's audiences? If yes, choose the primary audience. (1 of 4)
  2. What is the type of work requested?
  3. What is the impact of responding to this request?
    • Support a technology or policy need of one or more WM projects
    • Advance the understanding of the WM projects.
    • Something else. If you choose this option, please explain briefly the impact below.

Details

Due Date
Oct 1 2025, 5:00 PM

Event Timeline

Thanks Nico and Peter! I like this new way that you have framed the open questions - they make sense to me and seem like actionable approaches to inform Edit Check. ("how do experienced contributors currently think about encoding policy into tools? What do they value in current approaches? What limitations do they see? How do they feel about machine-assisted moderation that intervenes in real time?)

The part about how to ask about machine-assisted (i.e. AI-powered moderation) I have an idea on how to approach asking about in a useful way based on an approach I read about in an external research paper...I will share next week.

You all can meet at the time you scheduled on June 17, to discuss further (I don't think I need to be there unless you want me to be there, Daisy).

Leila, for context - this is a research request we've been discussing with Peter & Nico in other meetings so far.

Thanks so much Debra! Very curious to hear more about the AI part you mentioned.
Looking forward to our call, Daisy!

Here is the approach - it's from a paper where they were investigating the use of AI in a totally different context (algorithmic evaluation of employees at a workplace, for performance evaluation purposes) -- but the way they went about asking about it is one we could emulate.

From the abstract: "to avoid employees’ antipathy toward AI, it is important to understand what aspects of AI employees like and/or dislike. In this paper, we aim to identify how employees perceive current human resource (HR) teams and future algorithmic management. Specifically, we explored what factors negatively influence employees’ perceptions of AI making work performance evaluations. Through in-depth interviews with 21 workers, we found that 1) employees feel six types of burdens (i.e., emotional, mental, bias, manipulation, privacy, and social) toward AI's introduction to human resource management (HRM), and that 2) these burdens could be mitigated by incorporating transparency, interpretability, and human intervention to algorithmic decision-making. Based on our findings, we present design efforts to alleviate employees’ burdens." https://dl.acm.org/doi/abs/10.1145/3411764.3445304

If you read the Methods section of the paper, you can see they use a "scenario-based" interview approach to elicit more specific and useful feedback, based on specific examples of how AI might be used, in the context they were studying. They also list out the specific interview questions that they asked, which I think we could also use. (You can access the full-text paper via ResearchGate, if you search the paper's title in a search engine and go to the ResearchGate link).

These same researchers also did a followup paper in which they describe how they did participatory workshops with both 'stakeholders' of the AI system they were designing, to identify and discuss solutions to the areas where their values were in tension. This could also be relevant for us as an approach. https://dl.acm.org/doi/10.1145/3491102.3517672

Initial call last week with Peter and Nico, dug into background for the request and what is the core ask wrt actionability. Established that of the expected deliverables, core ask is C), screenshots of likely/potential future checks will be provided for D). Interview discussion guide will address elements A and B wherever possible.

Research brief in progress and initial DR manager and team review will occur after my OOO (returning 7/10; delivery of brief week of 7/14).

Shared research brief with team 7/22 and discussion guide is in progress. Pending confirmation on a couple clarifications on targeted wikis and recruitment support.

checked in with Sonja last week to clarify goals/brief details, followed up with team ambassadors/analytics support for recruitment details, received back legal-reviewed privacy policy and release form.

still waiting on ambassadors' response, but have some preliminary help from analytics; depending on available pool, may need to find ways to widen recruitment. finalizing discussion guide and starting on recruitment outreach this week.

additional recruitment occurring this week with a finalized list from analytics. attending meeting with ambassadors (most will be in attendance after wikimania/travel) this week to see if there are other means of supplementing recruitment from their perspectives. other project staging logistics ongoing.

Quick PA update:
We received a two item batch request; this was the second item.
I met with @dchen on 08/06 and and began developing the query for the supporting material once T395678 was completed.
After drafting the query, I shared it with RDS teammates for review.
An updated request came in on 08/11, and the final material was delivered on 08/15 after security engineering's approval and DM fine-tuning in Slack.
@dchen's feedback: "the final looks good, i will let you know if anything else comes up that could help refine any more."

hoping to schedule sessions soon, and continuing to send out recruitment outreach.

DKumar-WMF raised the priority of this task from Medium to High.Sep 2 2025, 3:41 PM
DKumar-WMF set Due Date to Sep 30 2025, 5:00 AM.

first sessions completed and additional sessions occurring this week and next. if desired, please view via my calendar events!

6 sessions completed. anticipating 3 more this week to conclude. please feel free to view any of interest on my calendar event attachments.

9 sessions completed! beginning analysis/synthesis this week.

completing analysis/synthesis this week and working on final report.

dchen changed Due Date from Sep 30 2025, 5:00 AM to Oct 1 2025, 12:00 AM.Sep 30 2025, 4:09 PM
dchen changed Due Date from Oct 1 2025, 12:00 AM to Oct 1 2025, 5:00 PM.

checked in with Peter; confirmed okay to complete some remaining report clean-up and content rearranging into 10/1.

final report completed and shared last week and will schedule a follow-up Q&A session with the team soon.

Shareout to the team completed on Oct 29.