**Draft sHypothesis**
If we develop heuristics-based data pipelines for measuring patrolling activity on Wikipedia, we can prototype a model to detect moderation gaps at scale.
**Scope**
- Wikipedia
- Largest language editions
- Main namespace (article pages)
- Non-bot editors (with high interest in newcomers)While focus is on newer moderators, measurement should cover all editor types. Lower priority on incorporating AbuseFilter (pre-publication filters) if that proves difficult.
**Product requirements**
There are no strict product requirements at the moment. However, the [[ https://www.mediawiki.org/wiki/Moderator_Tools | Moderator Tools ]] team is ideating a project to organize on-wiki activity in a centralized way, including moderation practices. Given this proposal is still in early stage, this should not become a blocker to start prototyping patrolling activity metrics. On the contrary, this research is aimed to proactively provide insights to guide Product.
**Draft output**
- Metrics for Wikipedia patrolling around frequency, editor experience and, optionally, language-agnostic article features (e.g., [[ https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Language-Agnostic_Quality | quality ]]Dataset of all edits for a month that captures the following:
-- Edit:
--- What was changed (we will start with issue templates from Q3 work and can add further functionality if needed and feasible)
--- Who performed the edit (edit count, [[ https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Importance | importance ]], [[ https://meta.wikimedia.org/wiki/Research:Evaluating_list_building_tools_for_ad-hoc_topic_models | topic ]]user rights)
--- Edit review status (at time of snapshot)
-- Reviewer:
--- Who "reviewed" or followed up on the edit (edit count, etc.)user rights) where applicable
- Data-driven strategy to identify candidates among editors for a newer moderator (patrolling) model-- Time to review
--- Time to next edit
--- Policies cited in edit summary if reverted
- Superset dashboard that allows for querying of the data. This will enable stakeholders to identify gaps in patrolling and assess opportunities for building interventions or identifying potential new moderators.
The above should be largely stable though will likely shift a little as progress is made on measurement and more feedback is received. We have also discussed article-related features such as [[ https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Language-Agnostic_Quality | quality ]], [[ https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Importance | importance ]], and [[ https://meta.wikimedia.org/wiki/Research:Evaluating_list_building_tools_for_ad-hoc_topic_models | topic ]]. These are not prioritized for now.
**Draft methods**
- (non-comprehensive) literature review, e.g.:
- https://doi.org/10.1145/3686954
- https://doi.org/10.1145/3555225
- https://meta.wikimedia.org/wiki/Research:FlaggedRevisions_investigation
- Quantitative analyses of multiple data sources recording traces of Wikipedia patrollers.
**Background**
This hypothesis emerges from the wider framework at T384860 and is aligned with work at T384600, where several metrics of crowdsourced content moderation based on article maintenance template usage were proposed.
> Our hypothesis is that if we build activity metrics of patrolling on Wikipedia, Our intention is to add in measurement of patrolling to expand our ability to understand moderation and provide insights to guide the development of an on-wiki moderator central place. we can provide insights to guide the development of an on-wiki moderator central place.Some examples of questions that can help us in decisions around what to include in the dataset:
* How many editors have reverted at least Y edits in 30 days?
* How many editors have added a messagebox to an article?
* How many editors provided feedback when rolling back an edit?