Page MenuHomePhabricator

WE 4.8.3 Patrolling metrics from enwiki SPI/AIV
Closed, ResolvedPublic

Description

As part of WE 4.8.3, we want to be able to assess the impact of Temporary Accounts on patrolling (here, "investigating accounts suspected of rule-breaking behaviour") times. In order to answer this question, I will need to do the following:

  • Identify promising ways to capture data on account investigation times. From initial feedback from English Wikipedia patrollers, it seems that Administrator Intervention against Vandalism (AIV) and Sockpuppet Investigations (SPI) are promising starting points, due to their semi-structured nature and their nature as reporting venues for rule-breaking. I'm leaning on my prior experience in manually collecting information from SPI.
  • I anticipate the bulk of actual data collection will be done by @STran.
  • Help clean and process any resulting data.
  • Process and present resulting findings. The final deliverable should be a written report plus recorded presentation for the PSI team. The outcome of this research will inform the team's understanding of the hypothesis, including important information on the validity of the KR's stated problem, and help guide the development of future features to ease patroller friction. If this is successful, the parent task will also provide comprehensive baseline metrics on this area of patrolling work for the first time, and provide a launching point for future investigations into patrolling times on other wikis.

Event Timeline

Quick summary of progress over the past week:

  • We have started by analyzing enwiki SPI, specifically, looking at revisions authored by Mz7 (bot), the bot which performs most archival and documentation tasks to the main table of SPI cases. This allows us to get case summaries for all cases dating back to 29 March 2022 (prior to that, a different bot called Amalthea (bot) performed an analogous function).
  • We plan to get the following statistics for SPI:
    • Number of cases filed to SPI from 29 Mar 2022 to present (where "case" is identified by the unique username of the "main" reported account)
    • Number of checkusers involved per case
    • Number of cases by filing status over time
    • Time between the opening of a case and its closure by a CheckUser; this is complicated by the fact that a single case can be repeatedly opened and closed as new information is found, or as investigations are re-filed under other names.
  • We are also investigating the suitability of AIV for this kind of metric gathering. AIV reports follow a very different format, but we are confident that relevant information can be gathered from the edit summaries for this page.

We have continued work on metrics from English Wikipedia's Sockpuppet Investigations (SPI) board, and obtaining baseline metrics from Administrator Interventions against Vandalism (AIV). All data and scripts used can be reviewed at T416469.
SPI data

  • Case durations. Overall, case duration (how long it takes from when a case is first filed, to when it is marked closed in some way) is very variable. For the years 2023-2025 (where we have full-year data), the median case close time is approximately 6 - 15.5 hours. It is relatively consistent year over year. Case durations overall range from a minimum of 0.0378 hours (a little over 2 minutes) to 2594 hours (108 days). Thus, we can say that case durations exhibit extreme variance, with the widest range in the year 2024. This likely prompts an intentional effort to clear the SPI backlog at the end of that year.
    • Median case duration goes from 15.43 hours in 2024, to 5.74 hours in 2025. The post-TA case duration median is 5.33 hours, but there is such high variance that it's hard to attribute this to the introduction of temporary accounts.
    • In conclusion there's no affirmative evidence proving temporary accounts affected case durations at SPI. We can say that there was a sharp drop in case duration metrics from 2024-2025, but this is almost certainly due to an intentional backlog clearing drive.
  • Number of checkusers over 2022-2026 at SPI. The total number of individual checkusers involved in SPI is 60, over the entire period; this number includes now-retired checkusers. The number of checkusers per case, over the entire period, varies between 0 - 3, with the vast majority involving 1 or fewer checkusers (mean 0.65, std 0.56)
  • Number of opened vs. closed investigations. After resolving a data-cleaning issue where cases could be reported as "closed" without ever having an opening date, we find that the number of closed and opened cases are almost 1:1 over the entire dataset. If checkusers at SPI were overwhelmed, we might expect to see more cases marked opened than closed. However, we don't see any significant periods of time where this is the case.

AIV data

  • From 2022-2025, there were approx. 266,000 user-reported cases filed at AIV, of which 133081 were open (50.0%), 126563 were closed (47.5%), and 6516 were stale (2.5%) (automatically removed after ~6 hours if the target of the report is not blocked).
  • The trendline for the total number of cases is extremely linear. There is no evidence of an increased rate in reports after the introduction of temporary accounts.

Next steps
Given the lack of evidence to support the assertion that either SPI or AIV records show any signs of being negatively affected by Temporary Accounts, I have made the decision to end analysis of SPI and AIV metrics. I will attach a fuller report in the coming week to this hypothesis.

cwylo renamed this task from WE 4.8.3 Identify suitable sites for investigation to WE 4.8.3 Patrolling metrics from enwiki SPI/AIV.Feb 6 2026, 11:00 PM
cwylo closed this task as Resolved.

Patrolling metrics on SPI/AIV report deck is available now (WMF-usage only for now). I will be updating this ticket shortly to reflect a continuation of this work. (This took way longer to put together than expected, including several rounds of additional requests for computed metrics and graphs.)