In October 2023 we observed -10%YoY drop in new active editors.
We are trying to investigate where this drop is coming from to be able to understand the root cause. While we have many datasets on editors (active editors, registrations etc.), unfortunately, we are limited in our analysis of new editors beyond project breakdown due to privacy reasons (we cannot store geo information of newly registered users and editors beyond 90 days)
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Mayakp.wiki | T347427 October 2023 Wikimedia movement metrics | |||
| Resolved | Hghani | T352919 November 2023 Wikimedia movement metrics | |||
| Resolved | Hghani | T351759 Investigate drop in New Editors |
Event Timeline
Here are some preliminary findings and descriptive stats
Summary:
Large decrease in eswiki and enwiki
Edits per newly registered user - productivity remains the same or has increased YoY.
Conversation Rate - Number of new registrants that make an edit in the month they registered has stayed the same YoY.
Distribution of edits - Distribution of edits has decreased across the board i.e., number of editors that made 1-4, 5-99, or 100 and more have all decreased which indicates less users registering.
User registrations down overall which appears to be the immediate cause.
Overall there is a drop in anonymous editing and registrant editing which may imply a broader issue at work than just registration as the sole cause. In some wikis, there is an increase in anonymous editing, such as dewiki.
Further investigation required.
Blocks have increased on all our major wikis starting October end. We are investigating to see if some of those blocks are impacting human editors more than we would want them to.
- The surge of blocks on single IP and IP range on eswiki is mainly due to the activity of BlockBot-es. In history, there was a similar bump between November 2021 and August 2022, which peaked in December 2021 (see the IP masking dashboard)
Update from slack channel:
To investigate the hypothesis that ip blocks may be contributing the new active editor drops some analysis was done in this spreadsheet.
Some notes:
To see the impact the block had on mobile isp users (which is the type of ip range being blocked): the first tab should plot out global, enwiki, and eswiki mobile registration counts for the last year or so. I am not sure if there is a decline in mobile registrations that could be attributed the ip range block, and 2023-11 reverses any down trends (or will prove to be an outlier). The trend is pretty consistent among wikis.
The second tab attempts to look at the ratio of new active editors that had their edits reverted within 48 hours versus total new active editors in each month to see if blocking 'a long time abuser' would be reflected in a lower ratio of editors who had their edits reverted. However, the trend seems to be in the opposite direction which could indicate other noise in the data. Individual wikis vary which suggests other factors going on at the same time at the wiki level.
More data is required as some trends are indicative that the block might be doing something, but the magnitude seems to be an open question as there are some seemingly counter-intuitive trends in other data.
More context on the block notices can be found here.
Update:
Initial hypothesis that automated blocks were driving the drop in editors has been disproven by continued observed decreases in new editors in Dec 2023 alongside a block notices returning to normal levels. Currently constructing a new investigation strategy focused on analysis of content contributions.
Update:
Attempted to test hypothesis that the increase in mobile traffic as a proportion of all pageviews (which increased by approximately 10% from 2022-2023) reduces registration, which in turn reduces editing, by looking at the correlation between % change in edits / registration and % change in mobile share of pageviews. Results were inconclusive as there was a small negative correlation in most wikis (approx -.10), and in some wikis there was a positive correlation between these 2.
Also looked to see if there is a shift in the proportion of new mobile registrants vs desktop registrants and if it could show a relationship between edits in months, where there were more mobile registrants as a proportion total registrants. Data looks inconclusive as the proportion stays the same for the most part or months with larger number of mobile registrants don't seem to coincide with a drop in new active editors.
Will continue to devise strategy to look at context of editors and to target content contributions.
I analyzed the variation in new active editors by observing the shift in their numbers across different Wikipedia page topics, calculating how each rise or fall in editors for specific topics contributed to the overall YoY change. For instance, if there was a year-over-year (YoY) decrease of 100 new active editors in a particular month, and the ‘Sports’ category saw a decline of 10 editors, then the decrease in sports contributed 10% to the overall decline in new active editors. The primary goal was to identify any significant patterns that might suggest the content itself is deterring edits (due to saturation or global events, etc.). A more extensive objective is to uncover if there's a specific type of editor that is ceasing to register and therefore, not contributing as an editor. Thus, I charted the trend of YoY editor number declines by topic (charts are only the largest topic contributors on average) for the last year or so (February 2023 until February 2024). The months prior to May 2023 are the months where we did not see any substantial decline in new active editors YoY. The time period after we began to see a decline which accelerated since October 2023.
Charts and dataset of this exploratory analysis is here.
Given we lack data on individuals who choose not to register, the next best approach is to examine the months before May 2023 (or, if we breakdown by wiki, then months where some wikis saw editor drops YoY while others didn't), which had fewer declines, and compare them to the latter half of the year (specifically the last 2 quarters) to discern if there’s a particular cohort or cluster of editor-types that are ‘missing’. This effort aims to categorize editors by topic. Since topics on Wikipedia articles overlap and several editors have exclusively edited unclassified new articles, there are inherent limitations to this analysis (the page classifications haven't been updated since Sept 2022 so subsequently created articles are not classified and were dropped). The overall findings do not pinpoint a specific topic or topics as being responsible for the most significant drops compared to earlier in the year. However, some topics did stand out in the last two quarters where the most substantial YoY declines were observed, which could indicate a trend worth further investigation.
For instance (all or observation for YoY declines),
• A considerable decrease in articles about North American geographic locations and articles about media, film, internet culture in October 2023.
• A significant drop in articles about North American geographic locations in November 2023.
• In December 2023, there was a notable decline in new active editors editing Sports-related articles.
• A substantial decline in Sports-related articles and North American geographic locations in January 2024.
Interestingly, the most marked decline, relative to earlier in the year, was seen in articles about North American geographic locations and articles about Sports (with the exception of Feb 2024 for Sports). At the very least the data does not seem to show indiscriminate drops across topics. Exploring the correlation between topics and others may yield insightful findings. The next steps will involve analyzing the type of edits to identify any trend that might offer insights into changes in editor behavior or if the cause is content-related.
Edit categories
I examined the proportions of edits categorized as ‘Easy’, ‘Medium’, and ‘Hard’, through the method outlined here, comparing year-over-year values to determine if there's a shift in editor behavior that might explain which user cohort, if any, has been absent in months where we've observed YoY drops. The data does not show much connection between months where there were YoY declines within certain wikis and the category that saw the most changes. This suggests that the decrease in new editors is broad and not confined to a specific type of editor, such as those making ‘Hard’ edits. Summarized data.
Some comments about mobile editors:
The proportion of edits made on mobile devices versus non-mobile devices has remained consistent YoY despite the overall decline in new editors. This implies that the composition of new active editors hasn't shifted. This observation is supported by data on mobile-dominant editors (those who edit via their mobile devices more than half the time), where the proportion remains relatively stable YoY. This finding is noteworthy given that 2023 data indicate a majority of pageviews are increasingly from mobile devices, yet this trend hasn't penetrated into new active editors.
I've included a summary of the hypotheses we've tested so far as well suggestions for next steps.
