Page MenuHomePhabricator

[SPIKE] How has the reviewer workload at English Wikipedia's Articles for Creation process changed over time?
Open, Needs TriagePublic

Assigned To
None
Authored By
Samwalton9-WMF
Aug 14 2025, 10:09 AM
Referenced Files
F68204775: Unique submissions vs Date.png
Oct 29 2025, 9:03 AM
F68192065: Review by type over time.png
Oct 29 2025, 8:12 AM
F68191726: Publish _ over time.png
Oct 29 2025, 8:12 AM
F67946141: AfC submissions by month(3).png
Oct 28 2025, 11:37 AM
F67926182: AfC submissions by month(2).png
Oct 28 2025, 10:23 AM
Restricted File
Sep 5 2025, 9:37 PM
F65750581: image.png
Aug 14 2025, 10:20 AM
F65750565: Backlog size vs Date.png
Aug 14 2025, 10:13 AM

Description

There is a concern that AI-generated content is adding a significant burden to Wikipedia's patrollers and administrators. One way this is manifesting is in English Wikipedia's Articles for Creation process, where newer editors can submit article drafts for review (in the Draft namespace) by an experienced editor. Reviewers use a helper script to facilitate approving or declining new drafts. Reviewers can take one of three actions:

  • Approving the draft, which subsequently moves it to the main namespace. An edit is published using the script on the article, to remove AfC templates, and to create a redirect from the Draft namespace.
  • Declining the draft, which adds a new templated decline reason to the page, encouraging the user to make adjustments and submit again.
  • Rejecting the draft, which declares the article subject unfit for inclusion on Wikipedia, preventing further re-submissions.

A draft may also be deleted outright before or after any of these steps are taken by using one of the general speedy deletion criteria. For example, an attack page may not be declined outright, but rather deleted on sight. A page which is merely promotional in tone might be declined, but then also tagged for deletion as an unambiguously promotional page.

All Draft pages that have not been edited in 6 months are deleted under the G13 speedy deletion criterion, regardless of whether they have been submitted, declined, or rejected.

Before June 2025, the review script had afch or AFCH in the edit summary. Since June, it uses the AFCH tag instead.

Draft acceptances always have an edit summary starting Publishing accepted . Declines start with Declining submission. Rejections start with Rejecting submission. It's worth noting that the article page edit, redirect creation edit, and page move edit summaries start with Publishing accepted, so counting one accepted page multiple times is possible if a query isn't written correctly.

Questions

  • How has the rate of submissions through AfC changed over time?
  • How has the rate of AfC drafts being accepted, declined, rejected, or deleted changed over time?

Goal: Understand the order of magnitude of this problem, don't necessarily need a complete data set.
Timeline: 1st week of September

Event Timeline

Backlog size vs date 2021 - June 2024 from https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Articles_for_creation/Backlog_chart/weekly:

Backlog size vs Date.png (371×600 px, 22 KB)

Unfortunately this data stopped being collected when the Graph extension was disabled, as that's what it was being collected for.

Dips are the backlog drives which took place in July 2021, January 2023, and November 2023.

Over the course of each year this is how many Draft namespace pages were accepted (moved to main namespace) and deleted:

YearApprovedDeletedApproved %
202013,088106,24211.0%
202112,752131,8548.8%
202210,408119,7028.0%
202312,377113,9479.8%
202412,209104,98610.4%

2025's data is not helpful for comparison because articles are not G13 deleted until 6 months after the last edit.

image.png (539×873 px, 52 KB)

The number of pages being deleted peaked around 2021-22 and has been declining year-over-year since then. This may reflect the active editor numbers which increased during COVID lockdowns. Approval numbers have fluctuated.

Samwalton9-WMF renamed this task from How has the rate of accepts and declines at English Wikipedia's Articles for Creation process changed over time? to How has the reviewer workload at English Wikipedia's Articles for Creation process changed over time?.Aug 14 2025, 10:25 AM

How has the rate of AfC drafts being accepted, declined, rejected, or deleted changed over time?

Note that I started looking at that data but ran into the issue that edit summaries on deleted pages are removed from our public datasets, so a query which combines edit data from both the revision and archive tables is needed, which is beyond my ability to do quickly in Superset :)

Initial attempt using MW History in case it's helpful:

SELECT substring(event_timestamp, 1, 7),
       COUNT(CASE WHEN regexp_like(event_comment, 'Declining|Rejecting') THEN 1 END) as declined,
       COUNT(CASE WHEN regexp_like(event_comment, 'Publishing') THEN 1 END) as published
FROM wmf.mediawiki_history
WHERE SNAPSHOT = '2025-06'
  AND wiki_db = 'enwiki'
  AND event_timestamp > '2020-01-01'
  AND event_entity = 'revision'
  AND page_namespace_historical = 118
  AND event_user_is_permanent = TRUE
  AND regexp_like(event_comment, 'afch|AFCH')
GROUP BY substring(event_timestamp, 1, 7)

How has the rate of submissions through AfC changed over time?

Since around 2021 a submission wizard has been used which generates an edit summary, Submitting using AfC-submit-wizard, however this is not used for every AfC submission. If users add {{AfC submission}}, without any parameters (which LLMs may suggest), the article is added to the review queue regardless of any formal step to submit the article for review.

It might be worth looking at this edit summary, but it will be incomplete, so perhaps overall submission numbers need to be tracked a different way.

Scardenasmolinar renamed this task from How has the reviewer workload at English Wikipedia's Articles for Creation process changed over time? to [SPIKE] How has the reviewer workload at English Wikipedia's Articles for Creation process changed over time?.Aug 14 2025, 6:17 PM
Scardenasmolinar moved this task from Inbox to Product backlog on the Moderator-Tools-Team board.

Broad overview: The Articles for Creation (AfC) process is a quality control mechanism for new articles on English Wikipedia. It allows newer editors to submit article drafts for review by experienced editors before they are published. AfC draft submission = someone adding the special “AfC submission” template to their draft. Experienced editors use a [[ en.wikipedia.org/wiki/WP:AFCH | helper script ]] to evaluate drafts. The accurate way to know the first submission time is to a) collect data as it comes in or b) to look at the wikitext diffs of each draft’s history and find the exact edit where the AfC template first appeared, but the former is not possible for this task and the latter is time consuming and tricky post-hoc; instead of parsing all the diffs, we can use heuristics when looking backwards:

  • Submissions:
    1. Comment-based: edit summaries mentioning "submit" / "AFC submission"
    2. Template-based: new drafts (namespace 118) with AfC template
    3. Deleted drafts: created, later deleted with AfC-related reasons (e.g., G13)
  • Acceptances (approval → move to mainspace):
    • Primary: move logs from Draft (ns=118 → ns=0)
    • Backstops: redirects with "accept" comments, archive accept comments
  • Declines:
    • Rev_comment or AFCH signal indicating decline
    • Draft stays in ns=118 with decline template
  • Rejections:
    • Rev_comment or AFCH signal indicating reject
    • Draft marked unfit; prevents resubmission
  • Deletions:
    • Delete logs in ns=118
    • Focus on AfC-related reasons (G13 applies here)
  • Pending [not requested]:
    • Drafts still in ns=118 with AfC submission template

Key Points:
a) Deletion: Can occur at any stage using general criteria (e.g., attack pages, promotional content)
b) G13 Deletion: Automatic deletion of draft pages inactive for 6+ months
c) Edit summary: Before June 2025: used afch or AFCH in edit summaries
d) Edit summary: Since June 2025: uses AFCH tag
e) "...article page edit, redirect creation edit, and page move edit summaries start with Publishing accepted, so counting one accepted page multiple times is possible if a query isn't written correctly."

Draft Query Approach:

  • tables: page, archive, redirect, logging

https://github.com/nettrom/actrial/blob/master/R/afc.R
https://www.mediawiki.org/wiki/Manual:Archive_table
https://www.mediawiki.org/wiki/Manual:Redirect_table
https://www.mediawiki.org/wiki/Manual:Page_table
https://www.mediawiki.org/wiki/Manual:Database_layout & https://www.mediawiki.org/wiki/Manual:Database_schema
https://github.com/IreneFlorez/GLOW/blob/master/scripts/data_wrangling/3b_collect_contest_cq.ipynb

From the 2018 ACTRIAL work we learn the following tips:

a) Use template detection
JOIN templatelinks tl ON tl_from = page_id
JOIN linktarget lt ON lt_id = tl_target_id
WHERE lt_title IN ('AFC_submission','AFC_submission/draft')

b) track template changes on Draft talk pages (namespace 119) too as decisions often happen on talk pages:
WHERE page_namespace IN (118,119) -- Draft and Draft talk

c) use regex patterns instead of exact string matching to catch variations like "accepted", "accepting", "acceptance", "declined", "declining", "decline":
SUM(rev_comment REGEXP 'accept') AS accepts_hint,
SUM(rev_comment REGEXP 'declin') AS declines_hint

For this spike we looked at submission, deletion, and acceptance data as those could be anchored to aggregated data for accuracy comparison. Deletion data from the logging table and acceptance data from redirect, page, revision, comment, archive tables did not show a significant change or spike over time.

The backlog dipped under 500 in early June 2025, after the backlog drive. We see an upward trend since that date when we look at net day-to-day backlog differences.

  • Observations
    • Upward trend in net day-to-day backlog differences; though not a "high volume" to indicate a major issue to address immediately and unbreak now.
    • No significant spike in accepted/deleted data over time since 2023.
    • No notable increase in monthly submission aggregates since May.
    • Backlog has grown notably since the recent backlog drive however upticks have occurred after previous backlog drives too.
  • Key points
    • AfC work is done in namespace = 118 (drafts)
    • Drafted → delete records can be seen in the Logging_table
    • The AfC template writes a ts=YYYYMMDDHHMMSS parameter in wikitext at submission time. That is the closest ground truth for submission
    • G13 Deletion: deletion of draft pages inactive for 6+ months
  • Fine print
    • templatelinks is snapshot-only (no added ts data)
    • rev_parent_id = 0 counts page creation, not submission
    • categorylinks.cl_timestamp is the last update, not the first time a page entered a category.
    • joining archive to logging is best done with page keys
    • Edit-summary regex is noisy/incomplete; helpers vary, some summaries are blank or non-standard; Example submission summaries:
      • “Submitting for AfC review”
      • “Resubmitting for AfC review”
      • “AfC submission”
      • “Submitting draft to Articles for creation”
      • “Submitted draft for review (AfC)”
      • “Submitting using AfC-submit-wizard”
      • “Script-assisted: AfC submission”
      • “Resubmitting via [[WP:AFC|AfC]] helper”
  • Limitations
    • Deleted drafts without an identifiable submission summary are hard to identify
    • Manual submissions with unusual summaries can slip through
    • Unique submissions that come in at the same time ts=YYYYMMDDHHMMSS can slip through with only 1 counted as the unique
    • Counting every delete in the drafts namespace is an overcount [slight?] as it also counts those made by experienced editors (not just new editors using the AfC process).
  • Assumptions
    • Per team feedback, pages submitted with immediately-deletable problems (which fully-AI-generated content often is) are deleted (vs accepted, declined, rejected)
  • Additional:

https://meta.wikimedia.org/wiki/Research:Autoconfirmed_article_creation_trial

Declined — A reviewer looked at the submission and found fixable problems (sources, notability, tone, etc.). The draft stays in Draft space and can be edited and resubmitted. Look for the decline template; the draft will usually be in “Declined AfC submissions.” If it isn’t edited for ~6 months, it can become eligible for G13 deletion.

Rejected — A stronger outcome used when the topic/page is clearly unsuitable for Wikipedia (e.g., would be uncontroversially deleted if in mainspace, or meets a speedy criterion). AFCH’s “Rejection” tells the author it cannot be resubmitted via AfC. Review guidance explicitly says only reject in clear-cut cases.

Deleted — The draft is removed per deletion policy. This includes G13: abandoned (no edits) for 6 months. Deleted drafts can sometimes be restored if someone intends to work on them.

-- Deletions query
WITH
params AS (
  SELECT '20200101000000' AS report_start, '20250901000000' AS report_end
),
-- First delete per page within the window (draft namespace)
deleted_first AS (
  SELECT
    CASE
      WHEN l.log_page <> 0 THEN CONCAT('id:', l.log_page)
      ELSE CONCAT('nt:', l.log_namespace, ':', l.log_title)
    END AS draft_key,
    MIN(l.log_timestamp) AS first_delete_ts
  FROM logging l
  JOIN params p
  WHERE l.log_type = 'delete'
    AND l.log_action = 'delete'
    AND l.log_namespace = 118
    AND l.log_timestamp >= p.report_start
    AND l.log_timestamp <  p.report_end
  GROUP BY draft_key
),
-- Monthly rollup
deleted_by_month AS (
  SELECT
    CONCAT(SUBSTR(first_delete_ts,1,4), '-', SUBSTR(first_delete_ts,5,2)) AS month,
    COUNT(*) AS deleted
  FROM deleted_first
  GROUP BY month
)
SELECT month, deleted
FROM deleted_by_month
ORDER BY month;

Update --

Inconclusive: At present we do not have the complete data picture and thus advise the team continue to monitor the backlog.

In my spare time I went in and took another look at this, using the production mediawiki replicas to query archive and revision edit summaries. By targeting edit summaries containing Submitting using%, we can get an order-of-magnitude sense of submissions, as I believe most submissions use this rather than other edit summaries. It's not a complete picture, but does have a clear increase this year.

My tactic was to:
Query the archive and revision tables for the edit summary in namespace 118 (Draft). Combine the output with a query for this edit summary in namespace 0 (article), limited to pages for which an earlier query identified as accepted AfC drafts.

AfC submissions by month(3).png (371×600 px, 22 KB)

There is a ~23% increase in monthly submissions this year compared to the previous monthly average, though it's worth noting that after a peak a few months ago it does seem to be decreasing in the 2nd half of this year.

Notebook is here: https://gist.github.com/Samwalton9/8004734c6d550411a4cf3917db1b250d

@Iflorez It would be great if you had time to scan this to see if it makes sense.

A chat with more bars / going back more years might be useful.

You should double check and make sure that the spikes don't correspond to AFC backlog drives. I would speculate that submissions go up during backlog drives because reviews also go up, then the author makes changes and resubmits.

If the hypothesis is that LLM is causing an increase in submissions, we should identify a date / dates that LLM usage began in earnest, and then compare data before and after that. Perhaps there is a "visits to chatgpt.com by month" bar chart floating around somewhere that we could super-impose over your bar chart.

Someone posted a "# of blocks related to LLM" statistic somewhere recently and it showed a definite increase. That might be a good metric to look at. You could recreate that by querying "LLM" in block messages in an SQL query.

How has the rate of AfC drafts being accepted, declined, rejected, or deleted changed over time?

I used a similar strategy to get ballpark figures for this. I used the revision and archive tables to find 'Declining' and 'Rejecting' edit summaries in the Draft namespace, and page move logs to find published articles. I didn't look at deleted pages yet, so this analysis misses any pages which were created/submitted, and then deleted without a formal review ever happening. Qualitatively I've been told that it's a mixed bag as to whether a draft might be reviewed & then tagged for CSD, rather than simply tagged for CSD or deleted outright by a reviewing admin. So we're missing some drafts that get 'reviewed' by being immediately tagged or deleted.

Publish _ over time.png (371×600 px, 17 KB)

The top-line insight I found here is that it looks like publish % hasn't significantly changed over time.

If we break this down to review counts, we can see the last-6-months increase in reviews, which corresponds directly to an increase in submission counts. Average monthly review count is up ~25% in 2025 compared to previous years, which correlates with the 23% increase in submission counts quite nicely.

Spikes are also obvious where backlog drives have happened. Interestingly the acceptance rate appears to increase during backlog drives (~16% -> ~31% during the November 2023 drive). Reviewers told me this is likely because there's more enthusiasm for tackling longer or more complex articles during drives, so good articles that have been sat around for a while awaiting a review are more likely to be reviewed.

Review by type over time.png (371×600 px, 21 KB)

One hypothesis I had was that the increase in submissions this year would be low quality (perhaps LLM-generated) submissions, which would be more likely to be declined, but the rate of acceptances and declines/rejections has increased by about the same amount in 2025, implying that we're seeing an increase in both submissions and newly created articles through AfC this year.

Once again, a review of my notebook would be appreciated in case I've done something wrong here: https://gist.github.com/Samwalton9/3138082c0df4acacf297b025924847e9

Addendum - submissions means 'submit actions', and so counts how many reviews are needed, it's not 1:1 with 'number of unique drafts' because a draft can be submitted, reviewed, submitted again, etc.

This is unique submissions by date. 2025 is ~13% more than the prior average but the distinction is much less.

Unique submissions vs Date.png (371×600 px, 22 KB)