I think we will need to process ~35K articles to make a meaningful decision in one month.
We can do it in 3 batches.
Details are in the doc. (12/06/2026)

link as a ref to the implementation: https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/tree/edit_suggestions_v0/edit_suggestions/v0?ref_type=heads

Sharing some stats and example suggestions. Noting that the suggestion scoring is in progress.

array([np.int64(27413), np.int64(22), np.int64(668),
       'Demographics_of_Sri_Lanka', np.int64(1358095229),
       '2026-06-06T14:47:18Z',
       {'reasoning': 'The description of the Moors uses informal symbols (“+”) and a vague phrasing that reduces clarity. Rewriting in plain language aligns with MOS:GEO and NPOV by presenting factual information without colloquial notation.', 'edit_description': 'Replace the informal notation and clarify the ethnic description of the Moors.', 'edit_type': 'simplify_language', 'prior_text': 'The Moors, descendants of Arab + Indian traders and native Sri Lankan Tamils, form the third largest ethnic group at 9.3% of the population.[31] These Tamil-speaking Muslims are mostly concentrated in urban areas in the southern parts of the island with substantial populations in the Central and Eastern provinces.', 'posterior_text': 'The Moors, who are Tamil‑speaking Muslims of Arab and Indian trader ancestry, constitute the third largest ethnic group at 9.3\u202f% of the population. They are mainly concentrated in urban areas of the south, as well as in the Central and Eastern provinces.'},
       'The description of the Moors uses informal symbols (“+”) and a vague phrasing that reduces clarity. Rewriting in plain language aligns with MOS:GEO and NPOV by presenting factual information without colloquial notation.',
       'Replace the informal notation and clarify the ethnic description of the Moors.',
       'simplify_language',
       'The Moors, descendants of Arab + Indian traders and native Sri Lankan Tamils, form the third largest ethnic group at 9.3% of the population.[31] These Tamil-speaking Muslims are mostly concentrated in urban areas in the southern parts of the island with substantial populations in the Central and Eastern provinces.',
       'The Moors, who are Tamil‑speaking Muslims of Arab and Indian trader ancestry, constitute the third largest ethnic group at 9.3\u202f% of the population. They are mainly concentrated in urban areas of the south, as well as in the Central and Eastern provinces.'],
      dtype=object)

array([np.int64(48234), np.int64(12), np.int64(728), 'Hedd_Wyn',
       np.int64(1356253265), '2026-05-26T17:04:03Z',
       {'reasoning': 'The passage contains wishful language that reflects an editorialized view rather than a factual description, breaching the neutral point of view guideline.', 'edit_description': 'The sentence uses speculative, wish‑ful language that is not appropriate for an encyclopedia article.', 'edit_type': 'NPOV', 'prior_text': 'It is wished that through his sacrifice, and his union with Merch y Drycinoedd at the end of the ode, a better age will come.', 'posterior_text': 'The ode ends with the union of Hedd Wyn’s hero figure and Merch y Drycinoedd, which some readers interpret as a hope for a better future.'},
       'The passage contains wishful language that reflects an editorialized view rather than a factual description, breaching the neutral point of view guideline.',
       'The sentence uses speculative, wish‑ful language that is not appropriate for an encyclopedia article.',
       'NPOV',
       'It is wished that through his sacrifice, and his union with Merch y Drycinoedd at the end of the ode, a better age will come.',
       'The ode ends with the union of Hedd Wyn’s hero figure and Merch y Drycinoedd, which some readers interpret as a hope for a better future.'],
      dtype=object)

The first batch has processed 4735 articles.
We have 11902 suggestions in total
9451 of them got 8 or higher overall score.
7863 of them got 9 or higher overall score.

Scores per edit type
Overall score: NPOV > MOS:GEO > simplify language. This is cool because it aligns with the F1 scores in evaluation by using edits. Good to see a correlation.
Actionability: MOS:GEO > NPOV > simplify language. This is reasonable because MOS:GEO suggestions are generally about fixing a city/region name. So that they are generally small edits.
Correctness: NPOV > simplify language > MOS:GEO. So the problem with MOS:GEO is that they are generally easy edits but they may not be correct.
Scores from previous experiment with edits:

LLM as a judge scores:

I've checked suggestions by their overall scores:

• Scores with 10: I agree 5/5. I've checked 5 of them and I agree all that they are good suggestions.
• Score with 8. I agree 3/5. I've checked 5 of them and I agree 3 of them are good suggestions.
• Score with 1. I agree 5/5. I've checked 5 of them and I agree all of them are not good suggestions.

I think scoring helps to separate good and bad suggestions. However, it's hard to draw a threshold line between them.
I think it's best to increase sample size of evaluated suggestions.
Therefore, my suggestion is to deploy suggestions with scores 8 and more. My initial thought was to pick only 10 or 9/10.
Then, we can check on production if the acceptance rates will correlate.
With this logic:

• with score 10: 24% (we expect high acceptance rate)
• with score between 9 and 10: 58% (we expect medium to high acceptance rate)
• with score between 8 and 9: 16% (we expect medium to low acceptance rate)

I'll upload the new suggestions to the api tomorrow. Please let me know if you agree.

How do we pick the articles:

• We pick articles that have ~1 or ~2 edits per month.
• The pages have less than 800 edits in total. This largely removes conflicting pages e.g. wars, removal of people from companies etc.
• enwiki has 342K articles in total. In this batch, we have 4735 articles processed from this sample and we use ~9K of the generated suggestions. We continue processing until the suggestion count rises to ~30K

full analysis: https://docs.google.com/document/d/19tOyArAzCrSbLIiOKRJFwFWc9E9VYTvaQSfEryMNh4k/edit?tab=t.ohlssxm3wuis

I could not upload the csv with score into phab due to size.
Therefore, I share it here:
https://docs.google.com/spreadsheets/d/1ShvfIAWPnVW9MNi1QYH83xzto6xIX8tVDdCPLJJvxvs/edit?gid=0#gid=0

Change #1305043 had a related patch set uploaded (by Ozge; author: Ozge):

[operations/deployment-charts@master] ml-services: Update editing-suggestions storage URI.

https://gerrit.wikimedia.org/r/1305043

Change #1305043 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Update editing-suggestions storage URI.

https://gerrit.wikimedia.org/r/1305043

Change #1305052 had a related patch set uploaded (by Ozge; author: Ozge):

[operations/deployment-charts@master] ml-services: Update editing-suggestions storage URI.

https://gerrit.wikimedia.org/r/1305052

Change #1305052 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Update editing-suggestions storage URI.

https://gerrit.wikimedia.org/r/1305052

api is updated with the new suggestions
shared with the team.
https://wikimedia.slack.com/archives/C0B350MRBBM/p1782215632966339?thread_ts=1780388489.868199&cid=C0B350MRBBM

You can find the full list of suggestions with the page names in the folder.

We believe this should enable the phase1 tests in experimental mode.
We continue generating more suggestions that will enable phase2 which is the default-on mode.
We can expect the phase2 suggestions to be ready next week. We aim for ~30K suggestions in total.
Afterwards, we can close the current goal on the MLTeam's side.

Noting that the previous suggestions are removed as they were not mostly up-to-date.
In the next steps, we will only continue appending more suggestions without removing the existing ones.
Current suggestions are based on the latest versions of the articles by the time they are generated. We also check if the target matches during the suggestion generation process.

folder: https://analytics.wikimedia.org/published/wmf-ml-models/editing-suggestions/v1/20260623090601/

second batch of suggestion generation is completed. started the scoring.
we need to run them sequentially mostly because of the GPU limits in MLLab

Scores are similar to the first batch and we have 9518 suggestions where overall score is >=8.0

We have received the final batch with 10381 suggestions equal or above to score 8.0

suggestion type specific scores are similar to previous batches.

I'll combine all suggestions and update the api. Afterwards, we can close this ticket.

Combined the suggestions. We have 29350 suggestions in total.

I share below the number of suggestions per suggestion type.

total size of the file is 17M

Change #1306656 had a related patch set uploaded (by Ozge; author: Ozge):

[operations/deployment-charts@master] editing-suggestions: bump model to 20260630103440

https://gerrit.wikimedia.org/r/1306656

Change #1306656 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: editing-suggestions model bump to 20260630103440

https://gerrit.wikimedia.org/r/1306656

Closing this task as we have completed generating suggestions for phase1 (experimental) and phase2 (default-on).
We have updated the api with the suggestions.
I believe this should enable us to close the goal as well after aligning with the Editing Team.

We have processed 30K articles in total. We kept suggestions from 14K of them after filtering with the judge prompt