Dataset for hackathon is created.
Please see the following notebook for a sample usage:
https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/blob/edit_suggestions_dataset/edit_suggestions/generate_suggestions.ipynb?ref_type=heads

models to pick
We currently discuss it here:
https://wikimedia.slack.com/archives/G01A0FNPLG4/p1768310436209719

@OKarakaya-WMF could you please add a weekly summary for me to pull into Asana?

Reporting format
Progress update on the hypothesis for the week, including if something has shipped:

• We have updated the format of the articles during html to text conversion.
• We have re-generated the most frequent edit types tables after the updates. (article level suggestions.)
• We have generated edit types in section level. We share the findings in scratchpad
• We have started working on creating a dataset for MOS and grammar related edits.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:

Any unresolved dependencies:

New lessons from the hypothesis:

Changes to the hypothesis scope or timeline:

• We have organized the phases as Preparing the edit suggestion types, Evaluation + Iteration and Data Access. We share the scope of the phases in the Timeline

Reporting (20/03/2026)

Progress update on the hypothesis for the week, including if something has shipped:

• We have identified a list of edit suggestion types that we aim to work on. The edit suggestion types are based on frequent edits from users, frequent suggestions from LLMs and suggestions from colleagues/community. We will share it with the editing team.
• Our first evaluation approach is to use user edits as much as possible. We have created a dataset to validate grammar and MOS (Manual of Style) related edit suggestions. We will use it after we create the related suggestions.
• We are working on the second evaluation method llm-as-a-judge. We are currently testing it on a small subset of suggestions.
• We are working on generating edit suggestions based on pre-defined list of edit suggestion types.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:
N/A
Any unresolved dependencies:
N/A
New lessons from the hypothesis:
N/A
Changes to the hypothesis scope or timeline:
N/A

Reporting (27/03/2026)

Progress update on the hypothesis for the week, including if something has shipped:

• We have generated pre-defined edit types for MOS and grammar related edits. Qualitatively, we find useful suggestions on grammar, mos and neutrality.
• We have evaluated pre-defined suggestions by using llm-as-a-judge. Based on a small sample of suggestions (20 suggestions) we have 62% human alignment with the llm-as-a-judge scores. We believe annotating a larger dataset can increase the human alignment and confidence of the llm-as-a-judge scores.
• We work on evaluation based on edits for grammar edit types. We find similarity based comparison useful. We will expand the evaluation based on edits to calculate precision, recall.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:
N/A
Any unresolved dependencies:
N/A
New lessons from the hypothesis:
N/A.
Changes to the hypothesis scope or timeline:
N/A

hey @Sucheta-Salgaonkar-WMF , we have finished the tasks on MLTeam side for this goal. I see there are a couple of for Editing Team though. Are we in favor of closing this goal?

Summary

• T416869 Edit Suggestions - Dataset for hackathon : We started with running an hackathon on a pre-defined dataset for consistency and diversity of the articles.
• First we started with wikitext but due to the formatting issues, we decided to use html of the articles.
• T419840 Edit Suggestions - Formatting for html to text: We solved issues related to converting html to human readable text for context.
• T418097 Edit Suggestions - Edit suggestion generation with loose edit types: We experimented with the loose edit types and decided to continue with pre-defined edit types as it gives more control over the generated sugestions. We have also experimented with different prompts e.g. simple prompt and complex prompt.
• Creating a list of most frequent user edits in enwiki: We use this as a base of target edit types. We enriched it with wished edit types by community and teams. We defined the priority on this list with the help of Editing Team.
• T418102 Edit Suggestions - Edit suggestion generation with pre-defined edit types: We experimented with pre-defined edit types.
• T424347 Editing Suggestions - Dataset generation logic from edits: We created a fixed dataset per edit type for evaluation.
• T421672 Edit suggestions - Precision recall on grammar edit suggestions based on edits: We evaluated edit types by using user edits and llm-as-a-judge. This helps us to identify which models are better, which edit types are more suitable and how the context (article, paragraph) and prompt should be shaped.
• T425957 Editing Suggestions - Eval pipeline for enwiki: Converting evalution to scripts that we can run on the mllab machines.
• T429470 Editing Suggestions - Phase I dataset: Assessing which dataset fits best for the experiment on production. We decided to create fresh suggestions.
• T428882 Editing Suggestions - Increase/refresh LLM generated suggestions: Increase the number of total suggestions to ~30K. This also includes the analysis about how many suggestions will be enough for the experiment. We also analysed which articles should be picked based on their yearly and total edit counts. We generated suggestions with one model and assessed their quality with another model by using llm-as-a-judge. We used llm-as-a-judge for filtering.
• T427794 Editing Suggestions - api: We created an api for serving suggestions including deployment. This enables us to experiment quickly with different suggestion types without requiring an investment on infrastructure. Noting that we have discussed and agreed on how we integrate (api rather than cassandra and open search) and the schema with the Editing Team.
• T424988 Editing Suggestions - Multi-language support: We explore identifying edit types in different wikis. We started working on dewiki and NPOV edit type as well but we haven't finished this task as it will be followed in another goal.
• T430029 Editing Suggestions - Updating documentation.

Fixes

• T428740 LLM Generated Editing Suggestions - API changes: Additional changes on the api requested by the Editing Team.
• T430812 Editing Suggestions - Replace reference [ and ] in target with a special character: Additional changes for target matching.

Repos and more documentation

• api
• experiments
• day-to-day reports
• Evaluation results
• Documentation for integration schema

We processed 30K articles in total based on their edit counts by using gpt-oss-120b model and filtered based on quality by using gemma4 model.
After filtering, we kept suggestions from 14K of the articles which yields 29350 suggestions in total from 3 different categories namely MOS:GEO, NPOV and simplify language.
We serve the suggestion publicly by using an api.